FTPS not working

Do you read the FAQ?

[X] Yes.
[ ] I am going to read now.

Describe the bug
FTPS does not work, although it should be supported since February 2nd of last year as stated here: Issue 41
To Reproduce
Steps to reproduce the behavior:

set protocol to FTPS (warning)
try to upload a file (error: protocol not supported)
set protocol to FTP (no warning)
try to upload a file again (error: must use encryption)

Expected behavior
FTPS should work without errors.
Desktop (please complete the following information):

OS: Win
VSCode Version: 1.31.1
Extension Version: 1.9.5

Extension Logs from Startup - required
Protocol set to ftps:
[debug] register command "Config" from "./commandConfig.ts"
[debug] register command "List Active Folder" from "./commandListActiveFolder.ts"
[debug] register command "Open Ssh Connection" from "./commandOpenSshConnection.ts"
[debug] register command "Set Profile" from "./commandSetProfile.ts"
[debug] register command "Toggle Output Panel" from "./commandToggleOutputPanel.ts"
[debug] register command "Upload Changed Files" from "./commandUploadChangedFiles.ts"
[debug] register command "Delete Remote" from "./fileCommandDeleteRemote.ts"
[debug] register command "Diff" from "./fileCommandDiff.ts"
[debug] register command "Diff Active File" from "./fileCommandDiffActiveFile.ts"
[debug] register command "Download" from "./fileCommandDownload.ts"
[debug] register command "Download Active File" from "./fileCommandDownloadActiveFile.ts"
[debug] register command "Download Active Folder" from "./fileCommandDownloadActiveFolder.ts"
[debug] register command "Download File" from "./fileCommandDownloadFile.ts"
[debug] register command "Download Folder" from "./fileCommandDownloadFolder.ts"
[debug] register command "Download Force" from "./fileCommandDownloadForce.ts"
[debug] register command "Download Project" from "./fileCommandDownloadProject.ts"
[debug] register command "Edit In Local" from "./fileCommandEditInLocal.ts"
[debug] register command "List" from "./fileCommandList.ts"
[debug] register command "List All" from "./fileCommandListAll.ts"
[debug] register command "Reveal In Explorer" from "./fileCommandRevealInExplorer.ts"
[debug] register command "Reveal In Remote Explorer" from "./fileCommandRevealInRemoteExplorer.ts"
[debug] register command "Sync Both Directions" from "./fileCommandSyncBothDirections.ts"
[debug] register command "Sync Local To Remote" from "./fileCommandSyncLocalToRemote.ts"
[debug] register command "Sync Remote To Local" from "./fileCommandSyncRemoteToLocal.ts"
[debug] register command "Upload" from "./fileCommandUpload.ts"
[debug] register command "Upload Active File" from "./fileCommandUploadActiveFile.ts"
[debug] register command "Upload Active Folder" from "./fileCommandUploadActiveFolder.ts"
[debug] register command "Upload File" from "./fileCommandUploadFile.ts"
[debug] register command "Upload Folder" from "./fileCommandUploadFolder.ts"
[debug] register command "Upload Force" from "./fileCommandUploadForce.ts"
[debug] register command "Upload Project" from "./fileCommandUploadProject.ts"
[info] config at d:\Documents\experia\demo {"remotePath":"/","uploadOnSave":false,"downloadOnOpen":false,"ignore":[],"concurrency":4,"protocol":"ftps","connectTimeout":10000,"interactiveAuth":false,"secure":false,"remoteTimeOffsetInHours":0,"name":"demo","host":"demo.experia.eu","port":115,"username":"******","password":"******","encryption":true}
[trace] run command 'Upload File'
[error] Error: Config validation fail: prop "protocol" fails because ["protocol" must be one of [sftp, ftp, local]].
	at module.exports.t.default.getConfig (C:\Users\Tobi\.vscode\extensions\liximomo.sftp-1.9.5\dist\extension.js:70:64343)
	at Object.u [as handleCtxFromUri] (C:\Users\Tobi\.vscode\extensions\liximomo.sftp-1.9.5\dist\extension.js:1:41708)
	at module.exports.t.createFileCommand.<anonymous> (C:\Users\Tobi\.vscode\extensions\liximomo.sftp-1.9.5\dist\extension.js:1:4045)
	at Generator.next (<anonymous>)
	at module.exports.r (C:\Users\Tobi\.vscode\extensions\liximomo.sftp-1.9.5\dist\extension.js:1:3325)
	at new Promise (<anonymous>)
	at module.exports.r (C:\Users\Tobi\.vscode\extensions\liximomo.sftp-1.9.5\dist\extension.js:1:3102)
	at module.exports.t.createFileCommand.doCommandRun.a.map.t (C:\Users\Tobi\.vscode\extensions\liximomo.sftp-1.9.5\dist\extension.js:1:3987)
	at Array.map (<anonymous>)
	at module.exports.t.createFileCommand.<anonymous> (C:\Users\Tobi\.vscode\extensions\liximomo.sftp-1.9.5\dist\extension.js:1:3980)
	at Generator.next (<anonymous>)
	at s (C:\Users\Tobi\.vscode\extensions\liximomo.sftp-1.9.5\dist\extension.js:1:3158)

Protocol set to ftp:
[debug] register command "Config" from "./commandConfig.ts"
[debug] register command "List Active Folder" from "./commandListActiveFolder.ts"
[debug] register command "Open Ssh Connection" from "./commandOpenSshConnection.ts"
[debug] register command "Set Profile" from "./commandSetProfile.ts"
[debug] register command "Toggle Output Panel" from "./commandToggleOutputPanel.ts"
[debug] register command "Upload Changed Files" from "./commandUploadChangedFiles.ts"
[debug] register command "Delete Remote" from "./fileCommandDeleteRemote.ts"
[debug] register command "Diff" from "./fileCommandDiff.ts"
[debug] register command "Diff Active File" from "./fileCommandDiffActiveFile.ts"
[debug] register command "Download" from "./fileCommandDownload.ts"
[debug] register command "Download Active File" from "./fileCommandDownloadActiveFile.ts"
[debug] register command "Download Active Folder" from "./fileCommandDownloadActiveFolder.ts"
[debug] register command "Download File" from "./fileCommandDownloadFile.ts"
[debug] register command "Download Folder" from "./fileCommandDownloadFolder.ts"
[debug] register command "Download Force" from "./fileCommandDownloadForce.ts"
[debug] register command "Download Project" from "./fileCommandDownloadProject.ts"
[debug] register command "Edit In Local" from "./fileCommandEditInLocal.ts"
[debug] register command "List" from "./fileCommandList.ts"
[debug] register command "List All" from "./fileCommandListAll.ts"
[debug] register command "Reveal In Explorer" from "./fileCommandRevealInExplorer.ts"
[debug] register command "Reveal In Remote Explorer" from "./fileCommandRevealInRemoteExplorer.ts"
[debug] register command "Sync Both Directions" from "./fileCommandSyncBothDirections.ts"
[debug] register command "Sync Local To Remote" from "./fileCommandSyncLocalToRemote.ts"
[debug] register command "Sync Remote To Local" from "./fileCommandSyncRemoteToLocal.ts"
[debug] register command "Upload" from "./fileCommandUpload.ts"
[debug] register command "Upload Active File" from "./fileCommandUploadActiveFile.ts"
[debug] register command "Upload Active Folder" from "./fileCommandUploadActiveFolder.ts"
[debug] register command "Upload File" from "./fileCommandUploadFile.ts"
[debug] register command "Upload Folder" from "./fileCommandUploadFolder.ts"
[debug] register command "Upload Force" from "./fileCommandUploadForce.ts"
[debug] register command "Upload Project" from "./fileCommandUploadProject.ts"
[info] config at d:\Documents\experia\demo {"remotePath":"/","uploadOnSave":false,"downloadOnOpen":false,"ignore":[],"concurrency":4,"protocol":"ftp","connectTimeout":10000,"interactiveAuth":false,"secure":false,"remoteTimeOffsetInHours":0,"name":"demo","host":"demo.experia.eu","port":115,"username":"******","password":"******","encryption":true}
[trace] run command 'Upload File'
[trace] handle upload file for d:\Documents\experia\demo\website\themes\kits\scss\_custom.scss
[debug] < '220 Monitored experia FTP service.\r\n'
[debug] > USER demo
[debug] < '530 Non-anonymous sessions must use encryption.\r\n'
[error] Error: Non-anonymous sessions must use encryption.
	at v (C:\Users\Tobi\.vscode\extensions\liximomo.sftp-1.9.5\dist\extension.js:70:110272)
	at d.<anonymous> (C:\Users\Tobi\.vscode\extensions\liximomo.sftp-1.9.5\dist\extension.js:70:110995)
	at d.emit (events.js:182:13)
	at d.module.exports.d._write (C:\Users\Tobi\.vscode\extensions\liximomo.sftp-1.9.5\dist\extension.js:70:123817)
	at doWrite (_stream_writable.js:410:12)
	at writeOrBuffer (_stream_writable.js:394:5)
	at d.Writable.write (_stream_writable.js:294:11)
	at Socket.p (C:\Users\Tobi\.vscode\extensions\liximomo.sftp-1.9.5\dist\extension.js:70:113366)
	at Socket.emit (events.js:182:13)
	at addChunk (_stream_readable.js:279:12)
	at readableAddChunk (_stream_readable.js:264:11)
	at Socket.Readable.push (_stream_readable.js:219:10)
	at TCP.onread (net.js:636:20)

https://github.com/liximomo/vscode-sftp/wiki/config#secure

Thanks, I still get errors.
To get around "unable to verify first certificate" I've disabled rejectUnauthorized in secureOptions.
With secure set to true I'm stuck at
[info] config at d:\Documents\experia\demo {"remotePath":"/","uploadOnSave":false,"downloadOnOpen":false,"ignore":[],"concurrency":4,"protocol":"ftp","connectTimeout":10000,"interactiveAuth":false,"secure":true,"remoteTimeOffsetInHours":0,"name":"demo","host":"demo.experia.eu","port":115,"username":"******","password":"******","secureOptions":{"rejectUnauthorized":false}}
[trace] run command 'Diff'
[trace] handle diff for d:\Documents\experia\demo\website\themes\kits\scss\_custom.scss
[debug] < '220 Monitored experia FTP service.\r\n'
[debug] > AUTH TLS
[debug] < '234 Proceed with negotiation.\r\n'
[debug] > PBSZ 0
[debug] < '200 PBSZ set to 0.\r\n'
[debug] > PROT P
[debug] < '200 PROT now Private.\r\n'
[debug] > USER demo
[debug] < '331 Please specify the password.\r\n'
[debug] > PASS ******
[debug] < '230 Login successful.\r\n'
[debug] > FEAT
[debug] < '211-Features:\r\n'
[debug] < ' AUTH TLS\r\n'
[debug] < ' UTF8\r\n'
[debug] < ' EPRT\r\n'
[debug] < ' EPSV\r\n'
[debug] < ' MDTM\r\n'
[debug] < ' PASV\r\n'
[debug] < ' PBSZ\r\n'
[debug] < ' PROT\r\n'
[debug] < ' REST STREAM\r\n'
[debug] < ' SIZE\r\n'
[debug] < ' TVFS\r\n'
[debug] < '211 End\r\n'
[debug] > TYPE I
[debug] < '200 Switching to Binary mode.\r\n'
[debug] > PASV
[debug] < '227 Entering Passive Mode (85,237,88,214,156,150).\r\n'
[debug] > RETR /website/themes/kits/scss/_custom.scss
[debug] < '150 Opening BINARY mode data connection for /website/themes/kits/scss/_custom.scss (8216 bytes).\r\n'

With secure set to implicit:
[info] config at d:\Documents\experia\demo {"remotePath":"/","uploadOnSave":false,"downloadOnOpen":false,"ignore":[],"concurrency":4,"protocol":"ftp","connectTimeout":10000,"interactiveAuth":false,"secure":"implicit","remoteTimeOffsetInHours":0,"name":"demo","host":"demo.experia.eu","port":115,"username":"******","password":"******","secureOptions":{"rejectUnauthorized":false}}
[trace] run command 'Diff'
[trace] handle diff for d:\Documents\experia\demo\website\themes\kits\scss\_custom.scss
[error] Error: Timeout while connecting to server
	at Timeout.module.exports.t.default._doConnect.setTimeout [as _onTimeout] (C:\Users\Tobi\.vscode\extensions\liximomo.sftp-1.10.0\dist\extension.js:70:109185)
	at ontimeout (timers.js:427:11)
	at tryOnTimeout (timers.js:289:5)
	at listOnTimeout (timers.js:252:5)
	at Timer.processTimers (timers.js:212:10)

and with it set to control:
[info] config at d:\Documents\experia\demo {"remotePath":"/","uploadOnSave":false,"downloadOnOpen":false,"ignore":[],"concurrency":4,"protocol":"ftp","connectTimeout":10000,"interactiveAuth":false,"secure":"control","remoteTimeOffsetInHours":0,"name":"demo","host":"demo.experia.eu","port":115,"username":"******","password":"******","secureOptions":{"rejectUnauthorized":false}}
[trace] run command 'Diff'
[trace] handle diff for d:\Documents\experia\demo\website\themes\kits\scss\_custom.scss
[debug] < '220 Monitored experia FTP service.\r\n'
[debug] > AUTH TLS
[debug] < '234 Proceed with negotiation.\r\n'
[debug] > USER demo
[debug] < '331 Please specify the password.\r\n'
[debug] > PASS ******
[debug] < '230 Login successful.\r\n'
[debug] > FEAT
[debug] < '211-Features:\r\n'
[debug] < ' AUTH TLS\r\n'
[debug] < ' UTF8\r\n'
[debug] < ' EPRT\r\n'
[debug] < ' EPSV\r\n'
[debug] < ' MDTM\r\n'
[debug] < ' PASV\r\n'
[debug] < ' PBSZ\r\n'
[debug] < ' PROT\r\n'
[debug] < ' REST STREAM\r\n'
[debug] < ' SIZE\r\n'
[debug] < ' TVFS\r\n'
[debug] < '211 End\r\n'
[debug] > TYPE I
[debug] < '200 Switching to Binary mode.\r\n'
[debug] > PASV
[debug] < '227 Entering Passive Mode (85,237,88,214,156,124).\r\n'
[debug] > RETR /website/themes/kits/scss/_custom.scss
[debug] < '522 Data connections must be encrypted.\r\n'
[error] Error: Data connections must be encrypted.
	at v (C:\Users\Tobi\.vscode\extensions\liximomo.sftp-1.10.0\dist\extension.js:70:110347)
	at d.<anonymous> (C:\Users\Tobi\.vscode\extensions\liximomo.sftp-1.10.0\dist\extension.js:70:111070)
	at d.emit (events.js:182:13)
	at d.module.exports.d._write (C:\Users\Tobi\.vscode\extensions\liximomo.sftp-1.10.0\dist\extension.js:70:123892)
	at doWrite (_stream_writable.js:410:12)
	at writeOrBuffer (_stream_writable.js:394:5)
	at d.Writable.write (_stream_writable.js:294:11)
	at TLSSocket.p (C:\Users\Tobi\.vscode\extensions\liximomo.sftp-1.10.0\dist\extension.js:70:113441)
	at TLSSocket.emit (events.js:182:13)
	at addChunk (_stream_readable.js:279:12)
	at readableAddChunk (_stream_readable.js:260:13)
	at TLSSocket.Readable.push (_stream_readable.js:219:10)
	at TLSWrap.onread (net.js:636:20)
[debug] > NOOP

Any idea how to fix or work around this?

@trashbytes,
having those lines in sftp.json should work (and work for me):
"protocol": "ftp",
"secure": true

If you're having certificate problems, include (and you already have, but just in case):
"secureOptions": {
    "rejectUnauthorized": false
}

and FTPS should work.
Apart from that, make sure your hostname and port are set correctly.
If you're still unable to connect, there might be a server-side issue. In that case inspect the server logs to see what is happening.

@urki0x Thank you very much.

same problem here, cannot connect to Cpanel FTP account
sftp.json has been edited according to @urki0x
debug log shows nothing regarding of failed connection
got popup notification :  "Connecting failed, Error: Connecting failed, Error: Timed out while waiting for handshake"
here i attach some packet capture from wireshark
ftps-failed.zip

I'm having the same problem. I am using FTPS with
"secure": true,
"secureOptions": { "rejectUnauthorized": false },
"passive": true,
but it hangs on the same place:
'200 Switching to Binary mode.\r\n'
[08-21 09:24:09] [debug] > PASV
[08-21 09:24:09] [debug] < '227 Entering Passive Mode (94,237,29,109,250,195).\r\n'
[08-21 09:24:09] [debug] > PASV
[08-21 09:24:09] [debug] < '227 Entering Passive Mode (94,237,29,109,250,17).\r\n'
[08-21 09:24:09] [debug] > LIST /var/www/html/cm/wp-content/themes/scrawl
[08-21 09:24:09] [debug] < '150 Here comes the directory listing.\r\n'
[08-21 09:24:09] [debug] < '226 Directory send OK.\r\n'
[08-21 09:24:09] [debug] > PASV
[08-21 09:24:09] [debug] < '227 Entering Passive Mode (94,237,29,109,250,213).\r\n'
[08-21 09:24:09] [debug] > RETR /var/www/html/cm/wp-content/themes/scrawl/style.css
[08-21 09:24:09] [debug] < '150 Opening BINARY mode data connection for /var/www/html/cm/wp-content/themes/scrawl/style.css (53339 bytes).\r\n'
I cannot use VSCode because of this and I am deeply unhappy. I am using the same settings in FileZilla and it's working perfectly.
Is there a solution to this issue?

@trashbytes,
having those lines in sftp.json should work (and work for me):
"protocol": "ftp",
"secure": true

If you're having certificate problems, include (and you already have, but just in case):
"secureOptions": {
    "rejectUnauthorized": false
}

and FTPS should work.
Apart from that, make sure your hostname and port are set correctly.
If you're still unable to connect, there might be a server-side issue. In that case inspect the server logs to see what is happening.

Thank you!
It really works!
And your picture is the best on GitHub I think ;)

@trashbytes,
having those lines in sftp.json should work (and work for me):
"protocol": "ftp",
"secure": true

If you're having certificate problems, include (and you already have, but just in case):
"secureOptions": {
    "rejectUnauthorized": false
}

and FTPS should work.
Apart from that, make sure your hostname and port are set correctly.
If you're still unable to connect, there might be a server-side issue. In that case inspect the server logs to see what is happening.

Saved Hell lot of time.

Worked for me as well

@liximomo what does
[07-08 11:40:28] [debug] Client: none auth failed

I am also having issues uploading on save. I will provide more details soon as soon as I figure out what might be useful to share here.

@trashbytes,
having those lines in sftp.json should work (and work for me):
"protocol": "ftp",
"secure": true

If you're having certificate problems, include (and you already have, but just in case):
"secureOptions": {
    "rejectUnauthorized": false
}

and FTPS should work.
Apart from that, make sure your hostname and port are set correctly.
If you're still unable to connect, there might be a server-side issue. In that case inspect the server logs to see what is happening.

Why do I need to change it to that?
I always had it like this and worked fine:
    "protocol": "sftp",

@trashbytes,
having those lines in sftp.json should work (and work for me):
"protocol": "ftp",
"secure": true

If you're having certificate problems, include (and you already have, but just in case):
"secureOptions": {
    "rejectUnauthorized": false
}

and FTPS should work.
Apart from that, make sure your hostname and port are set correctly.
If you're still unable to connect, there might be a server-side issue. In that case inspect the server logs to see what is happening.

This doesn't seem to work anyway with all changes you suggested.
That gives me new errors:
[07-08 11:44:56] [info] Using profile: prod
[07-08 11:44:56] [info] [file-save] /Users/brando/automl-meta-learning/automl-proj/experiments_meta_learning/meta_learning_experiments_submission.py
[07-08 11:44:56] [info] Using profile: prod
[07-08 11:44:56] [trace] handle upload file for /Users/brando/automl-meta-learning/automl-proj/experiments_meta_learning/meta_learning_experiments_submission.py
[07-08 11:44:58] [debug] < 'SSH-2.0-OpenSSH_7.4\r\n'
[07-08 11:45:06] [error] Error: Timeout while connecting to server
	at Timeout._onTimeout (/Users/brando/.vscode/extensions/liximomo.sftp-1.12.9/dist/extension.js:70:109790)
	at listOnTimeout (internal/timers.js:531:17)
	at processTimers (internal/timers.js:475:7) download /Users/brando/automl-meta-learning/automl-proj/experiments_meta_learning/meta_learning_experiments_submission.py
[07-08 11:45:15] [info] Using profile: prod
[07-08 11:45:15] [info] [file-save] /Users/brando/automl-meta-learning/automl-proj/experiments_meta_learning/meta_learning_experiments_submission.py
[07-08 11:45:15] [info] Using profile: prod
[07-08 11:45:15] [trace] handle upload file for /Users/brando/automl-meta-learning/automl-proj/experiments_meta_learning/meta_learning_experiments_submission.py
[07-08 11:45:15] [debug] < 'SSH-2.0-OpenSSH_7.4\r\n'
[07-08 11:45:25] [error] Error: Timeout while connecting to server
	at Timeout._onTimeout (/Users/brando/.vscode/extensions/liximomo.sftp-1.12.9/dist/extension.js:70:109790)
	at listOnTimeout (internal/timers.js:531:17)
	at processTimers (internal/timers.js:475:7) download /Users/brando/automl-meta-learning/automl-proj/experiments_meta_learning/meta_learning_experiments_submission.py

https://github.com/liximomo/vscode-sftp/wiki/config#secure

Why is this link here? I read but I didn't know what to do with it. What do I do with it?

One of the chats also had this link. I read it but I wasn't confused what I was suppose to do with it:
https://github.com/liximomo/vscode-sftp/wiki/config#secure

@brando90 is this a new connection?

I always had it like this and worked fine:

"protocol": "sftp",

If everything was working fine before, has it suddenly stopped today?
Have you checked all of the network pieces of the puzzle?
Are you able to connect to this FTP using something like filezilla?

@brando90 is this a new connection?

I always had it like this and worked fine:

"protocol": "sftp",

If everything was working fine before, has it suddenly stopped today?

yes

Have you checked all of the network pieces of the puzzle?

I'm sure I've not check them all, I'm not even sure what to check.
Though, I tried connecting to the server. That didn't worked. I tried rsyncing to it from the command line. That didn't worked. I had the administrators help me. We check the server's logs. That didn't work. I re-installed vscode, tried different versions of sftp...nothing worked.

Are you able to connect to this FTP using something like filezilla?

Is filezilla another extensions?

Are there any new firewall rules stopping it? (I know I've blocked myself before by accident by installing a FW management tool)

It says my firewall is off...
Any suggestions are appreciated. Thanks for replying.

I would download filezilla, or some other free ftp client, and try to make the connection. If you can't make it from a different ftp client either, odds are you won't be able to make it from this one.
Adam
This email was sent from a mobile device. Please excuse spelling and grammar.
On Wed, Jul 8, 2020 at 2:18 PM -0700, "brando90"<EMAIL_ADDRESS>wrote:
@brando90 is this a new connection?
I always had it like this and worked fine:
"protocol": "sftp",
If everything was working fine before, has it suddenly stopped today?
yes
Have you checked all of the network pieces of the puzzle?
I'm sure I've not check them all, I'm not even sure what to check.
Though, I tried connecting to the server. That didn't worked. I tried rsyncing to it from the command line. That didn't worked. I had the administrators help me. We check the server's logs. That didn't work. I re-installed vscode, tried different versions of sftp...nothing worked.
Are you able to connect to this FTP using something like filezilla?
Is filezilla another extensions?
Are there any new firewall rules stopping it? (I know I've blocked myself before by accident by installing a FW management tool)
It says my firewall is off...
Any suggestions are appreciated. Thanks for replying.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.


Just in case you're not aware...

There is this very similar React wrapper project (which also uses katex under-the-hood) here: https://github.com/talyssonoc/react-katex

Thanks @Venryx! We were using this react component inside our project because at that time there wasn't anything out there that could be used inside React. The alternative was Mathquill, but that was manipulating the DOM once it was rendered and it's hard to contol within react.
Later we decided to extract the module to be reusable across our projects and leave it out there for everyone. At the point we published we weren't aware of react-katex.
We are aware that some parts of our module are not standard, and at some point we'd like to improve them, but so far we haven't had the bandwidth to improve our implementation to be more open.


Add event handlers as tag helpers

This change adds support for mapping DOM event handlers as tag helpers
that function in a bi-modal way.
This is a new first-class feature for DOM events, and replaces a few
workarounds like using @onclick(...) or click=@{ ... }. I haven't
removed those things yet, this is a first pass to get the new support
in, we'll remove those things when we're totally satisfied.
When used with a string like <button onclick="foo" /> the result is
a simple HTML attribute .
But when used with an implicit expression like
<button onclick="@Foo" /> or
<button onclick="@(x => Clicked = true)" /> a C# function is bound to
the click event from the DOM.

I've addressed the minor feedback here, I'm going to follow up on three things:

Add a fallback for arbitrary events
Expand the list of known events and types
Use a different strategy for delegates that will result in fewer allocations and smaller diffs


go_enrichment  ValueError: Attempted relative import in non-package

python /Users/Genesis/Downloads/goatools-0.6.4/goatools/go_enrichment.py /Users/Genesis/Desktop/data/study.txt /Users/Genesis/Desktop/data/population.txt /Users/Genesis/Desktop/data/association.txt
Traceback (most recent call last):
File "/Users/Genesis/Downloads/goatools-0.6.4/goatools/go_enrichment.py", line 22, in
from multiple_testing import Methods, Bonferroni, Sidak, HolmBonferroni, FDR, calc_qval
File "/Users/Genesis/Downloads/goatools-0.6.4/goatools/multiple_testing.py", line 12, in
from .ratio import count_terms
ValueError: Attempted relative import in non-package

Thank you for your interest in GOATOOLS and taking the time to report this issue.
We were not able to duplicate this error. Please re-open if this is still an issue.


feat: status of intent

Summary
The main purpose of the PR is to formalize states of the swap flow and statuses of an intent. The main changes are made in swapIntentMachine. The swap flow states are grouped by intent creation outcomes:

"sad" states (we could not create an intent)

ERR_USER_DIDNT_SIGN
ERR_SIGNED_DIFFERENT_ACCOUNT
ERR_CANNOT_VERIFY_PUBLIC_KEY
ERR_CANNOT_ADD_PUBLIC_KEY
ERR_CANNOT_PUBLISH_INTENT

"happy" states (we managed to create an intent)

SETTLED
NOT_FOUND_OR_NOT_VALID
ERR_CANNOT_OBTAIN_INTENT_STATUS

Why NOT_FOUND_OR_NOT_VALID and ERR_CANNOT_OBTAIN_INTENT_STATUS statuses are considered as happy? Either these statuses were unintentionally added to this group, or the names are not consistent here.

Why NOT_FOUND_OR_NOT_VALID and ERR_CANNOT_OBTAIN_INTENT_STATUS statuses are considered as happy? Either these statuses were unintentionally added to this group, or the names are not consistent here.

You can call it either name you want, "happy" means we managed to created a intent.

:tada: This PR is included in version 1.23.0 :tada:
The release is available on:

GitHub release
npm package (@latest dist-tag)

Your semantic-release bot :package::rocket:


Document lumina-xdg-open utility

Utility added for 1.3.0 release needs full workup: Explanatory text and minimum one screenshot (luminautl.rst).

https://github.com/q5sys/qt5.desktopcreator/blob/master/README.md has some important information about this utility.


0.13.2 breaks Api client with kubeconfig if a serviceaccount is already in use(?)

Which project are you reporting a bug for?
kr8s
What happened?
A bug happened!
When using kr8s in a kubernetes pod which has access to the current API,
client = await kr8s.asyncio.api()

And trying to create new clients,
clients[name] = await kr8s.asyncio.api(
                kubeconfig=kubeconfig_file.name,
            )

The clients created with the kubeconfig= parameter use the same in-cluster client instead of connecting to the right place.
This does not happen on 0.13.0 and 0.13.1.

Thanks for reporting this regression. I've raised #311 to resolve it.

Should be fixed in v0.13.3 :).


Ignore unknown ConEmu and terminal sequences in console logs

Description
A bug in .NET SDK was adding non-display sequences to console logs. While the SDK is fixing their bug, it's possible a container or exe outputs these sequences in regular execution. The console logs UI should never try to display ANSI console sequences that it doesn't support.
This PR skips all unsupported CSI and ConEmu escape codes.
Fixes https://github.com/dotnet/aspire/issues/5716
Checklist

Is this feature complete?

[x] Yes. Ready to ship.
[ ] No. Follow-up changes expected.

Are you including unit tests for the changes and scenario tests if relevant?

[x] Yes
[ ] No

Did you add public API?

[ ] Yes

If yes, did you have an API Review for it?

[ ] Yes
[ ] No

Did you add <remarks /> and <code /> elements on your triple slash comments?

[ ] Yes
[ ] No

[x] No

Does the change make any security assumptions or guarantees?

[ ] Yes

If yes, have you done a threat model and had a security review?

[ ] Yes
[ ] No

[x] No

Does the change require an update in our Aspire docs?

[ ] Yes

Link to aspire-docs issue:

[x] No

Microsoft Reviewers: Open in CodeFlow

/azp run

/azp run

/azp run


noImplicitAny confusion

bad title but i don't know how to better describe it
TypeScript Version:  2.4.2 and 2.5.0-dev.20170808
Code
// needs noImplicitAny

function get123 () {
	return 123;
}

/**
 * @param original    some function
 * @param replacement another function with the same type
 */
function test <T> (original: T, replacement: T) {
	// ...
	return original;
}

// 'get123_' implicitly has type 'any' because it does not have a type
// annotation and is referenced directly or indirectly in its own initializer.
const get123_ = test(

	get123,

	// Function implicitly has return type 'any' because it does not have a
	// return type annotation and is referenced directly or indirectly in one
	// of its return expressions.
	function () {
		return get123_();
	}
);

Expected behavior:
i don't think there should be any errors at all, and get123_ and the anonymous function should have the same type as get123
the test function is meant to take two things with the same type but it's like it thinks they're the same value
Actual behavior:
it says the get123_ variable is referenced in its own initializer when its value is just get123
it says the anonymous function is referenced in one of its return expressions, when it's only referencing another function with the same type (get123_ which is just get123 returned from the test function)
(at least that's how i see it)

i was trying to write a function that would do Object.assign to an object's property descriptor and return its previous value, and using it would often give this error even though there's no circular references going on like it thinks
you'd use it like this:
const open = patchDescriptor(window, "open", {
	value: function open (url?: string, target?: string) {
		// Reflect.apply returns `any` so this doesn't error
		return Reflect.apply(open.value, this, arguments);
	}
});

@epicgirl1998 There is a circular reference here. Perhaps you meant function () { return get123() } instead of function () { get123_() }?

it would be a circular reference (infinite recursion) if test returned the second argument (the anonymous function using the return value) but it's returning the first one which is just get123
this modified version of the example works when pasted to a javascript console (no infinite loop):
function get123 () {
	return 123;
}

let replacement;

function test (original, replacement_) {
	replacement = replacement_;
	// ...
	return original;
}

const get123_ = test(

	get123,

	function () {
		return get123_();
	}
);

console.assert(replacement() === 123);
console.assert(get123_() === 123 && get123_ === get123);

@epicgirl1998 There's a difference between a circular reference and infinite recursion (e.g. it is possible to write let x = 10; function f() { return --x ? f() : x }, which is finitely recursive, but still runs into this issue). TypeScript doesn't actually evaluate your program, so it has no idea whether the circular reference will actually cause infinite recursion.
The problem from TypeScript's perspective is that the type of the variable get123_ depends on the type of the function expression function () { return get123_() }, for which the return type depends on the type of get123_; it can't reason its way out of this circle and gives you an error.

Looks like a duplicate of #17711, solved by @ahejlsberg at #17771.

In some super-ideal world we could detect when the type of an expression "doesn't matter" for purposes of type inference, or detect a closed loop, but this turns out to be rather difficult for reasons described above. get123_  definitely references itself in a position that matters for the purposes of inferring its type and I don't see any way out of that.


Having trouble getting videos of moderate size from server

Im trying to get a video from gridfs after successfully uploading it, checking mongoclient i can see that all the proper subfields lie in there. However I can't figure out why when i try to play certain videos it fails to fetch it (network tab shows 0 bytes received) however for some of them, for example a youtube video downloaded as a mp4, and then uploaded to my web application plays and is fetched fine.
If i go to my api that gets this video 'file/:someid' then the video downloads locally to my computer just fine, but trying to play it on the video player seems to fail. I don't know if im fetching or forgetting properties to set when loading it through busboy or if its a videoplayer problem or something.
I posted this issue with more discrete details on stackoverflow :
http://stackoverflow.com/questions/36760404/express-multer-uploading-moderate-large-files-not-playing-on-player
The code i have in express is this
app.post('/file/video', function(req, res) {
  //console.log(req.body);
  var busboy = new Busboy({ headers : req.headers });
  var fileId = new mongoose.mongo.ObjectId();
  //console.log(req.body);

  busboy.on('file', function(fieldname, file, filename, encoding, mimetype) {
    //console.log('got file', filename, mimetype, encoding);
    var writeStream = gfs.createWriteStream({
      _id: fileId,
      filename: filename,
      mode: 'w',
      content_type: mimetype,
    });
    file.pipe(writeStream);
  }).on('finish', function() {
    // show a link to the uploaded file
    req.user.videos.push(fileId);
    req.user.save();
    //console.log(req.user);

    res.end();

    /*var proc = new ffmpeg('/Documents/20000pitches/parkour.mp4')
  .takeScreenshots({
      count: 1,
      timemarks: [ '600' ] // number of seconds
    }, '/', function(err) {
    console.log('screenshots were saved')
  });
*/
  });

  req.pipe(busboy);
});

app.get('/file/:id', function(req, res) {
  gfs.findOne({ _id: req.params.id }, function (err, file) {
    if (err) return res.status(400).send(err);
    if (!file) return res.status(404).send('');

    res.set('Content-Type', file.contentType);
    res.set('Content-Disposition', 'attachment; filename="' + file.filename + '"');

    var readstream = gfs.createReadStream({
      _id: file._id
    });

    readstream.on("error", function(err) {
      console.log("Got error while processing stream " + err.message);
      res.end();
    });

    readstream.pipe(res);
  });
});

And video is played like this:
<Video style ={{height:'100%', width: '100%'}}poster={singleposts.thumbnail} controls ref="video" onProgress={this.onProgress}>
            <source style = {{width: '100%', height: '100%'}} src={"/file/" + singleposts.videoURL} type="video/mp4" />
        </Video>

It decides to hang on the get request for getting the src for my video, like the id and everything is correct i console log it, i check in the browser, etc.
An example is here
http://20000passions.kkotwal.me/gallery/57185f3e1b237af47940f370
Where it fails vs when it works is here
http://20000passions.kkotwal.me/gallery/56fe76dae94f67c974394d42
I noticed that when it didnt work it was of mp4 format from my phone and the quality and resolution were both high

Closing this as it doesn't seem like a busboy issue, especially since you've identified that downloading the file locally to your system using the same API endpoint works just fine.
My guess is that there is either something wrong with the video player or its configuration, or the server is not sending some information that the video player is expecting (or is sending information that the player is not expecting, such as the Content-Disposition header).


Feature/Add NFT

[x] Complete pixel perfect Step 1 of Add NFT screen.

[ ] Complete pixel perfect Step 2 of Add NFT screen.
[ ] Complete pixel perfect Step 3 of Add NFT screen.
[ ] Complete pixel perfect Step 4 of Add NFT screen.

@ihor-stack Please fix CI/CD issue


Update description of nonprofit projects

[x] I have read freeCodeCamp's contribution guidelines.
[x] My pull request has a descriptive title (not a vague title like Update index.md)
[x] My pull request targets the master branch of freeCodeCamp.
[x] None of my changes are plagiarized from another source without proper attribution.
[x] My article does not contain shortened URLs or affiliate links.

Closes #XXXXX

We have a ton of additional projects we plan to build in the near future, too.

Is there a way to elaborate on a few of the future planned projects?

You can contribute to these open source projects whenever you feel ready.

Also, maybe highlight a few other FCC open source projects with links to each?  If nothing else, maybe link to https://github.com/freeCodeCamp on the phrasethese open source projects in the second sentence?

@RandellDawson Thanks for your feedback. At this point, the only nonprofit project that organizations are actively using is Mail for Good, which I've linked to in the article.
We're still working on concrete plans for expanding our other projects, like League For Good, and getting them truly ready for adoption by nonprofits, so I'd rather not link to those yet.
If you think this article is otherwise OK, please merge it 👍


formatting differently than VSCode format on save

On some files, I lint, format and save in VSCode, but then on push, husky runs pretty-quick and formats some files differently that make it fail the eslint check on push that comes after it when it checks the prettier rules.

I ran into a similar issue. I think the problem is with the VS Code Prettier extension resolving your prettier config file. Prettier uses cosmic config  which allows you to define your prettier config in lots of ways. However prettier-vscode is only looking for .prettierrc or .editorconfig config files.  So if you aren't using one of those prettier-vscode falls back to their settings. I fixed this by configuring prettier with a .prettierrc file.

Well it looks like it should support more config formats https://github.com/prettier/prettier-vscode/blob/3c80018d69fa129fed638f624404e728870e4a90/src/PrettierEditService.ts#L32 but I had issues when it was defined as a prettier.config.js file

This might be the same thing as https://github.com/azz/pretty-quick/issues/95

Not sure to understand what's the issue.
A minimal but runnable online reproduction is required.

Hi !
I have a similar issue.
I am using pretty-quick and VsCode on a Vue.js project.
I've enabled the "format on save" option.
However, when I save, it formats the file differently than when formatting with npx pretty-quick.
I think maybe it's because VsCode can't find the Prettier configuration file
Here is my package.json file:
{
  "name": "xxx",
  "version": "0.3.20-0",
  "private": true,
  "scripts": {
    "serve": "vite",
    "build": "vite build",
    "preview": "vite preview",
    "prepare": "husky",
    "lint": "prettier --check .",
    "fix": "prettier --write ."
  },
  "dependencies": {
    "@oruga-ui/oruga-next": "~0.7.0",
    "keycloak-js": "~25.0.4",
    "vue": "~3.5.8",
    "vue-router": "~4.4.5",
  },
  "devDependencies": {
    "@tsconfig/node18": "^18.2.4",
    "@types/node": "^18.18.5",
    "@vitejs/plugin-vue": "^4.6.2",
    "@vue/tsconfig": "~0.5.1",
    "autoprefixer": "^10.4.20",
    "husky": "^9.1.6",
    "postcss": "^8.4.47",
    "postcss-grid-kiss": "^3.1.0",
    "postcss-loader": "^8.1.1",
    "postcss-nested": "^6.2.0",
    "postcss-nesting": "^13.0.0",
    "postcss-simple-vars": "^7.0.1",
    "prettier": "^3.3.3",
    "pretty-quick": "^4.0.0",
    "tslint-config-prettier": "^1.18.0",
    "typescript": "~5.2.0",
    "vite": "^4.5.5",
    "vite-plugin-webpackchunkname": "^0.2.4",
    "vue-tsc": "^2.1.6",
    "webpack": "^5.95.0"
  }
}


Execute command on leave & finish.

When the player leaves a parkour course, a command can be executed regardless of anything. This will allow for triggers of external plugins that don't use the Parkour API.
For example, to re-enable a custom scoreboard via "/(plugin) show" when the player is finished with Parkour, regardless if they failed, completed or quit a course.

There are now 6 parkour events that could trigger a configurable list of commands.

Join
Leave
Prize
Checkpoint
Death
Finish (Only triggered when "rewardonce" has been enabled and the player has finished the course again.

Each of these can be added to using the "/pa setcourse (course)" conversation, or a single command using "/pa setcourse (course) command (type) (command)". The %PLAYER% placeholder can be used to insert the player's name into the command before execution.


Handle SIGTERM properly.

SIGTERM is the default signal sent by most standard program: kill(1), task managers, etc...
It is nice if ninja can handle it properly.

What's wrong with how ninja currently handles sigterm?

It currently has no handler for SIGTERM so the process is terminated, the cleanup is not done and sub-processes are not stopped either. The PR handles SIGTERM exactly like SIGINT.

Ok, so the effect is that if I killall ninja, ninja will then forward this to its children and it might quit faster and in a more orderly way, right?
That seems fine to me.
@pcc @tzik , any concerns about this?

Seems reasonable.

Cool, I'll merge this then.
@nicolasdespres Out of interest, in which cases do you run into this in practice?

In practice, I run into this when a the CI server running ninja shutdown or when I stop ninja from OSX Activity Monitor or htop (default is SIGTERM). I admit that if you always run ninja as a foreground process SIGINT is enough.
Thanks for the merge.

When Emacs kills subprocesses it uses SIGHUP for some reason, which in my experience breaks all sorts of apps.  Thanks for caring about details like this, Nicolas!

Emacs' compile mode emulates a "dumb" terminal. When you close the
compile-mode buffer, it closes the terminal which sends SIGHUP. When you
close an ssh connection or a terminal window SIGHUP is also sent.
I added a SIGHUP handler: https://github.com/martine/ninja/pull/964
On Tue, Apr 28, 2015 at 7:16 PM, Evan Martin<EMAIL_ADDRESS>wrote:

When Emacs kills subprocesses it uses SIGHUP for some reason, which in my
experience breaks all sorts of apps. Thanks for caring about details like
this, Nicolas!
—
Reply to this email directly or view it on GitHub
https://github.com/martine/ninja/pull/743#issuecomment-97141620.

--
Nicolas Desprès


Download Live Streaming video with RTMP format

Browser: Chrome 69
OS: Mac OS 10.11.1
Clappr Version: 0.2.25
Steps to reproduce:
I wish to allow a record button on my web page that shows live streaming video of my car dashcam. I want to allow "Start Recording" and "Stop Recording" functionality. How can I do so??

Hi @aroraayush this is repository is only about the video player clappr if you want to know how to stream from your dashcam to a server you might look at OBS or other encoder and you can find tons of good tutorials about that.

@leandromoreira Thanks for helping out. But, I want to add clappr to a webapge of mine and want to record and download the recorded bit on the client web page. OBS is a standalone app that I cannot request my users to download

@aroraayush maybe you can take a look at webrtc.


ni doesn't seem to run with the correct node/pnpm version

Describe the bug
I just installed ni the first time with pnpm i -g @antfu/ni then executed ni inside a project.
Unfortunately, I ran into an error due to our strict-engine setting:
ERR_PNPM_UNSUPPORTED_ENGINE  Unsupported environment (bad pnpm and/or Node.js version)

Your pnpm version is incompatible with "/home/<user>/Applications/Webdevelopment/Freelancing/<repo>/<project>".

Expected version: ^7.8.0
Got: 6.32.9

This is happening because the package's manifest has an engines.pnpm field specified.
To fix this issue, install the required pnpm version globally.

To install the latest version of pnpm, run "pnpm i -g pnpm".
To check your pnpm version, run "pnpm -v".

Your Node version is incompatible with "/home/<user>/Applications/Webdevelopment/Freelancing/<repo>/<project>".

Expected version: ^16.15.0
Got: v16.14.2

This is happening because the package's manifest has an engines.node field specified.
To fix this issue, install the required Node version.

But when I check for node/pnpm version, I get this output:
$ node -v
v16.16.0

$ pnpm -v
7.8.0

ni seems to pick up an old node version that I have installed with volta, see:
$ volta list node

⚡ Node runtimes in your toolchain:

    v16.14.2
    v16.15.0 (default)
    v16.16.0 (current @ /home/<user>/Applications/Webdevelopment/Freelancing/<repo>/<project>/
    package.json)

Where it gets pnpm v6.32.9 from, I have no clue.
Reproduction

System Info
System:
    OS: Linux 5.18 Manjaro Linux
    CPU: (8) x64 Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz
    Memory: 11.56 GB / 15.61 GB
    Container: Yes
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 16.16.0 - ~/.volta/tools/image/node/16.16.0/bin/node
    Yarn: 1.22.19 - ~/Applications/Webdevelopment/Freelancing/myZorro/myZorro/node_modules/.bin/yarn
    npm: 8.11.0 - ~/.volta/tools/image/node/16.16.0/bin/npm
  Browsers:
    Brave Browser: <IP_ADDRESS>
    Firefox: 103.0

Used Package Manager
pnpm
Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guide.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Check that this is a concrete bug. For Q&A, please open a GitHub Discussion instead.
[X] The provided reproduction is a minimal reproducible of the bug.

I don't know about Volta, PR welcome if you find a fix for it

I was actually able to fix this by removing volta completely and reinstalling right now. No idea why it happened though.
Now at least people can find this thread when they search for similar issues.


adding NI deviation to traceroute test

adding NI deviation to gnoi traceroute test

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 51.567%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
872

Relevant Lines:
1691

💛  - Coveralls


remove jobs repo as we use the jenkins pipeline branch plugin to disc…

…over Jenkinsfiles

[merge]


Automatic builds on DockerHub are failing

There appears to be a problem with the Julia installation when building the container:
failed running [/bin/sh -c apt install julia -y && apt-get clean]: exit code: 100

from https://hub.docker.com/repository/registry-1.docker.io/esmvalgroup/esmvaltool/builds/3457364a-0078-4bb2-8950-8af07622ae5e

this seems to be a rather common fail https://github.com/docker/buildx/issues/101 maybe we should enable debug logs?

just out of curiosity, why is the latest build shown as 2 months old, did we not build one with 2.3.0? https://hub.docker.com/r/esmvalgroup/esmvaltool/builds

If you want to have a go at fixing this, that would be most welcome. I can give you a short introduction if needed.

just out of curiosity, why is the latest build shown as 2 months old, did we not build one with 2.3.0? https://hub.docker.com/r/esmvalgroup/esmvaltool/builds

The automatic builds were disabled for a while when Docker changed its policy. They used to be free for everyone and now they're only free for approved open source projects. We're an approved open source project now, but it took a while to get there.

reopening since now the builds fail with a very conda-like issue https://hub.docker.com/repository/registry-1.docker.io/esmvalgroup/esmvaltool/builds/990cd58c-4bce-4d7d-9f84-eb2655fd5b3a

@valeriupredoi, could you add me to the dockerhub organization?

sure what's your DockerHub ID? (send it in an email/whatsapp, this is public)

I'll be a monkey's uncle, that mamba thingie really works, check out the PR :grin:


Add ianwalter/gotime

Adds https://github.com/ianwalter/gotime -- Thanks for the cool project!

thanks!


Feature: Native ZFS Storage Driver

[//]: # kind feature
Description
Is there planned support for a proper zfs driver for the container storage (or documentation on one that already exists)? If not, this would be a very nice feature to have. It seems it is something that Docker supports so there is at least a reference implementation in a similar infrastructure.
Describe the results you received:
output of podman --storage-driver fuse-overlay:
time="2018-11-27T00:07:34-05:00" level=error msg="Failed to GetDriver graph fuse-overlayfs /var/lib/containers/storage"
could not get runtime: failed to GetDriver graph fuse-overlayfs /var/lib/containers/storage: driver not supported

Describe the results you expected:
The vfs driver works, but I would have expected one of the overlay options to work or there to be another way to take advantage of the COW capabilities of zfs.
Additional information you deem important (e.g. issue happens only occasionally):
output of podman --storage-driver zfs info (didn't find any documentation but just for shits and giggles...):
could not get runtime: Unknown option overlay.mountopt

Output of podman version:
Version:       <IP_ADDRESS>
Go Version:    go1.11.2
Git Commit:    ""
Built:         Mon Nov 26 23:44:37 2018
OS/Arch:       linux/amd64

Output of podman info:
time="2018-11-27T00:05:13-05:00" level=error msg="'overlay' is not supported over zfs"
could not get runtime: 'overlay' is not supported over zfs: backing file system is unsupported for this graph driver

Additional environment details (AWS, VirtualBox, physical, etc.):
bare-metal arch install, zfs on root with separate /home, /var, and /opt as zfs datasets. The /var dataset has the zfs property overlay set to on.

OK - a bit more digging and tried a plain setup guessing at sane settings and following the error log. I guess you do have ZFS support in the graph driver. Nice! I mounted a dataset at /var/lib/containers/storage with overlay=on and had to comment out a couple lines in the /etc/storage.conf (as you may have guessed from the output I included as "additional information I deem important").  I have two related items then.
First, a question: Is there documentation of this feature anywhere? A grep of the podman man-page yields no results for "zfs|ZFS". The only indication I found that ZFS may be supported is a reference to the existence of a driver on this "practical introduction" from RHEL.
Also, a suggestion/request: I think there should have been a more elegant response to the command podman --storage-driver=zfs info given that there is (apparently) such a driver and that the /var filesystem was a zfs dataset. Is it possible that the error had to with the mountpoint /var/lib/containers/storage was not defined as a dataset itself and instead just part of the filesystem mounted at /var? If this is the case, an error message indicating as much would be helpful. If, however, the specific mountpoint of the host dataset containing the relevant filesystem is not a hard requirement, my next best guess at the root cause of the issue I saw is simply a choke in the config file. If the way that I indicated on the command line that I want to use a ZFS backend is not the cannonical way to override the complete set of config options required to successfully change backends, may you please point me to the documentation that describes the proper way to do this for more complex setups (or just let me know it does not yet exist)?
Thanks! Enjoying this tool and its cousins so far.

I do believe that c/storage does provide support for ZFS, but you'll likely
need to edit /etc/containers/storage.conf and manually set the storage
driver from overlay to ZFS.
On Tue, Nov 27, 2018, 00:14 Guy Marcus<EMAIL_ADDRESS>wrote:

[//]: # kind feature
Description
Is there planned support for a proper zfs driver for the container
storage (or documentation on one that already exists)? If not, this would
be a very nice feature to have. It seems it is something that Docker
supports so there is at least a reference implementation in a similar
infrastructure.
Describe the results you received:
output of podman --storage-driver fuse-overlay:
time="2018-11-27T00:07:34-05:00" level=error msg="Failed to GetDriver graph fuse-overlayfs /var/lib/containers/storage"
could not get runtime: failed to GetDriver graph fuse-overlayfs /var/lib/containers/storage: driver not supported
Describe the results you expected:
The vfs driver works, but I would have expected one of the overlay
options to work or there to be another way to take advantage of the COW
capabilities of zfs.
Additional information you deem important (e.g. issue happens only
occasionally):
output of podman --storage-driver zfs info (didn't find any documentation
but just for shits and giggles...):
could not get runtime: Unknown option overlay.mountopt
Output of podman version:
Version:       <IP_ADDRESS>
Go Version:    go1.11.2
Git Commit:    ""
Built:         Mon Nov 26 23:44:37 2018
OS/Arch:       linux/amd64
Output of podman info:
time="2018-11-27T00:05:13-05:00" level=error msg="'overlay' is not supported over zfs"
could not get runtime: 'overlay' is not supported over zfs: backing file system is unsupported for this graph driver
Additional environment details (AWS, VirtualBox, physical, etc.):
bare-metal arch install, zfs on root with separate /home, /var, and /opt
as zfs datasets. The /var dataset has the zfs property overlay set to on.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/containers/libpod/issues/1870, or mute the thread
https://github.com/notifications/unsubscribe-auth/AHYHCKAHgxCrkIq8PYXaIQNmY2mTLQCPks5uzMongaJpZM4Y0vsl
.

For point 1 - I think the proper place is probably the storage.conf manpage (and maybe noting in the manpages for Podman that storage options are sourced from storage.conf if not explicitly set, and valid values are the same). I can't see any listing of valid drivers right now, so I think we do need to add that.
For point 2 - you can set storage driver in /etc/containers/storage.conf and not need to pass the command line option again.

(Details on storage.conf can be found via man storage.conf, but it's a little sparse in there - could definitely use some improvement)

storage.conf is part defined here: https://github.com/containers/storage/blob/master/docs/containers-storage.conf.5.md
Open a PR with changes you would like to see.

@guygma zfs should work, but I don't know of anyone who has tried it out.

I think we have an action item here to update the c/storage storage.conf manpage to include valid storage drivers. I've opened https://github.com/containers/storage/pull/250 to handle that.

Thank you all for the clarification!  The zfs driver did end up working for me, so I agree it would be nice to have that made a bit more clear in the documentation. I didn't know to look in man storage.conf but indeed it does leave some breadcrumbs for the motivated. My case for getting that container up and running was in a VM so I haven't used anything long-term on ZFS yet, but all my bare-metal systems are ZFS on root now so next time I need a container, this will be put to the test again.

C/storage manpages have been updated, so I'm going to close. If you encounter any further issues with ZFS, feel free to repoen


Issue when adding  import * as admin from 'firebase-admin';

Hi
A few months ago I did a few Cloud function in TypeScript using Webpack. Was working just fine.
I've put this aside and now coming back to the project. Now, these cloud functions are no longer working.
Seem that Firebase added a new TypeScript option to create cloud functions.
I've downloaded the sample and followed the steps from this site.
First "npm run deploy" work fine.
I add another function without any really code in the function and run and deploy just fine.
At the moment I add:
import * as admin from 'firebase-admin';
admin.initializeApp(functions.config().firebase);
and try to run and deploy, i get errors.  Even if I remove these lines now I'm stock in an endless error. So I can no longer run and deploy.
Here is the error.
node_modules/firebase-functions/lib/providers/firestore.d.ts(17,29): error TS2694: Namespace 'admin.firestore' has no exported member 'DocumentReference'.
npm ERR! code ELIFECYCLE
npm ERR! errno 2
npm ERR! functions@ build: tsc
npm ERR! Exit status 2
npm ERR!
npm ERR! Failed at the functions@ build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR!      C:\Users\Benoit\AppData\Roaming\npm-cache_logs\2018-03-22T14_08_58_788Z-debug.log
Error: functions predeploy error: Command terminated with non-zero exit code2
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! functions@ deploy: firebase deploy --only functions
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the functions@ deploy script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR!      C:\Users\Benoit\AppData\Roaming\npm-cache_logs\2018-03-22T14_08_59_419Z-debug.log
Any idea what is the cause?
Thank you

npm install firebase-functions@latest firebase-admin@latest --save inside your functions folder

The biggest issue with this answer right now @Alb93 is that.

<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>but none is installed. You must install peer dependencies yourself.


Question about AppCompatActicity

I read the comments of the last issue.
However, MainActivity of Resource1 is extends Activity,
and MainActivity of Resource2 is extends AppCompatActivity.
And these things work well.
How is that possible?

We have 3 MainActivity in ClassLoader, Resource1 and resource2.
You could import the same dependencies such as com.android.support:appcompat-v7 in both Resource1 and Resource2, and it works well because the ClassLoader app runs them on two separate processes.
But if you try to import the same dependencies in ClassLoader app and whatever apks, this will lead to ClassCastException errors:
java.lang.ClassCastException: android.support.v7.widget.ContentFrameLayout cannot be cast to android.support.v7.widget.ContentFrameLayout
In Java, the way you compare objects is to check their names and class loaders. Those two classes have exactly the same name, but loading from different class loaders. That's why you can't have AppCompatActivity loaded in ClassLoader app and Resource app at the same time.

Hello,
I am interested in getting to find solution for this issue too. So, if I understand correctly, if main app would just use simple Activity, the plugin/resource could use AppCompatActivity. So far all my attempts failed with Android got confused with finding AppCompat resources. Trying to use constraint layout library failed too (they have their own R. resources).

Exactly.


Add a fetch example to the README

Some of our installation docs get their information from the official
example apps in other repositories. This change explains this and
provides an example of how to use the fetch script to pull in the
changes.
Follow up to #232

My pleasure. Thank you for your helpful in-person feedback and review @thompiler!


[4.0] Fix for wrapperAttributes Unexpected Behaviour

Like described in #2188 and #2190 somes fields are behaving unexpectably when setting wrapperAttributes on field.
In this specific 4 files we were checking if wrapperAttributes does not contain any class, otherwise don't load the javascript.
At first i thought it would be only to allow developers to override the init js function, but maybe we can provide an option to completely remove this initialization from fields.
I decided to not introduce any new field keys, instead give the option for developer to able to override OR delete.
1 - If this two keys are not present in field we load the defaults.
2 - If they are present we check if they have or not value.
2.1 - In case they have value, we replace the default with the one provided.
2.2 - In case it don't have value, $field['wrapperAttributes']['js-init-function'] = '';  explicitly the developer want to remove this feature and that's what we do.
~~There is always time for a change.
Also we should think in making all fields consistent in loading/not loading/replacing the init function. If this is the path to go, i would happily apply this changes to other fields.~~
Best,
Pedro

I updated this PR as per #2188 discussion.

Thank you @pxpm , I can confirm this fixes the problem at hand, in a way that can be merged right now. Just merged and tagged it as 4.0.20
But 😄 I think I just figured out an even cleaner way to do this 😆 So if we were to redo stuff for all the fields, instead of:
   if (!isset($field['wrapperAttributes']) || !isset($field['wrapperAttributes']['data-init-function'])){
        $field['wrapperAttributes']['data-init-function'] = 'bpFieldInitUploadElement';
    }

    if (!isset($field['wrapperAttributes']) || !isset($field['wrapperAttributes']['data-field-name'])) {
        $field['wrapperAttributes']['data-field-name'] = $field['name'];
    }

I think a cleaner way would be:
$defaultWrapperAttributes = [
   'data-init-function' => 'bpFieldInitUploadElement',
   'data-field-name' => $field['name'],
];
// merge default wrapper attributes with the ones the dev has given (if any)
// if an attribute has been provided by both, use the one the developer has given (aka overwrite the default)
$field['wrapperAttributes'] = array_merge($defaultWrapperAttributes, $field['wrapperAttributes'] ?? []);

The way I see it, this avoids all the if/else statements that make it seem bloated, it's easier to much read and much easier to add other defaults. All thanks to array_merge(). Don't know why I didn't think of this sooner 😄 What do you think about it @pxpm ? Anything I missed, or is this indeed the droid we've been looking for? 😄
Cheers!


[AIRFLOW-52] Fix bottlenecks when working with many tasks

Dear Airflow Maintainers,
Please accept this PR that addresses the following issues:

https://issues.apache.org/jira/browse/AIRFLOW-52

@aoen please try this.
Fixes:

replace two slow list comprehensions with much faster hash lookups (using the new task_dict)
dag hash function tried (and failed) to hash the list of tasks, then fell back on repr-ing the list, which took forever. Instead, hash tuple(task_dict.keys()).

Tested this and it works, thank you! LGTM


[Carbondata-3883] Added filtering for the deleted rows for local dictionary fields

Why is this PR needed?
In Presto integration when reading records for table with string fields and local dictionary enabled gives improper result.
What changes were proposed in this PR?
Added a CarbondataColumnVectorWrapper for the dictionary vector for column for which internally create dictionary vector.
Wrapper is used to filter the deleted records.
Does this PR introduce any user interface change?

No

Is any new testcase added?

No

Can one of the admins verify this patch?

add to whitelist

Build Success with Spark 2.3.4, Please check CI http://<IP_ADDRESS>:12545/job/ApacheCarbonPRBuilder2.3/3281/

Build Success with Spark 2.4.5, Please check CI http://<IP_ADDRESS>:12545/job/ApacheCarbon_PR_Builder_2.4.5/1544/

@Nitin-Kashyap : Can you please add a test here in PrestoAllDataTypeLocalDictTest.scala for this scenario. May be just add one more query for existing table local dictionary column that can cause this problem.

@Nitin-Kashyap : Can you please add a test here in PrestoAllDataTypeLocalDictTest.scala for this scenario. May be just add one more query for existing table local dictionary column that can cause this problem.

Since the test step required me to insert and delete record using Spark-Integration; The same feature is unavailable in Presto-Integration; I'm not sure how to add the same in PrestoAllDataTypeLocalDictTest.scala.
Please suggest, I would be happy to add!

Since the test step required me to insert and delete record using Spark-Integration; The same feature is unavailable in Presto-Integration; I'm not sure how to add the same in PrestoAllDataTypeLocalDictTest.scala.

Please suggest, I would be happy to add!
we currently manually create store as presto module is independent of spark.
Need add spark dependency in test module to create store. We can take this as separate activity.

LGTM. Thanks for the contribution !


Passing sigils to Ecto.Query.from's select option produces subtly different query and nonsense results

Environment

Elixir version (elixir -v): Elixir 1.12.3 (compiled with Erlang/OTP 24)
Database and version (PostgreSQL 9.4, MongoDB 3.2, etc.): postgres (PostgreSQL) 14.1
Ecto version (mix deps): ecto 3.7.1
Database adapter and version (mix deps): postgrex 0.15.13
Operating system: macOS Catalina 10.15.7

Current behavior
Passing in a words-sigil with an atom modifier to the select option produces a different result than passing in an identical list with no syntactic sugar, and it is a very subtle and hard-to-find bug.
Specifically, the queries from a in "artists", select: [:id, :name] and from a in "artists", select: ~w(id name)a both produce similar-looking queries via the Inspect protocol (i.e. #Ecto.Query<from a0 in "artists", select: [:id, :name]>). However, when run on the repository, they produce completely different results. More specifically, running Repo.all/1 on the former in my test database results in [%{id: 1, name: "Miles Davis"}] and the latter results in [[:id, :name]].
Expected behavior
from a in "artists", select: [:id, :name] and from a in "artists", select: ~w(id name)a should return identical queries since [:id, :name] == ~w(id name)a.

I was able to reproduce this behavior (shown below). I think the root cause is you are passing a ~w sigil macro into another macro, Ecto.Query.from. Macros receive the representation of code instead of the results of code (see near the end of the intro section here. quote do: ~w(a b)a shows a different representation to quote do: [:a, :b].
If you assign the sigil to a variable and then pin that variable to the query, it will work as desired (see below).
The first query is standard with expected output. The second using the sigil with unexpected output. The third uses a pinned variable of the sigil with expected output.
iex(61)> q = Ecto.Query.from Artist, select: [:id, :name]
#Ecto.Query<from a0 in Friends.Artist, select: [:id, :name]>
iex(62)> q |> Repo.all

13:57:16.130 [debug] QUERY OK source="artists" db=0.6ms idle=1586.3ms
SELECT a0."id", a0."name" FROM "artists" AS a0 []
[
  %Friends.Artist{
    __meta__: #Ecto.Schema.Metadata<:loaded, "artists">,
    id: 1,
    name: "alice"
  },
  %Friends.Artist{
    __meta__: #Ecto.Schema.Metadata<:loaded, "artists">,
    id: 2,
    name: "bob"
  }
]

iex(63)> q = Ecto.Query.from Artist, select: ~w(id name)a
#Ecto.Query<from a0 in Friends.Artist, select: [:id, :name]>
iex(64)> q |> Repo.all

13:57:27.459 [debug] QUERY OK source="artists" db=0.2ms queue=0.4ms idle=1915.2ms
SELECT TRUE FROM "artists" AS a0 []
[[:id, :name], [:id, :name]]

iex(65)> s = ~w(id name)a
[:id, :name]
iex(66)> q = Ecto.Query.from Artist, select: ^s
#Ecto.Query<from a0 in Friends.Artist, select: [:id, :name]>
iex(67)> q |> Repo.all

13:57:51.867 [debug] QUERY OK source="artists" db=1.4ms idle=1322.6ms
SELECT a0."id", a0."name" FROM "artists" AS a0 []
[
  %Friends.Artist{
    __meta__: #Ecto.Schema.Metadata<:loaded, "artists">,
    id: 1,
    name: "alice"
  },
  %Friends.Artist{
    __meta__: #Ecto.Schema.Metadata<:loaded, "artists">,
    id: 2,
    name: "bob"
  }
]

related to #2958

Thanks for the very detailed explanation, @aegatlin!
This discrepancy in behaviour is still a bug, though. I opened a PR that fixes it, but until it gets released, the pin workaround should work perfectly :)


Pipeline Parallel Ingest (CSVFileScanFragment operator)

Currently, parallel ingest is only accessible through a REST call to the coordinator. The coordinator builds a query plan and sends a CSVFileScanFragment with the appropriate byte ranges to each worker.
We somehow need to introduce parallel ingest to MyriaL. The tricky part will be to figure out how to pipeline this operator. Should Raco be the one figuring out how to split the byte ranges? I think we discussed this before and we preferred not to introduce any AWS/S3 API to Raco.

tagging @senderista

I thought an alternative we discussed in the past was to have Raco encode a pseudo-operator run at the coordinator which would be initialized only with the S3 URL and would dynamically create and dispatch the local operators (i.e., CSVFileScanFragment initialized with the appropriate ranges)? Are there other examples of operators which create an operator tree at runtime?
adding @jingjingwang

I vaguely remember talking about this. I do remember that we agreed to give Raco only the S3 URL, but I'm not familiar with how it can create the plan from there or whether there is another existing operator that does something like this.

So would initializing the operator at runtime within MyriaX be the way to go? I ran into a situation where I need to use this feature, so I'm trying to tackle it at the moment.
So here was my thinking...

implement a fake operator on the Raco side
on the MyriaX side, implement the corresponding encoding for the fake operator
when initializing the operator, we need to make sure it is initialized with the correct byte ranges. This almost seems like it needs to be worker independent. The problem here is that when we initialize the operator, it does not know any information about the worker at this point.

We could make it so the fake operator encoding has information about the worker id (Raco could probably take care of this?). That way, when we initialize the operator, we have the worker id from the encoding. We can then create a new CSVFileScanFragment constructor that can take the worker id parameter and initialize the constructor with the correct byte ranges.
Not sure if this would be the best approach.

FWIW I'm doing something vaguely similar to this right now in this PR: https://github.com/uwescience/myria/pull/858. One question is whether we can assume that all worker IDs are contiguous. If so, then each worker operator instance could initialize its own byte range (taking the minimum partition size into account) after it determines the object size, but we probably can't assume this if we want to be fault-tolerant.
I think we can dispense with the assumption of contiguous worker IDs as long as we ensure that all workers share the same live-workers view. We should be able to do this in OperatorEncoding.construct() by querying the server and initializing a private serializable field (see the above PR for an example). With that information, the minimum partition size, and the object's size in bytes, the workers should be able to determine which byte range they're responsible for without coordination.
(FYI, an Operator can get its worker ID with ((WorkerSubQuery) getLocalSubQuery()).getWorker().getID().)

This sounds good, thanks! I'll try it

Sorry, I think I'm still missing something. So I'm adding this logic under the CSVFileScanFragmentEncoding.construct() method, but I don't see how we can call getLocalSubQuery since this requires the operator object. Basically from the operator encoding, is there a way to get the operator object itself?

What I meant was to pass the live-workers view to the Operator constructor by calling args.getServer().getAliveWorkers() from within OperatorEncoding.construct(), then calling getLocalSubQuery() from Operator.init() or elsewhere. The Operator instance returned from OperatorEncoding.construct() is only used for serialization (so any field you initialize in the constructor called from construct() must be serializable), while Operator.init() is only called on the "real" Operator instance when it is deserialized on the worker where it runs.


Reducing overhead for payload encoding

The payload field is currently defined as base64 encoded data, which is a reasonable choice for holding arbitrary data.
However, when the payload content is already a well-formed text string, the 33% size increase induced by the base64 encoding starts to feel a bit costly (See https://github.com/in-toto/attestation/issues/289).
Could DSSE offer an "unencoded" mode, where users can directly put raw text string in the payload?
Or are there other alternatives / recommendations?

IMO this needs to be fleshed out more.

For real-world data, how much better is UTF8 vs base64?
What is a real-world example where this extra size overhead matters? The linked issue mentions "some complex artifacts may generate 1GB provenance data" but doesn't provide details. I feel that if something is generating a 1GB SLSA Provenance, it's doing it wrong. (Then again, a 1GB SBOM might make sense.) An alternate would be to have the big payload "detached", which has other trade-offs.
How does the proposal compare to compression under the following scenarios?

Compressing the whole DSSE as-is (base64)
Compressing the whole DSSE using the proposal (UTF8)
Compressing before base64 (which is invasive since the payloadType would change)

Oh good call!
As another alternative, the bundle format selected by in-toto, JSON lines, also offer a compression mode.

Cc @dstufft who has an interest in this subject and has run a bunch of experiments

These are my unedited notes when I was looking into this previously for another project:

I’ve only skimmed the DSSE repository, but it looks like using DSSE turns all of the TUF metadata “opaque” anyways, by wrapping the signed data into a base64 blob that gets stuck in an envelope (could be json, could be something else).
If that’s the case, there’s very little reason to stick with JSON, since the primary benefit is human readability, and we should definitely explore more compact options, typically binary options. We could try CBOR, Protobuf, Smile, BSON, MessagePack, or Ion and see what kind of results we get.
We could also consider compression here as well.

TUF is defined in a serialization independent format, which allows particular applications to select the serialization format that makes the most sense for them, so we need to settle on a serialization format that makes sense for us.
Our Constraints and Goals:

Serialized size should be as small as reasonably possible.

Deserialization must be both memory and CPU efficient.

Deserialization must be able to be done in pure Python or using the standard library.

For convenience, we are ignoring how signatures themselves are constructed for this.

We cannot assume that we have control over the deployed version of either the serializer or the deserializer, and we need to maintain as much forwards and backwards compatibility as possible.

We can consider compression of the serialized content.
Options considered:

“Canonical JSON”, as the format initially used by TUF.

Pro: Can be implemented using nothing but the standard library.
Pro: Fully human readable.
Pro: No schema to synchronize between consumers and producers.
Con: Uses a canonicalization scheme for signing, which is more fragile than traditional signing envelopes.

DSSE + JSON

Pro: Can be implemented using nothing but the standard library.
Pro: No schema to synchronize between consumers and producers.
Pro: Traditional signing envelope that is misuse resistant.
Con: Envelope treats the inner payload as binary, which makes the non envelope content opaque until after opening the envelope.

DSSE + MessagePack

Pro: No schema to synchronize between consumers and producers.
Pro: Traditional signing envelope that is misuse resistant.
Pro: Binary serialization, can represent binary values without using base64 encoding.
Con: Requires a library with a C extension for speed, but it does have a Pure Python fallback available.

DSSE + Ion:

Pro: No schema to synchronize between consumers and producers.
Pro: Traditional signing envelope that is misuse resistant.
Pro: Binary and Text serialization are both available, allowing tuning between human readable or binary compactness.
Con: Requires a library with a C extension for speed, but it does have a pure Python fallback.
Con: Packaging appears to be fetching the library by downloading it from an URL in the setup.py.

DSSE + CBOR:

Pro: No schema to synchronize between consumers and producers.
Pro: Traditional signing envelope that is misuse resistant.
Pro: Binary serialization, can represent binary values without using base64 encoding.
Con: Requires a library with a C extension for speed, but it does have a Pure Python fallback available.

DSSE + Protobuf

Pro: Traditional signing envelope that is misuse resistant.
Pro: Binary serialization, can represent binary values without using base64 encoding.
Pro: Able to be backwards and forwards compatible.
Con: Compatibility requires being somewhat careful when evolving the schema.
Con: Requires distributing a schema (in the form of a .proto file) to producers and consumers.
Con: Requires a library with a C extension for speed, but it does have a Pure Python fallback available.

I was specifically looking at TUF, and I was looking primarily at the snapshot role since that role will almost certainly be the largest file in my particular use case (TUF on PyPI), and I created a sort of torture test with ~500k delegations.  I've left the signatures themselves encoded as base64 in this case since they are proper binary data (though it's possible there's a more compact encoding that could be used than base64, but there's only a few signatures so it's pretty low value to find a better serialization for the signatures).
As you can see, the current DSSE serialization, when used from JSON ends up producing a 101M TUF snapshot, but if you compress that it drops down to 36-45M. Without compression the ion binary encoding is the smallest at 55M, it's also the smallest compressed by by a much smaller margin.
The proposed utf8 encoding reduces the snapshot role from 101M down to 81M, and likewise there is a decrease in compressed size as well.
Sorry for the brain dump, but hopefully this is useful in some way?

Oh one other note, is that a possibly interesting property here is that the proposed JSON + UTF8 encoding, when compressed, isn't the absolute smallest in the snapshot test, but it's closer than the current JSON + Base64 is, and on the smaller root role test, it's basically tied for the smallest.
To me, that ends up representing a really nice trade off, because the other serialization options, while generally available, are not nearly as ubiquitous as JSON as, and if you're worried about space constraints, the fact that wrapping the entire thing in any of the very ubiquitous compression algorithms brings this JSON + UTF8 encoding scheme in line with the smallest of the other options is a strong incentive to use it.
One thing of note, is that my tests above assume you're using the same serialization scheme for both the payload and the DSSE envelope, but you're only compressing the final output of the DSSE. Arguably you might want to compress just the payload since it's bound to be the largest part of the DSSE output, and that would mean that you can validate the signatures prior to decompressing (which decompressing first should be safe in DSSE I think, but I'm always nervous around cryptography + compression).

Excellent, thank you @dstufft! Before we decide on anything here, I'd love to see: (1) requirements, (2) constraints, and (3) numbers (like Donald produced) to back up findings.

Arguably you might want to compress just the payload since it's bound to be the largest part of the DSSE output

@dstufft: maybe this is a bit out-of-scope, strictly for DSSE, but I described a rationale that whole DSSE compression (and deferred to the upper-level context) could practically work out better than payload-only compression in the in-toto thread: in-toto has selected JSON line as the attestation bundle format, and in a bundle there will likely be multiple attestations of various kind, all for the same set of artifacts. So every in-toto statement "subject" array will contain a copy of identical content. (And for some builds that generate 10s of thousands files, the "subject" array constitutes the major bulk of the attestation size.)
If we compress the payload, or compress the DSSE envelope piece-wise:

The compression will not have the context/visibility of the "subject" duplication of other envelope payloads;
The resulting compressed data may be scrambled such that the "subject" duplication is no longer evident at the bundle-level, hence deterring bundle-level compression from gaining effective reduction.

If we were to embrace bundle level compression, I think the best DSSE can do is to preserve the original data as much as possible. To that end, UTF8 encoding works better than Base64 because there is less "clobbering".
For example, there are two attestations in a bundle, both for the same set of two files:

The first attestation lists the subject:[{"name":"main", "digest":{"sha256":"1234abcd...."}},
 {"name":"aux_dat.xml", "digest":{"sha256":"cdef5678...."}}]`

The second attestation lists the subject (in the other order):[{"name":"aux_dat.xml", "digest":{"sha256":"cdef5678...."}},
 {"name":"main", "digest":{"sha256":"1234abcd...."}}]`

UTF8 encoding will allow a compression algorithm to easily discover the duplicated information; However, after Base64 encoding, the data will look like:

The first attestationW3sibmFtZSI6Im
1haW4iLCAiZGlnZXN0Ijp7InNoYTI1NiI6IjEyMzRhYmNkLi4uLiJ9fSx7Im5hbWUiOiJhdXhfZGF0LnhtbC
IsICJkaWdlc3QiOnsic2hhMjU2IjoiY2RlZjU2NzguLi4uIn19XQo=

The second attestationW3sibmFtZSI6Im
F1eF9kYXQueG1sIiwgImRpZ2VzdCI6eyJzaGEyNTYiOiJjZGVmNTY3OC4uLi4ifX0seyJuYW1lIjoibWFpbi
IsICJkaWdlc3QiOnsic2hhMjU2IjoiMTIzNGFiY2QuLi4uIn19XQo=

It would take a really smart compression algorithm to tell that the middle parts actually contain the same information.
(I did a quick test throwing the original subjects and the base64 encoded ones at gzip, xz, and lzma, in all cases the compress base64 is ~2x the size of compressed original, indicating those algorithm failed to detect the underlying redundancy -- probably not a very realistic test, just to illustrate the problem 😄 ).

@AdamZWu I would not conflate in-toto Bundle-level compression with in-toto Attestation-level compression, so if you're looking for the former, I argue that it is out of the scope of this project.

@trishankatdatadog given interaction between DSSE and in-toto, so it is not completely out of scope.
Since in-toto attestation (and bundle) is a major applicator of DSSE, I think supporting in-toto to efficiently reduce bundle size is to both specs' best interest. And for that, my current thinking is to add a "utf-8 encoding" mode for payload, which allows minimum "clobbering" for payload that is already in utf-8 encoding (which is always the case for in-toto).
Doing so would allow bundle level compression to more effectively discover data duplication across multiple envelopes, which is one of the big sources of bloats in an attestation bundle.

OK, I'm fairly convinced by the fact that the compressed UTF-8-encoded JSON is only ~5% larger than the compressed proto encoding, while compressed base64-encoded JSON is ~20-80% larger than compressed proto.
root.dsse.json.gz    2.3k  (~75% overhead)
root.dsse.jsont.gz   1.4k  (~5% overhead)
root.dsse.proto.gz   1.3k

snapshot.dsse.json.gz   45M  (~20% overhead)
snapshot.dsse.jsont.gz  37M  (~0% overhead)
snapshot.dsse.proto.gz  37M

It would still be nice to gather a larger corpus of real-world data (not just two files) and do the comparison, but assuming it holds, then that's fairly compelling.
I'm assuming we'd add a new field like this?
message Envelope {
  // Message to be signed.
  // REQUIRED.
  oneof payload_encoding {
    // Raw bytes. In JSON, this is encoded as base64.
    bytes payload = 1;
    // Unicode string, where the signed byte stream is the UTF-8 encoding. In JSON, this is a regular unicode string.
    string payloadUtf8 = 4;
  }
}

Note: The signature algorithm does not change at all, and existing signatures could be re-encoded with this new field without invalidating them.

Adam, do you want to put together a PR that implements this? I think you'd need to edit the proto, the envelope.md, and other references to the payload. (I don't think protocol needs to be updated.)
The other question here is backwards compatibility. Old consumers won't be able to DSSEs with the new field. I don't see any security concern, but I don't know how best to roll this out.

Sounds good. I will put something up for review next week. :D

Doing so would allow bundle level compression to more effectively discover data duplication across multiple envelopes (see my above post), which is one of the big sources of bloats in an attestation bundle.

Adam, are you arguing that Attestation-level compression will automatically help with Bundle-level compression? If so, then yes, I agree. However, the two levels of compression are distinct from each other.

Doing so would allow bundle level compression to more effectively discover data duplication across multiple envelopes (see my above post), which is one of the big sources of bloats in an attestation bundle.

Adam, are you arguing that Attestation-level compression will automatically help with Bundle-level compression? If so, then yes, I agree. However, the two levels of compression are distinct from each other.

Not exactly. Yes, what DSSE does to payload will definitely affect bundle-level compression. But it looks to me the more complex processing DSSE does, probably the worse the bundle-level compression will perform, because complex mutations will hide cross-envelope data duplication. So I think DSSE could offer a mode that does less, e.g. allowing UTF-8 encoding for payload, so that JSON serialized in-toto statement which is already in UTF-8 can be presented pretty much unchanged (except for JSON string escapes).
And yes, DSSE's payload encoding is completely orthogonal to in-toto attestation bundle compression. Just some encoding (e.g. UTF-8) is much friendlier to bundle-level compression than the other (e.g. base64).

Yes, what DSSE does to payload will definitely affect bundle-level compression performance. But it looks to me the more complex processing DSSE does, probably the worse the bundle-level compression will perform, because complex mutations will hide cross-envelope data duplication. So I think DSSE could offer a mode that does less, e.g. allowing UTF-8 encoding for payload, so that JSON serialized in-toto statement which is already in UTF-8 can be presented pretty much unchanged (except for JSON string escapes).

Oh, I see, thanks for the clarification! Hmm, now I'm curious about how the Attestation-level choice of payload encoding would affect the efficacy of Bundle-level compression. As you suggest, UTF-8 should work better for this.


chore(master): release 1.19.5

:robot: I have created a release beep boop
1.19.5 (2023-09-07)
Bug Fixes

修复全屏图标显示 (cd6d170)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/KYX1234/Element-Admin/releases/tag/v1.19.5 :sunflower:


Added Post/Thread CRUD

all basic crud endpoints are tested and validated.
in createPost, I commented out the image uploading for now. This is beacause of the assumption that posts do not need a required image.

Posts can have optional images associated with it.
Added createVerifiedPosts for posts created by organizations. Does not have a "verifiedPostStatus". Discussed with JC regarding this, will take it up with Aarsh.
Did not test image functionality, however code logic is similar to JC's event and org controllers.