Decompile of Binary Giving Strange Output

cmpdylib.zip
^
This executable came from the Mac, it's a command from /usr/bin/cmpdylib.
When I decompiled it, it seemed to be successful:
Users-MBP:bin user$ python3 retdec-decompiler.py /Users/user/Downloads/retdec_output/cmpdylib/cmpdylib
##### Checking if file is a Mach-O Universal static library...

##### Checking if file is an archive...
RUN: /Users/user/Downloads/retdec/bin/retdec-ar-extractor /Users/user/Downloads/retdec_output/cmpdylib/cmpdylib --arch-magic
Not an archive, going to the next step.

##### Gathering file information...
RUN: /Users/user/Downloads/retdec/bin/retdec-fileinfo -c /Users/user/Downloads/retdec_output/cmpdylib/cmpdylib.json --similarity /Users/user/Downloads/retdec_output/cmpdylib/cmpdylib --no-hashes=all --crypto /Users/user/Downloads/retdec/bin/../share/retdec/support/generic/yara_patterns/signsrch/signsrch.yara --crypto /Users/user/Downloads/retdec/bin/../share/retdec/support/generic/yara_patterns/signsrch/signsrch.yarac --max-memory-half-ram
Input file               : /Users/user/Downloads/retdec_output/cmpdylib/cmpdylib
File format              : Mach-O
File class               : 64-bit
File type                : Executable file
Architecture             : x86-64
Endianness               : Little endian
Entry point address      : 0x100000f73
Entry point offset       : 0xf73
Entry point section name : __text
Entry point section index: 0
Bytes on entry point     : 554889e58d47ff488d5608488d3d2900000031c989c6e800000000ff257c0000004c8d1d6d0000004153ff255d0000009068
Detected tool            : gc (compiler), 29 from 160 significant nibbles (18.125%)

##### Trying to unpack /Users/user/Downloads/retdec_output/cmpdylib/cmpdylib into /Users/user/Downloads/retdec_output/cmpdylib/cmpdylib-unpacked.tmp by using generic unpacker...
RUN: /Users/user/Downloads/retdec/bin/retdec-unpacker /Users/user/Downloads/retdec_output/cmpdylib/cmpdylib -o /Users/user/Downloads/retdec_output/cmpdylib/cmpdylib-unpacked.tmp --max-memory-half-ram
No matching plugins found for 'gc'
##### Unpacking by using generic unpacker: nothing to do
##### 'upx' not available: nothing to do

##### Decompiling /Users/user/Downloads/retdec_output/cmpdylib/cmpdylib into /Users/user/Downloads/retdec_output/cmpdylib/cmpdylib.bc...
RUN: /Users/user/Downloads/retdec/bin/retdec-bin2llvmir -provider-init -decoder -verify -x87-fpu -main-detection -idioms-libgcc -inst-opt -cond-branch-opt -syscalls -stack -constants -param-return -local-vars -inst-opt -simple-types -generate-dsm -remove-asm-instrs -class-hierarchy -select-fncs -unreachable-funcs -inst-opt -x86-addr-spaces -value-protect -instcombine -tbaa -targetlibinfo -basicaa -domtree -simplifycfg -domtree -early-cse -lower-expect -targetlibinfo -tbaa -basicaa -globalopt -mem2reg -instcombine -simplifycfg -basiccg -domtree -early-cse -lazy-value-info -jump-threading -correlated-propagation -simplifycfg -instcombine -simplifycfg -reassociate -domtree -loops -loop-simplify -lcssa -loop-rotate -licm -lcssa -instcombine -scalar-evolution -loop-simplifycfg -loop-simplify -aa -loop-accesses -loop-load-elim -lcssa -indvars -loop-idiom -loop-deletion -memdep -gvn -memdep -sccp -instcombine -lazy-value-info -jump-threading -correlated-propagation -domtree -memdep -dse -dce -bdce -adce -die -simplifycfg -instcombine -strip-dead-prototypes -globaldce -constmerge -constprop -instnamer -domtree -instcombine -instcombine -tbaa -targetlibinfo -basicaa -domtree -simplifycfg -domtree -early-cse -lower-expect -targetlibinfo -tbaa -basicaa -globalopt -mem2reg -instcombine -simplifycfg -basiccg -domtree -early-cse -lazy-value-info -jump-threading -correlated-propagation -simplifycfg -instcombine -simplifycfg -reassociate -domtree -loops -loop-simplify -lcssa -loop-rotate -licm -lcssa -instcombine -scalar-evolution -loop-simplifycfg -loop-simplify -aa -loop-accesses -loop-load-elim -lcssa -indvars -loop-idiom -loop-deletion -memdep -gvn -memdep -sccp -instcombine -lazy-value-info -jump-threading -correlated-propagation -domtree -memdep -dse -dce -bdce -adce -die -simplifycfg -instcombine -strip-dead-prototypes -globaldce -constmerge -constprop -instnamer -domtree -instcombine -inst-opt -simple-types -stack-ptr-op-remove -idioms -global-to-local -dead-global-assign -instcombine -inst-opt -idioms -phi2seq -value-protect -disable-inlining -disable-simplify-libcalls -config-path /Users/user/Downloads/retdec_output/cmpdylib/cmpdylib.json -max-memory-half-ram -o /Users/user/Downloads/retdec_output/cmpdylib/cmpdylib.bc
Running phase: Initialization ( 0.02s )
Running phase: LLVM ( 0.03s )
Running phase: Providers initialization ( 0.03s )
Running phase: Input binary to LLVM IR decoding ( 0.08s )
Running phase: LLVM ( 0.08s )
Running phase: x87 fpu register analysis ( 0.08s )
Running phase: Main function identification optimization ( 0.08s )
Running phase: Libgcc idioms optimization ( 0.08s )
Running phase: LLVM instruction optimization ( 0.08s )
Running phase: Conditional branch optimization ( 0.08s )
Running phase: Syscalls optimization ( 0.08s )
Running phase: Stack optimization ( 0.08s )
Running phase: Constants optimization ( 0.08s )
Running phase: Function parameters and returns optimization ( 0.08s )
Running phase: Register localization optimization ( 0.08s )
Running phase: LLVM instruction optimization ( 0.08s )
Running phase: Simple types recovery optimization ( 0.08s )
Running phase: Disassembly generation ( 0.08s )
Running phase: Assembly mapping instruction removal ( 0.09s )
Running phase: C++ class hierarchy optimization ( 0.09s )
Running phase: Selected functions optimization ( 0.09s )
Running phase: Unreachable functions optimization ( 0.09s )
Running phase: LLVM instruction optimization ( 0.09s )
Running phase: x86 address spaces optimization ( 0.09s )
Running phase: Value protection optimization ( 0.09s )
Running phase: LLVM ( 0.09s )
Running phase: LLVM instruction optimization ( 0.10s )
Running phase: Simple types recovery optimization ( 0.10s )
Running phase: Stack pointer operations optimization ( 0.10s )
Running phase: Instruction idioms optimization ( 0.10s )
Running phase: Global to local optimization ( 0.10s )
Running phase: Dead global assign optimization ( 0.10s )
Running phase: LLVM ( 0.10s )
Running phase: LLVM instruction optimization ( 0.10s )
Running phase: Instruction idioms optimization ( 0.10s )
Running phase: Phi2Seq optimization ( 0.10s )
Running phase: Value protection optimization ( 0.10s )
Running phase: LLVM ( 0.10s )
Running phase: Bitcode Writer ( 0.10s )
Running phase: Assembly Writer ( 0.10s )
Running phase: Cleanup ( 0.10s )

##### Decompiling /Users/user/Downloads/retdec_output/cmpdylib/cmpdylib.bc into /Users/user/Downloads/retdec_output/cmpdylib/cmpdylib.c...
RUN: /Users/user/Downloads/retdec/bin/retdec-llvmir2hll -target-hll=c -var-renamer=readable -var-name-gen=fruit -var-name-gen-prefix= -call-info-obtainer=optim -arithm-expr-evaluator=c -validate-module -o /Users/user/Downloads/retdec_output/cmpdylib/cmpdylib.c /Users/user/Downloads/retdec_output/cmpdylib/cmpdylib.bc -enable-debug -emit-debug-comments -config-path=/Users/user/Downloads/retdec_output/cmpdylib/cmpdylib.json -max-memory-half-ram
Running phase: initialization ( 0.04s )
 -> creating the used HLL writer [c] ( 0.04s )
 -> creating the used alias analysis [simple] ( 0.04s )
 -> creating the used call info obtainer [optim] ( 0.04s )
 -> creating the used evaluator of arithmetical expressions [c] ( 0.04s )
 -> creating the used variable names generator [fruit] ( 0.04s )
 -> creating the used variable renamer [readable] ( 0.04s )
 -> creating the used semantics [libc,gcc-general,win-api] ( 0.04s )
 -> loading the input config ( 0.04s )
Running phase: conversion of LLVM IR into BIR ( 0.04s )
 -> converting global variables ( 0.05s )
 -> converting function main ( 0.05s )
Running phase: removing functions prefixed with [__decompiler_undefined_function_] ( 0.05s )
Running phase: removing functions from standard libraries ( 0.05s )
Running phase: removing code that is not reachable in a CFG ( 0.05s )
Running phase: signed/unsigned types fixing ( 0.05s )
Running phase: converting LLVM intrinsic functions to standard functions ( 0.05s )
Running phase: obtaining debug information ( 0.05s )
Running phase: alias analysis [simple] ( 0.05s )
Running phase: optimizations [normal] ( 0.05s )
 -> running GotoStmtOptimizer ( 0.05s )
 -> running RemoveUselessCastsOptimizer ( 0.05s )
 -> running UnusedGlobalVarOptimizer ( 0.05s )
 -> running DeadLocalAssignOptimizer ( 0.05s )
 -> running SimpleCopyPropagationOptimizer ( 0.05s )
 -> running CopyPropagationOptimizer ( 0.05s )
 -> running AuxiliaryVariablesOptimizer ( 0.05s )
 -> running SimplifyArithmExprOptimizer ( 0.05s )
 -> running IfStructureOptimizer ( 0.05s )
 -> running LoopLastContinueOptimizer ( 0.05s )
 -> running PreWhileTrueLoopConvOptimizer ( 0.05s )
 -> running WhileTrueToForLoopOptimizer ( 0.05s )
 -> running WhileTrueToWhileCondOptimizer ( 0.05s )
 -> running IfBeforeLoopOptimizer ( 0.05s )
 -> running LLVMIntrinsicsOptimizer ( 0.05s )
 -> running VoidReturnOptimizer ( 0.05s )
 -> running BreakContinueReturnOptimizer ( 0.05s )
 -> running BitShiftOptimizer ( 0.05s )
 -> running DerefAddressOptimizer ( 0.05s )
 -> running EmptyArrayToStringOptimizer ( 0.05s )
 -> running BitOpToLogOpOptimizer ( 0.05s )
 -> running SimplifyArithmExprOptimizer ( 0.05s )
 -> running UnusedGlobalVarOptimizer ( 0.05s )
 -> running DeadLocalAssignOptimizer ( 0.05s )
 -> running SimpleCopyPropagationOptimizer ( 0.05s )
 -> running CopyPropagationOptimizer ( 0.05s )
 -> running SelfAssignOptimizer ( 0.05s )
 -> running VarDefForLoopOptimizer ( 0.05s )
 -> running VarDefStmtOptimizer ( 0.05s )
 -> running EmptyStmtOptimizer ( 0.05s )
 -> running GotoStmtOptimizer ( 0.05s )
 -> running SimplifyArithmExprOptimizer ( 0.05s )
 -> running DeadCodeOptimizer ( 0.05s )
 -> running DerefToArrayIndexOptimizer ( 0.05s )
 -> running IfToSwitchOptimizer ( 0.05s )
 -> running CCastOptimizer ( 0.05s )
 -> running CArrayArgOptimizer ( 0.05s )
Running phase: variable renaming [readable] ( 0.05s )
Running phase: converting constants to symbolic names ( 0.05s )
Running phase: module validation ( 0.05s )
 -> running BreakOutsideLoopValidator ( 0.05s )
 -> running NoGlobalVarDefValidator ( 0.05s )
 -> running ReturnValidator ( 0.05s )
Running phase: emission of the target code [c] ( 0.05s )
Running phase: finalization ( 0.05s )
Running phase: cleanup ( 0.06s )

##### Done!

However, when I opened the .c file, I saw that there was barely any code in the main function:
//
// This file was generated by the Retargetable Decompiler
// Website: https://retdec.com
// Copyright (c) 2019 Retargetable Decompiler<EMAIL_ADDRESS>//

#include <stdint.h>

// ------------------------ Functions -------------------------

// Address range: 0x100000f73 - 0x100000f8e
int main(int argc, char ** argv) {
    // 0x100000f73
    return argc - 1;
}

// --------------------- Meta-Information ---------------------

// Detected compiler/packer: gc
// Detected functions: 1
// Decompilation date: 2019-05-03 23:32:35

If you man cmpdylib, the executable is supposed to do much more than that:
NAME
       cmpdylib - compare two dynamic shared libraries for compatibility

SYNOPSIS
       cmpdylib oldLibrary newLibrary

DESCRIPTION
       cmpdylib  compares  two versions of a dynamic shared library to see if
       they are compatible with each other...

Attaching the output of the decompile as well:
cmpdylib_retdec_output.zip
Please help me check how come the output doesn't tally with what the binary is expected to do.

Unfortunately, decompiling 64-bit binaries still doesn't work as well as 32-bit. This sample might help further that work, though.

Thank you for the report. @PeterMatula, can you please take a look?

Oh I see. Was there any indication in the README or the Retdec website that 64-bit binaries have problems, which I have missed?

No, you haven't missed anything. It is just that support for decompilation of 64b binaries was added only recently (in the v3.3 release on March 18, which is 1,5 months ago), and we have been working on improving it since then. @PeterMatula will verify the binary you have submitted and will let you know.

ok noted, is there any update? @PeterMatula

Hello @PeterMatula @s3rvac, has there been any progress made on this issue?

Hi @humanitiesclinic. You will have to wait until @PeterMatula takes a look at this issue.

Hi, I tried to decompile the file that you have provided and I was able to reproduce your issue.
From analyzing dissasembly output it is possible to see that there is function stub called from main (also located in main). This function then jumps into (invokes) _xcselect_invoke_xcrun function.
Dissassembly of input file:
mov rbp, rsp
0x100000f77:   lea eax, [rdi - 1]
0x100000f7a:   lea rdx, [rsi + 8]
0x100000f7e:   lea rdi, [rip + 0x29]
0x100000f85:   xor ecx, ecx
0x100000f87:   mov esi, eax
0x100000f89:   call 0x100000f8e
0x100000f8e:   jmp qword ptr [rip + 0x7c] <_xcselect_invoke_xcrun>
...

This file seems to be very optimized, nonetheless, retdec should be able to decompile it properly.
After decoder passage it can be seen (in LLVM IR) that function stub was successfully transformed into function definition:
define i64 @function_100000f8e() {
dec_label_pc_100000f8e:

; 0x100000f8e
  store volatile i64<PHONE_NUMBER>, i64* @_asm_program_counter
  %0 = call i64 @_xcselect_invoke_xcrun()
  store i64 %0, i64* @rax
  ret i64 undef
}

Despite its detection, this function is not called later from the main function and thus it is discarded in later stages of decompilation (it is not used anywhere else too). This is not right because from disassembly it can be seen that this stub function is called in the main function.
When it comes to IDA and Ghidra, their output for this particular file is shown below. When it comes to IDA, it seems to struggle with the same problem as RetDec. The output of Ghidra however seems to be the closest to the real content of the binary file.
IDA
int __cdecl main(int argc, const char **argv, const char **envp)
                 push    rbp
                 mov     rbp, rsp
                 lea     eax, [rdi-1]
                 lea     rdx, [rsi+8]
                 lea     rdi, aCmpdylib  ; "cmpdylib"
                 xor     ecx, ecx
                 mov     esi, eax
                 call    $+5
                 endp ; sp-analysis failed

Ghidra

void entry(int iParm1,long lParm2)

{
                    /* WARNING: Could not recover jumptable at 0x000100000f8e. Too many branches */
                    /* WARNING: Treating indirect jump as call */
  (*(code *)_xcselect_invoke_xcrun)("cmpdylib",(ulong)(iParm1 - 1),lParm2 + 8,0);
  return;
}


Decrease queue mem usage

Queues uses too much memory by default, this PR tries to mitigate that.

With 100k queues this branch uses 5.7G RSS, while main branch uses 5.8G.


[video_player] Add VideoPlayerMediaItem and VideoPlayerMediaSource data structure

When I try to apply video_player to build my online programming education application(Coursea-like), QtClass, for all platforms, I find that the video_player plugin is limited in its management on media items and media source. After I studied the requirements on iOS and Android, AVPlayer and ExoPlayer, I find that it is necessary to support something like AVPlayerItem/MediaItem and AVAsset/MediaSource，so I am writing this proposal to discuss this idea with the video_player development team.
Background
I am using the VOD services of TencentCloud as the server. It has no Flutter support and Desktop support for the client, so I am going to develop a Flutter client SDK by my own. The format of data source is HLS, which is similiar with AliCloud, Huawei Cloud, and all other VOD services I can see in China.
As we know, the media of HLS is encrypted, but the content key is not. It is as like that you use a enhanced door, but the key of the lock is put by the door. So the original HLS is not secure in reality. VOD services of different cloud computing services use different methods to encrypt the content key for the media. For example, Tencent Cloud use DRM to make sure the right of the client at first, and then transform the HLS content key by RSA+AES.
Therefore, it is necessary to explore a API so that the HLS content key can be customized by the developers.
Possible Solutions
At first, I try to do this without changing the official plugin. Then I find that all the possible methods may not be workable. So I am considering add this feature by Native side. When I study the Native solutions for this problems, I find that AVPlayer and ExoPlayer are similiar to solve this problem.
For AVPlayer, the possible way is: https://stackoverflow.com/questions/39124793/playing-aes-encrypted-hls-stream-using-avplayer-ios-swift
For ExoPlayer, the possible way is https://github.com/google/ExoPlayer/issues/5362
I am not sure whether they can work since I am not familiar with Native development, and I am going to write demos recently.
Both of them support data structure for media info. AVPlayerItem in AVPlayer or MediaItem from ExoPlayer can manage the item played by the controller. AVAsset and MediaSource can manage the original media info.
We can define similar data structures called VideoPlayerMediaItem and VideoPlayerMediaSource in the existing plugin, and add new API to existing VideoPlayerController. This is no-breaking change on the APIs, but maybe need to rewrite some implementations, for example, the default construct can use Media class as input.
With them, it is much easier to customize the HLS content key. We just need to define a sub data structure for HLS, and explore a API so that the media info can be modified. After we get the HLS content key, we can use this API to modify the media info before playing.
More information on the Native side:

AVPlayerItem: https://developer.apple.com/documentation/avfoundation/avplayeritem
AVAsset: https://developer.apple.com/documentation/avfoundation/avasset
MediaItem: https://exoplayer.dev/media-items.html
MediaSource: https://exoplayer.dev/media-sources.html

Notice that AVPlayerItem seems to represent a video, and MediaItem represent a piece of video. I am new to them so I am not sure whether my understanding is right.
For Web, I think the best method right now is to apply video.js with import_js_library(https://pub.dev/packages/import_js_library), since the native support of Web on audio and video is limited.
I have read a rejected PR on Mac support of this plugin in 2019, since the platform view is not finished. I am not sure whether it is OK at now. I am not familiar with Windows and C#, and will read more docs on that.
My Plan
I am going to develop a private version at first so that my project can go on. I really hope that this feature can be contributed to the official plugin, so I want to know about how this plugin and its APIs are designed, and try my best to correspond with the API design.
In addition, I am confused that I found no git submodule settings in the Flutter plugins repository. I have no idea on how to build the CI workflow to make PRs to the official repository.
Thanks for reading it! Hope to discuss deeply on this topic.

In addition, I want to get some suggestion on the versioning. I know that at present this plugin has a plan to migrate to version 2.x with null-safety support.
I am not going to support null-safety at present, since it will add a lot of work for me at my current projects. If I follow my plan to develop, what is a good versioning for me? I am going to rename the project as like video_player_qt so that I can replace it only in my settings. My idea is that my version is 1.x until now at present, and then 2.x after I migrate to null-safety. I am not sure whether it will be a good versioning strategy for Flutter.

In addition, I want to get some suggestion on the versioning. I know that at present this plugin has a plan to migrate to version 2.x with null-safety support.
I am not going to support null-safety at present, since it will add a lot of work for me at my current projects. If I follow my plan to develop, what is a good versioning for me? I am going to rename the project as like video_player_qt so that I can replace it only in my settings. My idea is that my version is 1.x until now at present, and then 2.x after I migrate to null-safety. I am not sure whether it will be a good versioning strategy for Flutter.

@Guo-Zhang what is the current status of this development?

@jswietek
No one has discussed this proposal with me, and I have no idea on how to solve this problem. Due to additional problems by TencentCloud, I have given up video features in my APP at present. Maybe I will consider to solve this problem after one or two years, after the desktop support of video player is stable.
I wrote some basic code:

https://quanttide.coding.net/public/qtopen-flutter/qcloud-vod-player-flutter/git/files
https://quanttide.coding.net/public/qtopen-flutter/video-on-demand-flutter/git/files
https://quanttide.coding.net/public/qtopen-flutter/video-player-desktop-flutter/git/files

Since they are based on older versions, I am not sure whether it can work.
Hope it helps! If you want to discuss my code with me, free to email me.


Dup bug

Dup bug

Please use the security tap at the top of the page. It's built for that kind of stuff.

I sent you in private message on the sponge forum because the security page is not active (for me on github)

News ? If the dup spreads, all servers that use Flexiblelogin could be impacted

Fixed in #172


Removed 1.6 Objects class and unneeded imports.

Now compiles with 1.5 rt.jar.

I will try to get this released later today, now at work so can't do it right away.


feat: Refactor @remix-run/eslint-config with added support for Testing Library

This PR does a few things needed before we integrate our ESLint config with the starter templates:

Updates dependencies for all test-related tooling
Integrates some useful rules for Testing Library. This will make migration easier for users of Next or Create React App which both ship Testing Library ESLint rules by default.

Because this change will also change the lint results for users already using our config, this would be a breaking change if introduced directly in our jest module. This PR also deprecates that module and introduces a new one named @remix-run/eslint-config/jest-testing-library which is also more explicit about what the package includes.

Breaks up rulesets into individual modules. This makes it simple to extend a specific ruleset with a different set of conditions. For example, a user who uses Cypress might want to use the same Testing Library rules for their Cypress and Jest tests, but they might not need other Jest rules for those tests. Those users can import the testing-library rules as a module and extend them directly via their config:// eslint.config.js
const testingLibRules = require("@remix-run/eslint-config/rules/testing-library");
module.exports = {
  extends: ["@remix-run/eslint-config", "@remix-run/eslint-config/jest-testing-library"],
  overrides: [
    {
      files: ["cypress/**/*.{js,jsx,ts,tsx}"],
      rules: {
        ...testingLibRules,
        // override these because they don't make sense in Cypress:
        "testing-library/prefer-screen-queries": "off",
        "testing-library/await-async-query": "off",
      },
    },
  ],
};

Updates our internal ESLint config to use the new rules and fixes some internal files and examples that are flagged as a result
Reduces the severity of more opinionated rules in the jest ruleset to a warning instead of an error
Adds a few more useful rules for Jest:

"jest/no-deprecated-functions"
"jest/no-disabled-tests"
"jest/no-export"
"jest/no-focused-tests"

I'mma fan 👍


warn: parse failure: bitcoin.org

Today I started seeing this error in my terminal when entering a project with dev activated.
cd projectx
warn: parse failure: bitcoin.org
warn: parse failure: bitcoin.org
env +go.dev~1.21 +npmjs.com +git-scm.org

rm pkgx.yml
cd ..
env -go.dev~1.21 -npmjs.com -git-scm.org
warn: parse failure: bitcoin.org
warn: parse failure: bitcoin.org
warn: parse failure: bitcoin.org

cd projectx
env +go.dev +git-scm.org

cd ..
env -go.dev -git-scm.org
warn: parse failure: bitcoin.org
warn: parse failure: bitcoin.org

An org-wide search for bitcoin: https://github.com/search?q=org%3Apkgxdev+bitcoin&type=repositories
I have to be frank here and say I'm not super happy to see something is trying to potentially reach out to bitcoin.org from my machine...

Don't worry, nothing in our code contacts bitcoin.org. What has likely happened is the package file for the bitcoin server (named: bitcoin.org > https://github.com/pkgxdev/pantry/blob/main/projects/bitcoin.org/package.yml) is likely corrupt in your pantry. Running pkgx --sync should re-download the pantry (you can review the whole thing in that repository if you're concerned, but it's all YAML files that describe how to build/test the software we provide).
It could have happened to any file in the pantry, and is almost certainly a coincidence that it's that specific package. @mxcl is tracking reports of pantry corruption (these have gone down significantly as we've improved synchronization methods), so he might have further questions.
Obviously, since our software is open source, you can search this repository and pkgxdev/libpkgx if you still have concerns, but I can assure you we have not implemented any code to contact bitcoin.org, which is just the homepage for the bitcoin core project.

Thanks @jhheider, I'll try the sync command. I see now that there's been other reports of this parse failure too.

Yes, pkgx --sync fixed it. 👍

It should sync periodically, but if a sync is broken, it might be a bit before the next one.


Does it support pi?

sysuh3c --user xxx
Argument Error. Not option.
sysuh3c -h is ok.

I don't think pi (Linux) has anything special compares to other platforms.
Have you tried sysh3c -u xxx?

The reason is function getopt_long does not support arm.
It will always catch default tag.
I trid to modify the code and have fixed it.
It works. Thank you!


SASS class divider-or:after needs to be internationalized

SASS element divider-or needs to be split into three parts : one pulled left, one centred and one pulled right.  The value of the CSS tag "content" needs to be an I18n lookup key.

ofc you can do that.

Do you have a SASS specialist?  This is not a strong area for me.

Yes I had. @Gembalaboy
me too, I am not good at design, css etc


Fix #126178

Previously, opening two markdown files side-by-side caused unexpected scrolling. This was due to a bug introduced in #123727, which was fixed by separating the last scroll locations of static/text editors and consuming the info entry upon access.
This PR fixes #126178

Merging as candidate in #126293


Write NDEF record

Hi,
First thanks for this library!
I cannot manage to write an NDEF record to a card. I believe it should be possible...
Here is what I tried:
const { NFC } = require('nfc-pcsc');
const nfc = new NFC();
const ndef = require('@taptrack/ndef');

nfc.on('reader', reader => {
  reader.aid = 'F222222222';

  reader.on('card', async card => {
    try {
      var textRecord = ndef.Utils.createUriRecord("http://www.exemple.com");
      var message = new ndef.Message([textRecord]);
      var bytes = message.toByteArray();

      await reader.write(4, bytes);
    } catch (err) {
      console.error(`error while writing data`, err);
    }
  });
});

Which gave me this error:
error while writing data TypeError: "list" argument must be an Array of Buffers
    at Function.Buffer.concat (buffer.js:332:13)
    at /home/pi/node_modules/nfc-pcsc/dist/Reader.js:562:26
    at Generator.next (<anonymous>)
    at step (/home/pi/node_modules/nfc-pcsc/dist/Reader.js:18:191)
    at /home/pi/node_modules/nfc-pcsc/dist/Reader.js:18:437
    at /home/pi/node_modules/nfc-pcsc/dist/Reader.js:18:99
    at Reader.write (/home/pi/node_modules/nfc-pcsc/dist/Reader.js:583:5)
    at /home/pi/node_modules/nfc-pcsc/dist/Reader.js:546:27
    at Generator.next (<anonymous>)
    at step (/home/pi/node_modules/nfc-pcsc/dist/Reader.js:18:191)

Am I missing something?

Hi @nicolasbadia,
I think that the problem is here:
The data argument of reader.write(blockNumber, data) must be an instance of Node.js Buffer, but message.toByteArray() returns ES Uint8Array (according to their documentation).
So, you need to convert the Uint8Array into to the Buffer. You can do it like this:
Buffer.from(arr.buffer) // in your code it is Buffer.from(bytes.buffer)

That's the most efficient way (the one which shares the memory between Uint8Array and Buffer instances). You can read more about the differences between Buffers and TypedArrays here.
Here is the fixed code:
const { NFC } = require('nfc-pcsc');
const nfc = new NFC();
const ndef = require('@taptrack/ndef');

nfc.on('reader', reader => {
  reader.aid = 'F222222222';

  reader.on('card', async card => {
    try {
      const textRecord = ndef.Utils.createUriRecord("http://www.exemple.com");
      const message = new ndef.Message([textRecord]);
      const bytes = message.toByteArray();
      // convert the Uint8Array into to the Buffer
      const data = Buffer.from(bytes.buffer);

      // data is instance of Buffer
      await reader.write(4, data);
    } catch (err) {
      console.error(`error while writing data`, err);
    }
  });
});

Hope it helps. 🙂

Hi @nicolasbadia,
I found out another important fact about writing NDEF data on card.
In order to create NDEF formatted card/tag, you have to encapsulate the NDEF message into TLV blocks (more info here and in the links mentioned there).
TLV structure is needed for the tag to be recognized as NDEF formatted tag. It allows the applications to read NDEF messages from tags without knowing the data length (I am preparing implementation of reader.readNdef and reader.writeNdef to this library in order to make it easier to work with NDEF).
Let's get back to writing NDEF data. I implemented the encapsulation process.
Here is the updated code (assuming you are using Mifare Ultralight):
const { NFC } = require('nfc-pcsc');
const nfc = new NFC();
const ndef = require('@taptrack/ndef');

const encapsulate = (data, blockSize = 4) => {

  if (data.length > 0xfffe) {
    throw new Error('Maximal NDEF message size exceeded.');
  }

  const prefix = Buffer.allocUnsafe(data.length > 0xfe ? 4 : 2);
  prefix[0] = 0x03; // NDEF type
  if (data.length > 0xfe) {
    prefix[1] = 0xff;
    prefix.writeInt16BE(data.length, 2);
  } else {
    prefix[1] = data.length;
  }

  const suffix = Buffer.from([0xfe]);

  const totalLength = prefix.length + data.length + suffix.length;
  const excessLength = totalLength % blockSize;
  const rightPadding = excessLength > 0 ? blockSize - excessLength : 0;
  const newLength = totalLength + rightPadding;

  return Buffer.concat([prefix, data, suffix], newLength);

};

nfc.on('reader', reader => {
  reader.aid = 'F222222222';

  reader.on('card', async card => {
    try {
      const textRecord = ndef.Utils.createUriRecord("http://www.exemple.com");
      const message = new ndef.Message([textRecord]);
      const bytes = message.toByteArray();
      // convert the Uint8Array into to the Buffer and encapsulate it
      const data = encapsulate(Buffer.from(bytes.buffer));

      // data is instance of Buffer containing encapsulated NDEF message
      await reader.write(4, data);
    } catch (err) {
      console.error(`error while writing data`, err);
    }
  });
});

I have testes it and it works! 🚀 The tags/cards written with this code are being correctly recognized by Android.
As I have written above, the encapsulation process will be integrated into nfc-pcsc (as reader.writeNdef).
Please let me know if everything works for you. 🙂
Hope it helps. 🙂

Thanks a lot for the help and the quick answer.
I managed to write NDEF records to the card with your code.
I do not have an Android device yet to test if it works, but I will have one soon...
I tried to read the NDEF records with the library, but couldn't manage to do it.
Not sure where the issues come from... but this is another issue...

@nicolasbadia Hey Nico, can you share a reading ndef example? thanks

@pokusew do you have a reading example?

@chelinho139 Hi, take a look at our discussion on discord. I think you will find what you need there.

How to find application id

Whats the update on this .readNDEF and .writeNDEF


Update dialog.md with info about filter extensions

There was some confusion about the correct way to add an All Files filter (see #2525), so I added it to the example.
Also added a short note about the extension syntax.
This is related to PR #2547.

Thanks for the following updated work!

@hokein, it's fixed.

Thanks!


cargo crev crate clean not working

If I run cargo crev crate clean and then cargo crev verify, verify always says I have 2 unclean packages. I think they were two crates I was reviewing on an older version of crev, before I un-installed it and later re-installed it.
This is with crev 0.21.4, installed from Cargo. I'm using Arch Linux on x86_64.
Running the clean command under strace shows that it's trying to delete some files that don't exist. It looks like the files in e.g. ~/.cargo/registry/src/github.com-1ecc6299db9ec823/anyhow-1.0.44/ are in a half-deleted state where some files remain (making verify say they're unclean) but clean doesn't know which ones still need deleting.
I'll tar up the unclean folders as a backup, in case there's anything interesting in the presence / absence of files.

After trying it out myself I immediately started getting similar problems. After some investigation I've changed the reported diagnostics and now it looks like this:
4 local crates with digest mismatch detected. Use `cargo crev crate clean [<name>]` to clean any potential unclean local copies. If problem persists, contact the reporter.
Crate aead                 0.4.3           ; local digest: sWHmGSevjVwjDxyT631QUCRIiA0Kq8SpIkkRGx5i9go != sBQJ4TW9c324zYgd5DvWuaz-fdDrJC1fdc8f0JW8RG4 reported by 24YKeuThJDNFSlJyxcl5diSZcKcRbh-0zXM0YxTOFJw (https://github.com/Lucia
noBestia/crev-proofs)
Crate atty                 0.2.14          ; local digest: WJaqS5xu6ORo2px2Rku1ynYYN4TuCdxeS8LR68ngyw4 != 43bdBdz29rGPtdjhLdE0zA-ZudRkwB1ISyTTj0vft-I reported by 24YKeuThJDNFSlJyxcl5diSZcKcRbh-0zXM0YxTOFJw (https://github.com/Lucia
noBestia/crev-proofs)
Crate cipher               0.3.0           ; local digest: RWl72FpMPCylFnYYtdCaNUhg_7BBwcocTAgjl_sLQTc != vDRlIx2w0uBLUqrDmno2yli_uUZ4E8-LUexlJLW3Cjo reported by 24YKeuThJDNFSlJyxcl5diSZcKcRbh-0zXM0YxTOFJw (https://github.com/Lucia
noBestia/crev-proofs)
Crate libz-sys             1.1.3           ; local digest: rEGXjh1JwMUUN0Cgcx-24kbPJ2eZUGba3K4TQsgolPo != ZU3ZXFo-2MXtbDQu_s0jFzS-2eFSKSK_jB8HIKEyOAU reported by 24YKeuThJDNFSlJyxcl5diSZcKcRbh-0zXM0YxTOFJw (https://github.com/Lucia
noBestia/crev-proofs)

Hi @LucianoBestia!

Other than this crate clean [<name>] seems to work just fine.

I Will look into it. I use the function from crev lib to calculate the digest.

@LucianoBestia Oh. In that case I can already tell you, that you probably want to use https://github.com/crev-dev/cargo-crev/blob/ae2c6ffc4c4b9e3b6dd8513e20da4f0d61c5f03b/cargo-crev/src/shared.rs#L379 instead.
For Rust we need to do bunch of sheningans before calculating digest. See comments.

I have this code for the digest:

fn ....
    crate_name: &str,
    crate_version_str: &str,

...
let crate_root = crate::cargo_registry_mod::cargo_registry_src_dir_for_crate(crate_name, crate_version_str)?;
let digest_clean = crev_lib::get_recursive_digest_for_dir(&crate_root, &cargo_min_ignore_list())?;
...

pub fn cargo_min_ignore_list() -> fnv::FnvHashSet<std::path::PathBuf> {
    let mut ignore_list = std::collections::HashSet::default();
    ignore_list.insert(std::path::PathBuf::from(".cargo-ok"));
    ignore_list
}

/// cargo registry src directory for a crate
pub fn cargo_registry_src_dir_for_crate(crate_name: &str, crate_version: &str) -> anyhow::Result<std::path::PathBuf> {
    let crate_dir_name = format!("{}-{}", crate_name, crate_version);
    let crate_src_path = format!(
        "{}/registry/src/github.com-1ecc6299db9ec823/{}",
        home::cargo_home()?.to_str().unwrap(),
        crate_dir_name
    );
    let crate_root = std::path::PathBuf::from_str(&crate_src_path)?;
    // return
    Ok(crate_root)
}

Just for curiosity: Does crates.io have a digest or hash to verify if the downloaded files are not compromised?
If it has, is it the same as the crev digest?

Just for curiosity: Does crates.io have a digest or hash to verify if the downloaded files are not compromised?

Last time I check it did not, at least not in the unpacked source.

I found this "cksum" field in the json cargo registry files, not in the src directory:
\\wsl$\Debian\home\luciano\.cargo\registry\index\github.com-1ecc6299db9ec823\ca\rg\cargo_auto_lib
Is this perhaps the checksum of the source code folder?
I suppose cargo should have a mechanism to check if the downloaded files are compromised.
{
    "name": "cargo_auto_lib",
    "vers": "0.7.23",
    "deps": [{
...
        }
    ],
    "cksum": "ba680d84f32d0dc2be953b5f7b3f2e6a3fd61fbe7d220b3ad18a6e7bd5d27daa",
    "features": {},
    "yanked": false,
    "links": null
}

I was also wondering why we are computing digests, instead of using the checksumming mechanism from the Cargo.lock files?

Mostly because cargo is not obligated to stick with these. They are free to repackage all their tar.gz (or whatever is being used) to .xz tomorrow to save on space, and leave cargo-crev hanging. Same with other package managers. That is why crev is using https://github.com/crev-dev/recursive-digest over extracted files to checksum the source code.


Last entry on the btreemap is missing on iter

Hi @seniorjoinu
https://github.com/gabrielnic/ic-rust/blob/main/backend/master/src/lib.rs
Based on this btree
#[derive(StableType, AsFixedSizeBytes)] pub struct Db {   data: SBTreeMap<u64, SBox<Vec<u8>>>, }
if you run this commands:
`build master and install and in order to recreate it run:
dfx canister call master len_val -> should be 0
dfx canister call master insert
dfx canister call master len_val -> should be 3
dfx canister call master all
it should return 3 values but all I get back is
(
vec {
record { 1 : nat64; blob "\01\02\03\04\05\06\07\08\09\0a" };
record { 2 : nat64; blob "\01\02\03\04\05\06\07\08\09\0a" };
},
)`
Not really sure what the issue is and why the last element is omitted but see if you can replicate and find out what the issue is.

Fixed and published as 0.4.3


Add events support to DiagnosticScope

Introduces DiagnosticScope.AddEvent method to enable Cosmos SDK tracing needs.

LGTM!


Add Evmos bridge

We should add a native bridge integration for transferring to/from Evmos.

Closing since it's an idea in ClickUp. Can reopen if pursued.


Consider making internal SDK components private

Currently we export a lot of functions, types, objects, and classes from our core packages that we consider to be internal. This leads people to depend on these components which we do not recommend (example: https://github.com/prisma/prisma/blob/9c6e80e6fadc3068a54bc6cbff2edb218e88d345/packages/engine-core/src/tracing/createSpan.ts). As much as possible we should encourage users to use the API interfaces only.
This could be done by making them private, making the constructors private, or making all properties not included in the interface private (maybe there is some way to exclude some properties from the published types while keeping them internally?)
non-exhaustive list:

SDK Span and Tracer
SDK Meter and Metrics
TracerProvider (still need a way to configure it though)
Everything in core package

We could export interfaces instead classes where possible. This has the additional benefit that typescript doesn't complain about incompatible types just because private fields differ.


Can't stop debugging since "1.81.1" version.

Does this issue occur when all extensions are disabled?: Yes

VS Code Version: Visual Studio Code-1.81.1 and higher
OS Version: MacOS Sonoma 14.1.1 (Apple M1)

Steps to Reproduce:

"Disable All Installed Extension"
Removing ~/.vscode for creating a new one by vscode
Create a vscode .code-workspace file
Inside it, create the config for debugging Launch for browser (chrome) and node sides
Start Debugging
Wait until the application and the debugger is running (for demo I've used "Create React App", a fresh installation)
When trying to stop debugger process it is not possible.

1.80.2 is the last version when everything works fine.
1.81.1 (next release) doesn't work.
1.84.2 (the latest version) doesn't work.
DEMO (video):
https://www.youtube.com/watch?v=GrrfDiqReGk

Please collect trace logs using the following instructions:

If you're able to, add "trace": true to your launch.json and reproduce the issue. The location of the log file on your disk will be written to the Debug Console. Share that with us.
⚠ This log file will not contain source code, but will contain file paths. You can drop it into https://microsoft.github.io/vscode-pwa-analyzer/index.html to see what it contains. If you'd rather not share the log publicly, you can email it to<EMAIL_ADDRESS>

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.
Happy Coding!


Loaders & Dependency Management

Hey, nice tool you've got here!
I am running a simple test with the json-loader, with code like this:
var foo = require('./package.json');
module.exports = foo.version;

My config:
module.exports = {
    context: __dirname,
    entry: "./index.js",
    module: {
        loaders: [
            {
                test: /\.json$/,
                loader: "json"
            }
        ]
    }
}

Let's say I was to author this package on NPM as foo. Now, I'm writing another module called bar, which depends on foo. The code for that new module:
var test = require('foo');
console.log(test)

Unfortunately, bundling the above gives me Cannot find module "package.json" errors. Presumably because it doesn't know to use the JSON loader when resolving the foo module.
Browserify handles this by putting transforms in the package.json, so that when it reaches that module, it knows to transform it. This seems like a pretty important feature for code re-use and dependency management.
Or am I simply missing some configuration?

Related: http://spmjs.io/ (was mentioned in a comment to the npm blog post)
Let's hope concord will make its way in the nearest future. Very promising.

Suggestion: move concord out of the webpkac org and into a new org for modules specs, since it should apply to all build systems. f.e. github.com/concord/spec

I'm working on packaging ogv.js and its audio output component audio-feeder in npm, and this static assets problem is definitely biting me.
audio-feeder (and thus ogv.js) needs to ship a Flash .swf file for audio support on IE 11, which must be served from the actual web server -- you can't cheat with a data: or blob: URI. There are also several emscripten-compiled .js files meant to be loaded from web workers, for which blobs might work though I'm less sure about IE support.
As long as I declare 'file-loader' in my dependencies and specify my loader options inline like require("file?name=audio-feeder/[name].[ext]!./dist/dynamicaudio.swf") or require("file?name=ogv.js/[name].[ext]!./dist/ogv-worker-video.js") everything seems to work ok in a test project that does a require('ogv').
However having to repeat the full loader setup on every invocation gets a little verbose if I want my files in a particular directory structure like that... currently I need to have consistent filenames within a base directory, though I can probably rewrite things to pass full hashable paths down to the Flash loader and the worker threads.
Has there been any progress on providing loader configuration defaults per-module in the package.json, or an alternate interface versus overloading require()?

@brion The core idea of webpack is that the app developer, not module developer, decides how the assets should be packaged and served. In your case, you should make your static resources published into your npm package as raw files, and your module should accept a URL for each of them. The app developer who does the bundling would require these assets and will provide your module with every needed URL. They can be hosted differently, e.g. on a CDN, and you never know what app dev would need. You can make the API of your module so that it demands specifying the URLs.

Sure, all the paths are configurable. But it'd be nice if webpack could pack the assets for a web app, both reliably and configurably.
Reliable: default case should Just Work. Assets belonging to dependencies and dependencies of dependencies and dependencies of dependencies of dependencies should be packed/bundled/copied along with the other output.
Configurable: all paths should be customizable in the app's webpack config.
Both: Things like inlining images should be configurable in general, but module developer still needs to be able to specify "this .swf file cannot ever be inlined as a data URI because Flash won't load it that way" or "this .PNG file needs to be embedded as data, not a URL".
It seems to me that having defaults in package.json that can be overridden in the app config globally and on a per-module basis would go a long way. Webpack is supposed to help the app developer, and I'm all in favor of that!

Also I'm going to go on record as saying that requiring the app developer to read readmes of every module in the dependency graph to get a list of every static asset they have, and manually require all those resources individually, and pass every URL through into the configuration of every affected module, seems like a very developer-unfriendly configuration system.
Not only is it poorly scalable to an initial setup, it would require every package update in the dependency graph to be checked for additional asset files. Maintenance would be ... Unpleasant in this model.
This is literally what computers are for: put the list of assets in a manifest file and have the computer manage them but with the human's oversight.

Also I'm going to go on record as saying that requiring the app developer to read readmes of every module in the dependency graph to get a list of every static asset they have, and manually require all those resources individually, and pass every URL through into the configuration of every affected module, seems like a very developer-unfriendly configuration system.

Yep, that's true. The situation has not improved – still it's the best way to configure things on app level instead of on library level.
@brion I think it's currently the best solution to separate actual source files that should be added to the webpack bundle from static assets that should just be emitted to the output folder. In your case, I'd place the dynamicaudio.swf and the ogv-worker-video.js into a dedicated assets folder and then configure the file-loader to be applied on all these assets by default.

@jhnns yeah, the trick with the worker .js assets is that default require('foo.js') behavior is to execute, not to return a URL. So if I just write this:
var worker = new Worker(require('./dist/ogv-worker-video.js'));

it won't actually work unless the webpack configuration includes a suitable binding, maybe:
{ test: /ogv-.*.js$/, loader: "file?name=/some-dir/ogv.js/[name].[ext]" },

I'm also a bit unclear from reading https://webpack.github.io/docs/using-loaders.html on whether I can specify an inline loader type like these to guarantee I get a URL to an actual file from require():
flashObject.src = require('file!dynamicaudio.swf');
var worker = new Worker(require('file!ogv-worker-video.js'));

while still allowing the loader options to be overridden in the app's config. Do inline loader options override the regex-configured ones, or the other way around?
I think I'm going to do an initial package release without the external webpack exposure -- just manual documentation of the dist-ready assets and how to set the asset paths at runtime -- and keep refactoring and testing until I'm happier with the resulting output.
(I am now using webpack to build my pre-packed distribution bundle for use outside the npm universe -- it's pretty awesome at that level! Still getting the hang of it though, and really looking forward to more module composability. :)

Can this help with the worker? https://github.com/borisirota/webworkify-webpack

whether I can specify an inline loader type

This is technically possible but highly discouraged in the webpack world.

@sompylasar webworkify-webpack looks like it does the blob loader trick I
would need to support cross domain loading of the modules, but doesn't seem
like it would handle my use case correctly, in that it appears to pack the
worker modules into the main source and provides no asynchronous load
mechanism.
(Also, I need to confirm that IE 10/11 can load workers from blob URLs,
which I have not tested.)
I'm supporting multiple codec modules in a media player widget, and need to
avoid loading codecs that are not needed for whatever files get played (or
if playback never is started, at all). I also want to avoid parsing
JavaScript for those modules -- large, dense code bases cross-compiled from
C libraries -- on the main thread, which would happen with
webworkify-webpack as it seems to fetch live code modules and turn their
source back into strings. So even if I loaded them asynchronously, it would
need me to load them into the main thread's webpack_require world as
live modules first, wasting CPU time and memory.

Yeah, looks like async loading with code splitting per http://webpack.github.io/docs/code-splitting.html would work for the on-demand loading, but would still parse on main thread and then again on the worker. I can play games with turning source files into modules that eval strings to reduce that perhaps, but that feels a bit ugly when I have to pack into a string anyway for the worker blob load trick.
Back to the original  topic...
I think it would be very useful to expose a list of static asset files in the package.json, which would be machine-readable versus just a mention in the read me. At the least I would love as an app packager to have an automated report generated every time I run webpack.
Better still would be a common runtime registry interface allowing modules packed using webpack to ask for the app-packager-configured URLs.
This sounds a lot like require('foo.png') but with the specificity that it always returns a URL, rather than only returning a URL when file-loader or url-loader is configured and trying to import JavaScript code otherwise.
The app packager could then set all the loader options they want by making a giant mapping of registered asset keys to require() invocations, with inline loaders or with configuration in the webpack config object.
Going one step further, an initial such mapping can be auto generated by a build tool, with overrides specified by the app packager.
This would I think pretty much do what I want, would allow for both automatic and manual configuration of the asset loader details, as long as dealing specifically with URLs to static assets. And it remains much less ambitious than the concord ideas linked from discussin last year -- packages are responsible for prebuilding their static assets if they want to make js from coffee script or css from sass.
I think his could be done pretty straightforwardly as a plugin to webpack, in a way that could still allow for full manual configuration of everything if you didn't want to use the build tool plugin.
Eg something like this in usage:
var staticLoader = new require('static-asset-loader')(
	'audio-feeder', // package name
	'assets' // base dir for assets in package
);

function AudioFeeder(options) {
	if (options.base) {
		staticLoader.base = options.base;
	}
	flashObject.src = staticLoader.url('dynamicaudio.swf');
	// ...
}

with added to the package.json:
"assets": ["dynamicaudio.swf"],

and in the app's calling code, choose either:
// paths configured by build tool
var audioFeeder = new require('audio-feeder');

or
// dont forget to copy the assets
var audioFeeder = new require('audio-feeder', {
	'manually/specified/path/to/copied/assets'
});

The build plugin would scan through the package manifests and build a default map, like:
require('static-asset-loader').override({
	'audio-feeder': {
		'dynamicaudio.swf': require('audio-feeder/assets/dynamicaudio.swf')
	}
});

which would get processed for loaders and included in with the other modules, with a call forced to make sure it runs before app code tries to use the assets. The tool could even suggest mappings for your config file as defaults.
Suitable static analysis could probably avoid the need for adding a list of files to the package.json.
Thoughts?

@brion

Better still would be a common runtime registry interface allowing modules packed using webpack to ask for the app-packager-configured URLs.
This sounds a lot like require('foo.png') but with the specificity that it always returns a URL, rather than only returning a URL when file-loader or url-loader is configured and trying to import JavaScript code otherwise.

The thing is, you don't always have a URL for an arbitrary module if you don't use file-loader or url-loader. The module can be embedded into a chunk of JS next to other modules.

packages are responsible for prebuilding their static assets

This part looks reasonable, and even looks like almost what I suggested: a package exports its static assets for the app to use (it may bundle or just manually expose at a URL). This is how ES6 npm modules are made compatible with the surrounding ecosystem: they are transpiled into ES5 CommonJS which are consumable by Node.js natively.

with added to the package.json:

This looks exactly what I suggested, but with added automation.

Eg something like this in usage:

This looks even better. But I'd make something like:
// In a module that provides static assets.
var url = require('static-asset-provider')('./assets/dynamicaudio.swf');  // Can be found and resolved statically by webpack
if (!url) {
    throw new Error('Cannot get URL of `dynamicaudio.swf`. Check your bundler configuration.');
}

// In an app that uses webpack to bundle.
plugins: [
    // Captures all `require('static-asset-provider')`,
    // and replaces it with `require('/full/path/to/assets/dynamicaudio.swf')`
    new StaticAssetProviderPlugin()
]

No need to scan through some manifests, every dependency is expressed with the code itself (that's one of the core ideas and benefits of webpack which I extremely like).

@sompylasar
Definitely agree that static analysis beats explicit manifests in this case; I like your plugin config example. :)
I guess what I want is the power of static analysis in the packing system that webpack already provides with require() and readFileSync(), plus more flexible machine-readable annotations about the return type and the sync-ness of the load that will allow working default behavior that is also reconfigurable.

The thing is, you don't always have a URL for an arbitrary module if you don't use file-loader or url-loader. The module can be embedded into a chunk of JS next to other modules.

In many cases it doesn't matter to the library whether you use a local URL, remote URL, or pre-pack them into data: URLs, or even runtime-convert a string literal into a blob: URL. What you care about as a library author is you need to feed a URL into some HTML or CSS or JS API.
At other times, that packing simply will not work, such as the Flash .swf file example -- you specifically need a URL, but you also specifically need an externally-referenced file. For this reason I would consider them separate data types ('static URL' versus just 'URL').
Similarly, strings and array buffers can be either embedded as modules returning literals in the packed JS, or they can be loaded from static files at runtime via XHR. Supporting both effectively requires using an async interface for the asset loading -- something not needed for URL references because the loading is done in someone else's code in the browser engine.
To extend your version of the example a bit with types and sync-ness:
var assets = require('static-asset-provider');

// These always return URLs, but we don't care if you pre-packed the file data or not.

img.src = assets.url('./assets/loading-spinner.svg');

worker = new Worker(assets.url('./assets/prepacked-worker.js'));

// When you need a URL that *must* be served as a separate asset to work,
// but you still don't care what it's named or what directory it's in etc.

flashObject.src = assets.staticUrl('./assets/dyamicaudio.swf');

We could extend this further to allow string & ArrayBuffer imports to be either separate files loaded via XHR or strings/arrays packed into the source, at the app packager's desire:
// Async loader interface allows the files to be either loaded via XHR or packed in JS.

assets.string('./assets/example.js').then(function(str) {
  textarea.text = str;
});

assets.buffer('./assets/codec-setup.mem').then(function(buf) {
   audioCodecInitializerData = buf;
});

// When you need a string or buffer and you need it NOW
// sync XHR may be unsafe or difficult to use, as browser makers hate it.
// Possibly redundant to readFileSync()

textarea.text = assets.stringSync('./assets/example.js');

audioCodecInitializerData = assets.bufferSync(./assets/codec-setup.mem');

I think these can all fit with the static analysis plugin model you suggest in a way that leaves the packaging location/method up to to the app packager, while the library author remains in control of what datatypes they'll get back.
So if the app packager tried to configure the .swf file in a way that couldn't be loaded as a real remote URL, the asset loader plugin would yell and scream at build time that this would not work because it's the wrong data type.
Similarly, the default behavior of static-asset-provider when not rewritten could be configurable with a runtime interface to supply URLs or pre-packed string/buffer data. That helps avoid the 'webpack vendor lock-in' problem described in a few places earlier on this thread -- a library that needs static assets using this system would benefit from webpack + a plugin, but would not require it.

I'll try putting together a plugin based on this model and see how it works out...

Haha, you've summoned a monster by reviving this thread @brion :wink:
Your quote summarizes the essence of the previous discussion:

I think these can all fit with the static analysis plugin model you suggest in a way that leaves the packaging location/method up to to the app packager, while the library author remains in control of what datatypes they'll get back.

Writing require("./some/img.png") is like requesting a hook from the build system to do something. The build system cannot know what the library author is expecting from this require(): It could return a url, a data url or just the size of the image. So this should be up to the library author. The library author, on the other hand, should not have any assumptions about how these things end up into the final bundle. It should "just work"™.
The proposed concord spec tries to solve this with a declarative solution which maps paths to type definitions and then type definitions to build system transformations.
Example: require("./some/img.png") is referenced by using *.png, then mapped to url/image which is then mapped to url-loader!./some/img.png in a webpack environment.
The concept sounds really nice, but it requires a lot of community effort. And I'm still unsure if a declarative approach is flexible enough. Perhaps it is better to move more responsibility to the library author (like you should compile your ES2015 stuff before pushing it to npm).

@substack 's approach is different and very node-focused: You would not require("./some/img.png"), but write something like this:
var sizeOf = require("image-size");
var dimensions = sizeOf("images/funny-cats.png");

Then, your build system would need a transform/loader which "anticipates" the API calls by reading the AST. Thus, you can do things at build time what you cannot do at runtime (like reading files from disk). This approach is very isomorphic friendly because you don't have to do anything in node.js to make this code work.
On the other hand, it's not very scalable because you'd need a image-sizeify for browserify, an image-size-plugin for webpack, etc. It basically pushes responsibility to the eco-system.

@brion @sompylasar your approach is similar to @substack 's one, but it sounds really interesting. I'm curious about your prototype...

One thing I wanted to add to subtack's approach:
It looks really nice, because you don't have to compile your stuff before feeding it to node. But there are also disadvantages: Depending on the use-case it could really slow down your application startup because every require() is synchronous. What if you'd required several LESS files? In node, compilation could only be done sequentially. This is basically why enhanced-require is deprecated.
Although it looks like an overhead, in practice it's better to streamline the client and the server environment by always having a build step before execution.

@jhnns hehe, monster indeed! :)
I want to avoid overriding require() in part because files of the same file type may need to be referenced and used in different ways... But I also don't want to go too crazy on transformations. For instance an image-size build time plugin would be awesome! But... As a module author I could use a webpack loader, or some similar transform system, in my pre-publish build stages to do that.
I'd like to concentrate on an interface for that last step, where my module code gets packed into someone else's project either manually or via webpack or browserify. At that point I already know the image's size from my build, so I just need either its contents or a URL reference to it.
So, hoping to keep it unambitious, which may make it easier to get working...

So, hoping to keep it unambitious, which may make it easier to get working...

That's a valid point. In this case, it's certainly better to solve the asset referencing issue then to catch all potential edge-cases. That's what I was trying to say about pushing responsibility to the library authors. They should do as much as possible without having any assumptions about the build system.

@brion

In many cases it doesn't matter to the library whether you use a local URL, remote URL, or pre-pack them into data: URLs, or even runtime-convert a string literal into a blob: URL. What you care about as a library author is you need to feed a URL into some HTML or CSS or JS API.

Yes, it doesn't matter which URL you get, but sometimes you really don't have a URL, neither local, nor remote, nor data, nor blob. What you have is, e.g. base64 of the file contents. Yes, you can somehow transform it into a data: or blob: URI.

To extend your version of the example a bit with types and sync-ness:

Yes, I like your approach. It will be harder to statically extract, but nonetheless it's possible.

We could extend this further to allow string & ArrayBuffer imports to be either separate files loaded via XHR or strings/arrays packed into the source, at the app packager's desire:

Yes, great!

Similarly, the default behavior of static-asset-provider when not rewritten could be configurable with a runtime interface to supply URLs or pre-packed string/buffer data. That helps avoid the 'webpack vendor lock-in' problem described in a few places earlier on this thread -- a library that needs static assets using this system would benefit from webpack + a plugin, but would not require it.

Exactly! One thing is that if 'static-asset-provider' is not rewritten by something like webpack or require-hacker, there cannot be any URL, because this library knows nothing about URLs, and how the assets are served, and where this server will be located (the 'L' for locator in 'URL').

I'll try putting together a plugin based on this model and see how it works out...

Wow would be very cool! Please link from here as soon a something is pushed.


Improve security.

Add oauth

use OAUTH for communication between services? How to integrate with future changes to accounts/user management?


Build a new image just to get rid of CVEs

Greetings,
When we scan the alxrem/prometheus-logstash-exporter container image with a tool like Snyk, Snyk reports a ton of vulnerabilities in this container base image:
Tested 14 dependencies for known issues, found 23 issues.

Base Image     Vulnerabilities  Severity
alpine:3.12.3  23               2 critical, 16 high, 3 medium, 2 low

Recommendations for base image upgrade:

Minor upgrades
Base Image   Vulnerabilities  Severity
alpine:3.13  0                0 critical, 0 high, 0 medium, 0 low

Here, upgrading the base image would remove a lot of noise.
On top of that, Alpine 3.12 is reaching end of live on 01 May 2022.
Since the Dockerfile is always using the latest version of alpine, all we need is to build a new image, ship it to Docker Hub as a version 0.8.0, and all our security tools will stop telling us we are doing bad things.
Can we do it? I would do it myself, but I don't have access to this project CI/CD setup, nor to the Docker Hub project.
Thanks!

Hi @JPLachance .
I attempted to fix some CVEs in this PR.
@alxrem What do you think about it?


fix: chunkwriter hangs on cancel

Chunkwriter copyFromReader will hang when canceling in response to a StageBlock error. sendChunk blocks on writing to the channel, but the writers exit on cancel without draining the channel.
This change allows sendChunk to exit with an error in this scenario, rather than block on channel write.
Test added to repro.

Running CI on https://dev.azure.com/azstorage/Go SDKs track 1.5/_build/results?buildId=3589&view=results


Add Symbol and timedelta interop features to cirq.Duration

Introduce DURATION_LIKE type to replace many instances of Union[Duration, timedelta]
Add micros and millis as supported units
Improve the __str__ and __repr__ methods to use better scaled units
Allow _picos to be symbolic, add _parameterized_ and _resolve_parameters_ methods
Allow comparisons with timedelta and with 0
Deprecate Duration.create in favor of Duration's constructor
Add __bool__ method for easy non-zero checking

 >>> print(cirq.Duration(micros=1.5 * sympy.Symbol('t')))
(1500.0*t) ns

Fixes https://github.com/quantumlib/Cirq/issues/2279

I added the DURATION_LIKE to _GateOperation.

Automerge cancelled: A status check is failing.


python3 compatibility

added python3 compatibility
The change from dict to OrderedDict was necessary because the order in a dict depends on the hash. Since python3.3 whenever hash collisions occur, randomness will be included in the order which we can't manipulate with random.seed() (see https://stackoverflow.com/questions/15479928/why-is-the-order-in-dictionaries-and-sets-arbitrary/15479974#15479974)

Thank you for your contribution!


[FLINK-14343][coordination] Remove uncompleted YARNHighAvailabilityService

What is the purpose of the change
Corresponding mailing list thread.
Noticed that there are several stale & uncompleted high-availability services implementations, I start this thread in order to see whether or not we can remove them for a
clean codebase.
Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not applicable)

Hi @tillrohrmann , as reached consensus on mailing list, please take a look at this patch. I feel the conflict is unnecessary if we take this patch before modify LeaderElectionService interface.

Thanks for your review @tillrohrmann !


Support compound XOR with '~='

Seems cursed. Sure, this is technically a compound XOR, but it's also Lua's inequality operator. Being used as a statement does not read off well.

I think it's shit that they used ~ for both XOR and inequality (via ~=), so this is a mistake we have to continue supporting now, but we shouldn't let that mistake drag us down, so if I want to quickly perform an XOR like this, I should be able to.
Why I think it's not a problem to introduce this:

Unambiguous to parse because of "statement vs expression" which already has tons of parsing differences in Pluto (class, switch).
XOR is rather rare. You'd only see this in "low level" code such as hash functions and protocols. I doubt this would be anyone's first look at Pluto code.
We already do offer != so it would be perfectly possible for someone to be disciplined about their usage of ~= vs !=.

The fact it's the same exact operator is the ambiguous part. Same operator, different meaning, different metamethods called. Ambiguities always have instances where they are clear, the whole point is that they can be easily confused for something else and this is a text-book example of that.
And besides, cursed. I'd like a XOR compound operator, just not one that's exactly the same as the inequality operator. Unfortunately there are not many new operators suitable for this task, if any at all.

It's not like this would be the first instance of the same token or keyword having different meanings depending on context:

: is probably our worst offender with method calls, ternaries, and switch statement cases
strings and tables don't just declare data, they can also initiate a function call
class and switch have entirely different layouts and semantics depending on context (statement vs expression)

Also, I think it's fair to say that != is the inequality operator for Pluto, because ~= doesn't make a lot of sense for inequality, but it's a "supported for legacy reasons" kinda thing. — Not saying this is my opinion, but it's a way someone could look at Pluto, and it does match some of the Pluto code I've seen.

Switch statement cases is a well-agreed upon fault in our syntax so following through with that as precedent would be doubling down on an error. For the method syntax, the short-hand ternary provided substantial value over if then else. We've adopted imperfect syntax before so long as it provided a significant amount of value despite its flaws, exactly like how our compound operators don't support XOR. At best, this feature is neutral weight. The problem with neutrality is that neutral features either don't age at all, or age badly.

I don't really think the way we did switch statement cases is bad. It's exactly the same syntax everyone knows for them, and we made it work in Lua, with consideration for the ambiguities with the :. I think it works great, especially with the warning coverage.
This feature is certainly less problematic than that, because I don't see anyone making a mistake with this, so we wouldn't even need warnings to complement this syntax.

Will this break the compatibility with existing Lua scripts?

No, existing Lua scripts only use ~= as an expression. This also adds a meaning for it as a statement.


DRAFT: Resolve function calls by name during planning

Addresses #8157
Implement an AnalyzerRules that rewrites Expr::ScalarFunction replacing ScalarFunctionDefinition::Name into ScalarFunctionDefinition::BuiltIn or ScalarFunctionDefinition:ScalarUDF at plan time.
Introduces a new AnalyzerConfig to wrap a FunctionRegistry and ConfigOptions
Ignore from here, left for reference
Unclear how to pass the FunctionRegistry down to the analyzer rule.
Option 1:  When creating the analyzer
Using this approach means giving up the Default implementation, as well as doing some refactoring in https://github.com/apache/arrow-datafusion/blob/c8e1c84e6b4f1292afa6f5517bc6978b55758723/datafusion/core/src/execution/context/mod.rs#L1348, since the analyzer needs a FunctionRegistry and the SessionState implements that trait
Option 2: When invoking the analyzer
Using this approach means changing the pub trait Analyzer adding a fourth parameter.
pub trait AnalyzerRule {
    /// Rewrite `plan`
    fn analyze(&self, plan: LogicalPlan, config: &ConfigOptions, registry: impl FunctionRegistry + Send + Sync) -> Result<LogicalPlan>;

    /// A human readable name for this analyzer rule
    fn name(&self) -> &str;
}

However this creates on its own a problem
the trait `AnalyzerRule` cannot be made into an object
consider moving `analyze` to another traitrustc[Click for full compiler diagnostic](rust-analyzer-diagnostics-view:/diagnostic%20message%20%5B2%5D?2#file%3A%2F%2F%2Fworkspace%2Farrow-datafusion%2Fdatafusion%2Foptimizer%2Fsrc%2Fanalyzer%2Fmod.rs)
mod.rs(55, 8): for a trait to be "object safe" it needs to allow building a vtable to allow the call to be resolvable dynamically; for more information visit <https://doc.rust-lang.org/reference/items/traits.html#object-safety>
the trait `analyzer::AnalyzerRule` cannot be made into an object
consider moving `analyze` to another traitrustc[Click for full compiler diagnostic](rust-analyzer-diagnostics-view:/diagnostic%20message%20%5B5%5D?5#file%3A%2F%2F%2Fworkspace%2Farrow-datafusion%2Fdatafusion%2Foptimizer%2Fsrc%2Fanalyzer%2Fmod.rs)
mod.rs(55, 8): for a trait to be "object safe" it needs to allow building a vtable to allow the call to be resolvable dynamically; for more information visit <https://doc.rust-lang.org/reference/items/traits.html#object-safety>
No quick fixes available

Maybe the best solution is to create an intermediate struct that contains the various HashMaps of SessionState, an pass that struct to the Analyzer, as well as the SessionState constructor? Suggestions are welcome :)

Thanks @edmondop  -- I hope to check this out in more detail tomorrow

Maybe the best solution is to create an intermediate struct that contains the various HashMaps of SessionState, an pass that struct to the Analyzer, as well as the SessionState constructor? Suggestions are welcome :)

Do you mean follow the model of try_optimize
/// `OptimizerRule` transforms one [`LogicalPlan`] into another which
/// computes the same results, but in a potentially more efficient
/// way. If there are no suitable transformations for the input plan,
/// the optimizer can simply return it as is.
pub trait OptimizerRule {
    /// Try and rewrite `plan` to an optimized form, returning None if the plan cannot be
    /// optimized by this rule.
    fn try_optimize(
        &self,
        plan: &LogicalPlan,
        config: &dyn OptimizerConfig,
    ) -> Result<Option<LogicalPlan>>;
...

Maybe we could do something similar for AnalayzerRules like
pub trait AnalyzerRule {
    /// Rewrite `plan`
    fn analyze(&self, plan: LogicalPlan, config: &AnalyzerConfig) -> Result<LogicalPlan>;
...
}

/// Options to control  DataFusion Analyzer Passes.
pub trait AnalyzerConfig {
    /// Return a function registry for resolving names
    fn function_registry(&self) -> &dyn FunctionRegistry;
    /// return datafusion configuration options
    fn options(&self) -> &ConfigOptions;
}

I am not sure if the compiler will be happy with that construction or not so we may have to play around with it
Let me know if you want me to give it a try
Thanks for pushing this along
cc @2010YOUY01

@alamb apologies, I would need some guidance also on what to do on some unit tests.
For example, this test https://github.com/apache/arrow-datafusion/blob/main/datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs#L1533-L1549
will never pass now, because  it uses the now function, that uses call_fn which doesn't get resolved if they analyzer is not applied
How do you think we should proceed?

That would be great. I am not sure how to do that without duplicating the
code from the Analyzer that resolves built in functions
On Mon, 11 Dec 2023 at 07:21, Andrew Lamb @.***> wrote:

For example, this test
https://github.com/apache/arrow-datafusion/blob/main/datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs#L1533-L1549
will never pass now, because it uses the now function, that uses call_fn
which doesn't get resolved if they analyzer is not applied
I think these tests will need to be updated to resolve the expressions
"now" to the actual now() function
However, I think this brings up a good point about making sure the expr_fns
can be easily used with the lower level expr api (like simplification).
For any API on the SessionContext or DataFrame, expr_fn::now() and
related functions will be run through the analyzer.
However, if we create the exprs directly like in expr_simplifer (and
there are some examples) we have to arrange for the expressions to be
resolved.
Maybe we could add a function like Expr::resolve to do this more easily?
So it could be used like
let expr = now()
.cast(DataType::Integer) // cast now to integer
.resolve() // recursively resolve all functions calls, like now() into actual function referenes
🤔
I can try and sketch up something like this later today if you want?
—
Reply to this email directly, view it on GitHub
https://github.com/apache/arrow-datafusion/pull/8447#issuecomment-1850294075,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAGGOKITQEOGLT7F5IF4AKTYI4QIBAVCNFSM6AAAAABAKNZHHWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJQGI4TIMBXGU
.
You are receiving this because you were mentioned.Message ID:
@.***>

Helping with this PR is on my list, but I am behind on other reviews and bug fixes. Sorry for the delay

Thank you for taking this issue! @edmondop
Looks like Expr API is exposed in multiple places

DataFrame API
Expr which can be directly evaluated
Constructing LogicalPlan with Expr

I found there are several issues for this rewrite to resolve approach:

This can only make call_fn() backward compatible for DataFrame API, because only DataFrame API is created using context/SessionState, other APIs are created statelessly and can't access context(FunctionRegistry). The inconsistency might be confusing.
Even for DataFrame API, rewrite approach seem also not easy to do: inside DataFrame it will construct LogicalPlans before analyzing, which will validate the schema by checking the function return type. We might have to resolve function names before analyzing. https://github.com/apache/arrow-datafusion/blob/7f312c8a1d82b19f03c640e4a5d7d6437b2ee835/datafusion/expr/src/logical_plan/plan.rs#L1783

Is it possible to just do ctx.call_fn() instead? I think this API change can make function Expr creation consistent, also easier to implement. @alamb
Reference: current Expr for UDFs are created like, ctx.call_fn() is a bit simpler and also can support both builtins and UDF during transition period
    let pow = df.registry().udf("pow")?;
    let expr1 = pow.call(vec![col("a"), col("b")]);

🤔  I played around with this PR some this afternoon.
I came to the same conclusion that we would have to make non trivial API changes to the expr_fn() APIs if we go the "must resolve string names to functions" as part of an analysis pass.
The only other thing I can think of is to change the dependency structure
Right now, the expr_fns are in the datafusion_expr crate
We could potentially move them from datafusion_expr to a new crate (perhaps datafusion_expr_fn) that could rely directly on the built in function definitions 🤔
Let me give that a whirl to see how it would look


Leader Line when working on full-screen mode

Hi,
I want to use this library in a module which works with the fullscreen mode of the browser. When calling to constructor with the browser at fullscreen, it works but the arrow is not showing. When exiting the fullscreen mode, the arrow is there where the two divs were when the browser was at fullscreen. Is it possible to set z-Index to the arrows or how can I work with them into fullscreen mode?
Thanks

Hi @mcarpinter, thank you for the comment.
Open this page: https://anseki.github.io/leader-line/ , and press F11 key.
Then, it seems to work fine.
Maybe, the cause of the issue is not fullscreen.
To reproduce that, could you show me an example by using e.g. https://jsfiddle.net/ ?

Hi,
Thanks for the quick response. It is complicated to provide an example since the module that I want to implement this is part of a bigger system. I will try it anyway.
One thing that is in my mind is the following; I am moving to fullscreen a specific div of the module, not all of it, so when it draws the arrow, it does it over the body, not that div. It may be this the issue?
I am uploading 3 screenshots to see if they help.
Screen1, I click into the tab that is within the red box, and it triggers the fullscreen into the div that is within the orange box
Screen2, The div is at fullscreen, I executed the constructor method for the new line
Screen3, when exiting the fullscreen, the arrow is there at the coordinates of the divs that I wanted to connect

position method might help you if you didn't call the method.
Anyway, I or someone might be able to help you when you show example.

Please note that objects dragging, screen or div scrolling and page size change will cause misposition of the leader-line. It was suggested to use line.position() binding to the events I mentioned above.
Example:
ondrag, jquery.draggable - drag ,document.body.onresize, $(selector).on('scroll', function(){ line.position();}

@debugaman, thank you for the information.
Note that the method is called automatically when the window was resized.
And also, attaching the method to events directly is not suggested for a performance.
See document: https://anseki.github.io/leader-line/#position

No reply came, then this abandoned issue is closed.


Pos tags

Currently, the pos-tag resides in the .tag_ instead of the pos_ where it should (also) be.

This is a misunderstanding the morphologizer provides the pos tag pos_ (UPOS) while the tagger provides the fine-grained pos tag tag_ (XPOS).


Increased font size and changed color for home section titles

Small tweaks to home section titles. Also extracted the home section title into a separate component to prevent duplication.
Sections after

Section before

👍


Migrate to Seccomp profile in security Context :warning:

Description
This PR changes seccomp auditor to scan seccompProfile instead of annotations as requested in https://github.com/Shopify/kubeaudit/issues/343
The following changes were done:

seccomp auditor changed to scan securityContext in pod and containers to find seccompProfile. :warning:
New fix created in seccomp auditor to create seccomp profile in a pod and containers and to remove profile in containers. The auditor updated to use the new fix.
SeccompAnnotationMissing rule renamed to SeccompProfileMissing. :warning:
SeccompDeprecatedContainer rule dropped. :warning:
Tests updated to cover changes in auditor and fix.
Documentation updated.

Fixes #343
Type of change

[X] This change requires a documentation update :book:
[X] Breaking changes :warning:

How Has This Been Tested?

Mostly by automated tests.
I've also executed kubeaudit for examples from documentation to update them.

Checklist:

[ ] I have :tophat: my changes (A 🎩 specifically includes pulling down changes, setting them up, and manually testing the changed features and potential side effects to make sure nothing is broken)
[X] I have performed a self-review of my own code
[X] I have made corresponding changes to the documentation
[X] I have added tests that prove my fix is effective or that my feature works
[X] New and existing unit tests pass locally with my changes
[X] The test coverage did not decrease
[X] I have signed the appropriate Contributor License Agreement

hi, @Ser87ch ! Thank you for your contribution! I will take a closer look at your PR either today or tomorrow. In the meantime, would you mind adding support to both annotations and seccompProfile?

@dani-santos-code Thanks for your reply. May I ask, why? Seccomp annotation support will be removed quite soon.
Additionally, supporting both will add more complications, like what to do if we've got both security context and annotations set.

Thanks @dani-santos-code , I'll let you know tomorrow.

If the annotation is going to stop working and the security profile has already been available for some time, I think dropping annotation support is fair. +1 to @dani-santos-code's suggestion though, if it's not too much of a hassle.
I think ideally Fix would remove the seccomp annotation as well, if it's present.

I've encountered an issue, that I don't know how to fix yet. I'm testing a change that you suggested with the following pod spec:
apiVersion: v1
kind: Pod
metadata:
  name: pod
  namespace: seccomp-profile-missing-annotations
  annotations:
    seccomp.security.alpha.kubernetes.io/pod: runtime/default
    container.seccomp.security.alpha.kubernetes.io/container: localhost/bla
spec:
  containers:
    - name: container
      image: scratch

The problem is when I fetch security context the pod spec, it returns me a security context with seccomp profile set:
&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[]Sysctl{},WindowsOptions:nil,FSGroupChangePolicy:nil,SeccompProfile:&SeccompProfile{Type:RuntimeDefault,LocalhostProfile:nil,},}

I have a feeling that k8s library returns seccomp profile in security context even if only seccomp annotation set. If it's the case I'm not able to differentiate between seccomp set in security context and annotations.
I've committed this change, this test fails seccomp-profile-missing-disabled-container.yaml. Am I missing something?

It's not a k8s library problem, it the way k8s API works. If I export the pod spec I mentioned in the previous comment by using kubectl get pod pod -n seccomp-profile-missing-annotations -o yaml, it returns:
apiVersion: v1
kind: Pod
metadata:
  annotations:
    container.seccomp.security.alpha.kubernetes.io/container: localhost/bla
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{"container.seccomp.security.alpha.kubernetes.io/container":"localhost/bla","seccomp.security.alpha.kubernetes.io/pod":"runtime/default"},"name":"pod","namespace":"seccomp-profile-missing-annotations"},"spec":{"containers":[{"image":"scratch","name":"container"}]}}
    seccomp.security.alpha.kubernetes.io/pod: runtime/default
  creationTimestamp: "2022-09-13T08:46:02Z"
  name: pod
  namespace: seccomp-profile-missing-annotations
  resourceVersion: "1735"
  uid: cc1671c0-bbde-4a98-91ee-5d4761cb19d4
spec:
  containers:
  - image: scratch
    imagePullPolicy: Always
    name: container
    resources: {}
    securityContext:
      seccompProfile:
        localhostProfile: bla
        type: Localhost
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-jh5xb
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: kubeaudit-test-control-plane
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    seccompProfile:
      type: RuntimeDefault
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
 ...

So, I don't believe it's possible to catch a situation when seccompProfile is absent in securityContext but is present in annotations.
@genevieveluyt @dani-santos-code what do you think? Should I just raise a warning if seccomp annotations are present?

Based on these docs, it sounds like having seccomp enabled is the new default behaviour, so kubeaudit might not even have to check that seccomp is enabled but rather should produce an error if seccomp is explicitly disabled (eg. if the profile is set to Unconfined). Did you run your kubelet with the --seccomp-default command line flag? I wonder what the API returns for the seccompProfile if you don't use that flag?

@genevieveluyt thanks for your comment. As far as I understand from the link you shared and feature gates, it is a default behaviour but only since version 1.25 (which is in beta). I tested it with the default v1.20.15, where this behaviour isn't even available.

I see. If neither the annotation nor the seccomp profile are set, what does the library return for the profile?

There is misunderstanding here. Annotation is set in the example I put above. The problem is that Kubernetes API returns also seccomp profile in the security context when only annotation is set.

If neither is set, then it returns empty securityContext.

Thanks, your proposal sounds good. I will change accordingly.

Well that's a rather unexpected and annoying way for the API to behave lol. There's no way in kubeaudit to only run a check manifest mode, no. What if, for number 1 above ie

Check the pod for seccomp annotations. If there are any, produce a warning result where the pending fix removes all said annotations

We change that check to "the pod has only seccomp annotations"? That should only ever happen in manifest mode then right? I don't think there's anything we can do about the annotations in cluster and local mode given how the API behaves...

@genevieveluyt I've introduced a separate method that audit annotations only if seccomp profile is missing for both pod and all containers. This way it will be easier to remove this logic once seccomp annotation support is dropped.

@genevieveluyt could you review PR?

@Ser87ch thank you for making the changes, I will review and 🎩 by end of next week.

@dani-santos-code thanks, how can we merge this one first?


GPG keys: add about how to save passphrase against typing it on every commit

What article on docs.github.com is affected?
https://github.com/github/docs/blob/main/content/github/authenticating-to-github/managing-commit-signature-verification/signing-commits.md
What part(s) of the article would you like to see updated?
Near To store your GPG key passphrase so you don't have to enter it every time you sign a commit, we recommend using the following tools: add about Linux, how to configure gpg-agent (as I tried it will start automatically after first commit, without any configuration), and how to not enter passphrase even for the first time.

At minimum man gpg-agent and info gnupg might be reviewed by STAFF.

👋 @vitaly-zdanevich Please fill out the issue template in the top comment in this issue and then I can get this triaged for you 💛
The issue template helps us triage things easier, means we don't have to go back and forth to get all the required information, and it ensures that anyone that contributes a PR based on your issue has all the information needed in one place without having to go through the entire list of comments attached in order to get started.

@janiceilene done

Done

Thanks @vitaly-zdanevich! I'll triage this for the team to take a look 👀

Thanks @vitaly-zdanevich, that sounds like useful information to include. Because that tip box is already quite long, we should remove the callout (so the existing content it just regular text) and add an "About commit signatures" header above all the conceptual content, which will mean also adding a "Signing commits" header to the procedure content as well. You or anyone else is welcome to open a PR doing that. ⚡


Two Hamlet Fixes

Hi!
I fixed two bugs with the hamlet DLC (one per commit):

spawner and childspwaner components would sometime spawn their child in the void if the spawner is in an interior. The fix makes the prefab try a 100 times every 5 seconds before giving up. This works well for the few spawners that can be in interior.
the pig queen would sometime try to give something to wilba (when playing her), but does not and end up following her forever, messing up the dialogues. This fix simply sets the gift delay to infinite, effectively preventing the gifts.

I hope the code is clean enough for you.
I would like to contribute more in the future, do you have anything you'd like me to work on ? Also, would you mind pushing the 12.5 update (or last one) to this repo ?
Cheers.

Hello Benjamin,
I will take a look at your code soon. Ty.
I have a big backlog of bugs to fix and investigate, maybe I'll add a markdown with them.
About the repository not having x.x.x versions, it's because there's usually not much interest in here, but I'll start committing them too.

Nice! Thanks a lot, looking forward to all of that.

Okay, a link to the backlog is on the main MD.
The repository is in the current state of my development.

Now you need to pull the commit, so that the branch doesn't have conflicts (I think).

Ok! Should be good to go.
Thanks for the list, I'll see what I can do soon and come with more pull requests.
Also, I found some mistakes in strings.lua, where some strings are overidden/unused, is this a bug that needs fixing ?

Also, I found some mistakes in strings.lua, where some strings are overidden/unused, is this a bug that needs fixing?

Depends, unused strings are fine. Most overrides are usually the same phrase (just a duplicate) and when they aren't, it's hard to figure out which one is correct.

Benjamin, I couldn't understand the function of your spawner fix.
In theory, what is written in the code is: the spawner doesn't spawn childs when the player is outside the interior (when the spawner is in limbo), but your explanation doesn't reflect that.
The code checks if the spawner is not in limbo (if the spawner is loaded) before spawning a child.
And there's another point in this: spawners that always spawn off screen (in limbo) will never spawn childs.
I shouldn't have accepted the merge without further checking, but it's done...

Hum yes, this is not py proudest code, sorry.
I thought IsInLimbo tested if the entity was in an interior AND despawned, but it appears it only checks the later.
Maybe replacing not self.inst:IsInLimbo() by not (self.inst:CheckIsInInterior() and self.inst:IsInLimbo()) would do it ?
As for the offscreen-only spawners, I don't think any can be present in an interior anyway. The only spawners I can think of are: rabbithouse, pighouse, wildborehouse, molehill, beebox, dragoonden, monkeybarrel and spiderden.

I think IsInteriorLimbo exists, but anyway, what exactly is the bug about?
Do mobs that spawn from these entities ignore wall barriers because the walls aren't loaded? I wanted to understand how to reproduce it.
Also, the code is fine! I just don't know the bug.

Ah ok my bad, didn't understand that.
The bug is that if a spawner is built in an interior, and the interior isn't loaded when the spawner tries to spawn it's child, the child will be lost somewhere very far from the map, in the void.
To reproduce:

Place a rabbit house in an interior during the day, c_spawn("rabbithouse").
Exit the interior.
Wait for dusk, and then some more so that the rabbit has time to spawn from it's house.
Enter the interior, the rabbit should not be there.
You can try finding the rabbit with c_gonext("bunnyman"). It will most likely be wandering in the void.

Oh ok. Interiors are built in a weird way... They're in one place, but at the same time they're not.
Perhaps running the code for "transition to the interior" (what happens to the player when he enters the interior) might work, and thus not preventing the spawn.
I'll take a closer look at the possibilities, but thanks for the clarification.

Yeah, I'm sure there are smarter ways to "fix" this. My code is more of a patch than anything.


Add DB command processor to multiplex commands to several idle threads.

Lets suppose that storage system has pool of threads and we do not add/read data there directly but instead we ask one of threads in pool to do job for us.
So for example if user starts to push data in system it will not use single thread and get let say 40 000 records per second but will use all CPU cores and will get (if we have 8 cores for example) 120 000 records per second which will be really nice.

Contradicts with current tx bounds, but may be possible to do in cheap way if we implement concept of detached transactions. Delayed till now.

We consider using SEDA architecture on storage level which would include this solution as part of it for embedded storage. I am closing this issue because the implementation of only this feature will not provide expected speed up.