fix: resolve error in deepimport

Description
Mentioned https://github.com/vitejs/vite/pull/3039, there are only tryindex when no exports field. It will be error in below code.
Maybe there should tryindex again when no resolved or tryindex always.

// error
import Button from 'vant/lib/button'
// succeed
import Button from 'vant/lib/button/index.js'

Additional context
There is a reproduction
$ git clone git@github.com:zhangyuang/vite-ssr-error-require.git
$ git checkout bug/tryindex
$ yarn && yarn dev

What is the purpose of this pull request?

[x] Bug fix
[ ] New Feature
[ ] Documentation update
[ ] Other

Before submitting the PR, please make sure you do the following

[x] Read the Contributing Guidelines.
[x] Read the Pull Request Guidelines and follow the Commit Convention.
[x] Check that there isn't already a PR that solves the problem the same way to avoid creating a duplicate.
[x] Provide a description in this PR that addresses what the PR is solving, or reference the issue that it solves (e.g. fixes #123).
[x] Ideally, include relevant tests that fail without this PR but pass with it.

Hi, the current behavior seems consistent with node resolver:
Welcome to Node.js v17.1.0.
Type ".help" for more information.
> require.resolve('vant/lib/button/index.js')
'/tmp/bar/node_modules/vant/lib/button/index.js'
> require.resolve('vant/lib/button')
Uncaught Error: Cannot find module '/tmp/bar/node_modules/vant/lib/button'
    at createEsmNotFoundErr (node:internal/modules/cjs/loader:954:15)
    at finalizeEsmResolution (node:internal/modules/cjs/loader:947:15)
    at resolveExports (node:internal/modules/cjs/loader:482:14)
    at Function.Module._findPath (node:internal/modules/cjs/loader:522:31)
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:919:27)
    at Function.resolve (node:internal/modules/cjs/helpers:108:19) {
  code: 'MODULE_NOT_FOUND',
  path: '/tmp/bar/node_modules/vant/package.json'
}

The spec may not be clear around whether an export mapping from ./lib/* to ./lib/* should resolve index.js but the implementation doesn't.
Retrying resolve with index lookup seems to defeat exports specs in many (other) ways.
My 2c.

I vaguely remember the code can be executed at the previous vite version,maybe some underlying dependencies change cause it failed. Below code can pass ci in previous vite version

And i don't think the behavior is expected which can be executed succeed in webpack.

If @gluck confirmed this is intended behaviour by node, I think we should lean on that and not have this change. Unless we figure out the try index only if no exports field comment.

Maybe it's vant bug ,let me take a look

@bluwy Maybe we can fix the problem in vite layer? The kind of writing is frequent, some third-party ui library like antd, vant which will use babel-plugin-import to transform below code
import { Button } from 'antd'

=> import Button from 'antd/lib/button'
import 'antd/lib/button/style'

not tryindex if has exports maybe is not exactly in some case

I don't think Vite should deviate from the spec as it's one of its philosophies. Re-reading try index only if no exports field now, I think it makes sense, since if we have an exports field, that should be respected no matter what.
But regarding vant, it looks like they're not correctly exporting the /lib. I don't see a trailing slash being in the spec, maybe something like:
"./lib/*": "./lib/*/index.js"

would work.
Re antd, they don't seem to be using exports.

Ok, thanks a lot  😄


Fast path for common paths

Most times, we are not parsing file URLs, the URLs are special and we don't need percent encoding, there is no '%' or '\' or '.' in the path. Hence, we can dedicate a fast path.

Closing. Let us discard.


Need to reduce the device code size for RISC-V quantized vectorized workload

Move the discussion for another issue to here so we can have an on-topic discussion.
After the vectorization path of the quantized conv1x1 op was enabled, the library size for the workload (.so or .o) becomes way larger than before. For example, for person detection translated with the static library mode, the resulting .o size goes from 215KB (with candidate-20220225.59) to >1MB
Currently RISC-V codegen shares the same matmul configuration with x86, so there may be some improvement we can do.
To repro:

Download tflite file: https://github.com/google/iree/blob/main/integrations/tensorflow/test/python/iree_tfl_tests/person_detect_test.py#L14
import it into MLIR with iree-import-tosa

iree-import-tosa -o /tmp/tosa.mlir person_detect.tflite

Run the translation targeting on RISC-V:

iree-compile --iree-mlir-to-vm-bytecode-module \
--iree-hal-target-backends=dylib-llvm-aot \
--iree-llvm-target-triple=riscv32-pc-linux-elf -iree-llvm-target-cpu=generic-rv32 -iree-llvm-target-abi=ilp32 \
--iree-llvm-target-cpu-features=+m,+a,+f,+zvl512b,+zve32x \
--riscv-v-vector-bits-min=512 --riscv-v-fixed-length-vector-lmul-max=8 \
--iree-llvm-debug-symbols=false \
--iree-llvm-link-embedded=false --iree-llvm-link-static --iree-llvm-system-linker-path=lld  \
--iree-llvm-static-library-output-path=/tmp/a.o
--iree-input-type=tosa /tmp/tosa.mlir -o /tmp/a.vmfb

Inspect the size and content of the static library output of /tmp/a.o

Thanks for creating the issue, Cindy!
I discussed this with the team and did some exploration today. Our main suspicion was that the code size issue is due to some kind of unrolling. LLVM's default unrolling has been enabled by default in IREE recently (#8408, thanks for the pointer, @MaheshRavishankar!) but I didn’t see a major difference when turning it off.
When looking at the IR, I found some vector shapes that called my attention: vector<64xi32>, vector<8x32xi32>, vector<16x32xi32>, etc. From an x86 perspective (assuming 512-bit registers), a simple vector<16x32xi32> operation would lower to 16x2=32 512-bit vector instructions,  8x2=16 instructions for vector<8x32xi32> and 4 instructions for vector<64xi32>, respectively. That’s effectively a lot of “unrolling” that is increasing the code size and maybe even leading to register spilling. I guess the problem is similar for RISC-V but I’m not sure if the vector length multiplier makes this situation any better (still catching up with hardware details). Maybe you can clarify.
These large vector shapes whose fastest varying dimensions span across multiple physical vector registers are the result of vectorizing operations with mixed data lengths, like i8 and i32. We recently introduced some great initial logic (#8552) to compute the vector size for these cases. This logic is currently choosing a vector size based on the smallest data type found, which will lead to multi-register vector sizes for larger data types (e.g., vector size=64 for i8/i32 ops, vector size=32 for i16/i32 ops). I've seen this being problematic in the past because it may inadvertently increase the register pressure too much and lead to register spilling. I would suggest that we make the default logic a bit more conservative by computing the vector size based on the largest data type instead of the smallest one. We may leave some performance on the table initially but we can refine this logic incrementally by:

enabling more aggressive vector sizes only for specific ops (i.e., i8/i32 matmuls),
taking register pressure into account,
making sure that no other unrolling happens if an aggressive vector size is used.

I run some experiments along these lines:

Version
Code Size

Current baseline
2.0 MB

Disabling LLVM’s unrolling
1.9MB

Computing vector size using the smallest data type
580 KB

+ disabling LLVM’s unrolling
540 KB

Please, let me know what you think and I'll send an initial PR if this makes sense to you.
Thanks

Sounds good, we can examine the generated workload and discuss the next steps.
For RISCV v-ext, in the configuration above the vector register is also defined as 512 bits
--iree-llvm-target-cpu-features=+m,+a,+f,+zvl512b,+zve32x \
--riscv-v-vector-bits-min=512 --riscv-v-fixed-length-vector-lmul-max=8 \

This translates to the system having 32 vector registers (set in RVV spec) with 512 bits in length for each register, and the length multiplier can be up to 8, to represent 4 groups of vector registers of 4096 bits

Sounds good, we can examine the generated workload and discuss the next steps.

I created #8883 for you to give it a try.

4 groups of vector registers of 4096 bits

Do registers need to be consecutive within a group? That would make the problem even worse than for x86 register-wise but I at least you can handle it with just one instruction, I guess.

With LMUL>1, the group of registers are consecutive. For example, LMUL=8 you have 4 groups that are only accessible with v0, v8, v16, v24. A simple register initialization code is like this
vsetvli t0, zero, e8, m8, tu, mu
vmv.v.i v0, 0
vmv.v.i v8, 0
vmv.v.i v16, 0
vmv.v.i v24, 0

However, IIUC there are some additional LLVM options you need set to utilize LMUL >1 properly, otherwise in most cases it is just m1 even though the flag --riscv-v-fixed-length-vector-lmul-max=8 is set.

We have another regression of the code size since https://github.com/google/iree/releases/tag/candidate-20220418.111, possibly due to https://github.com/google/iree/pull/8723. The code size for person_detect grew to 2.5MB. :(

oh shucks.. well, I dont know what the root cause is. The cause might be "more fusion" at that level now leads to more vector instructions being generated for a dispatch....

Yep. I've been working on an heuristic that reduced the code size from 2MB to 500KB. Unfortunately, I see that with the latest version of IREE the baseline is 3.9MB and it goes down to only 3.2MB with the heuristic. Looking into it.
What are you measuring exactly, @hcindyl? I'm just looking at the size of the whole .o file.

You'll want to look at the .so - the .o's we emit aren't stripped (IIRC).

We measured the size during the link time while we link the static library with the runtime (similar to iree/samples/static_library and set the size alarm inside the linker script.
If you want to double check the embedded .so size, you can update the CLI with
iree-compile --iree-mlir-to-vm-bytecode-module \
--iree-hal-target-backends=dylib-llvm-aot \
--iree-llvm-target-triple=riscv32-pc-linux-elf -iree-llvm-target-cpu=generic-rv32 -iree-llvm-target-abi=ilp32 \
--iree-llvm-target-cpu-features=+m,+a,+f,+zvl512b,+zve32x \
--riscv-v-vector-bits-min=512 --riscv-v-fixed-length-vector-lmul-max=8 \
--iree-llvm-debug-symbols=false \
--iree-llvm-link-embedded=true --iree-llvm-embedded-linker-path=lld  \
--iree-llvm-keep-linker-artifacts \
--iree-input-type=tosa /tmp/tosa.mlir -o /tmp/a.vmfb

Yep. I've been working on an heuristic that reduced the code size from 2MB to 500KB. Unfortunately, I see that with the latest version of IREE the baseline is 3.9MB and it goes down to only 3.2MB with the heuristic. Looking into it.
What are you measuring exactly, @hcindyl? I'm just looking at the size of the whole .o file.

Could you post some info of the dispatches that are creating the issue here....

After looking at the .so file, the numbers make more sense. Thanks! Before the latest fusion changes, the baseline code size was 1.3MB and it went down to 250 KB with the initial version of the heuristic I've been working on. After the fusion changes, the baseline code size is 2.2MB and it goes down to only 1.8MB with the heuristic :(.
The latest code size increase happened because recent changes in the fusion algorithm let us fuse matmul ops with subsequent element-wise ops more aggressively. For example, in the snippet below, you can see how ops %28-%34 are fused with the producer matmul and vectorized using the matmul vector size vector<8x16xi32>. Without the recent fusion changes, they were not fused and were vectorized using vector<1x16xi32>, resulting in 8x less unrolling.
  scf.for %arg0 = %c0 to %c64 step %c8 {
    %23 = vector.transfer_read %9[%arg0, %c0], %c0_i8 {in_bounds = [true, true]} : memref<64x8xi8, affine_map<(d0, d1)[s0] -> (d0 * 8 + s0 + d1)>>, vector<8x8xi8>
    %24 = arith.extsi %23 : vector<8x8xi8> to vector<8x8xi32>
    %25 = vector.contract {indexing_maps = [affine_map<(d0, d1, d2) -> (d0, d2)>, affine_map<(d0, d1, d2) -> (d2, d1)>, affine_map<(d0, d1, d2) -> (d0, d1)>], iterator_types = ["parallel", "parallel", "reduction"], kind = #vector.kind<add>} %24, %13, %cst : vector<8x8xi32>, vector<8x16xi32> into vector<8x16xi32>
    %26 = arith.subi %25, %22 : vector<8x16xi32>
    %27 = arith.addi %15, %26 : vector<8x16xi32>
    %28 = "tosa.apply_scale"(%27, %19, %21) {double_round = true} : (vector<8x16xi32>, vector<8x16xi32>, vector<8x16xi8>) -> vector<8x16xi32>
    %29 = arith.addi %28, %cst_0 : vector<8x16xi32>
    %30 = arith.cmpi slt, %29, %cst_0 : vector<8x16xi32>
    %31 = arith.select %30, %cst_0, %29 : vector<8x16xi1>, vector<8x16xi32>
    %32 = arith.cmpi slt, %cst_1, %29 : vector<8x16xi32>
    %33 = arith.select %32, %cst_1, %31 : vector<8x16xi1>, vector<8x16xi32>
    %34 = arith.trunci %33 : vector<8x16xi32> to vector<8x16xi8>
    vector.transfer_write %34, %10[%arg0, %c0] {in_bounds = [true, true]} : vector<8x16xi8>, memref<64x16xi8, affine_map<(d0, d1)[s0] -> (d0 * 16 + s0 + d1)>>
  }

We probably can hack something for this specific case but this doesn’t scale well to future issues. I see this becoming a recurrent source of problems and there is no simple solution. Any small change or new optimization in the pipeline will likely have an impact in code size that might be pretty significant and not necessarily a problem for all the targets. To make an informed decision on how to proceed, we should know what the specific code size requirements are (i.e., what the acceptable code size upper bound is), how important code size is vs performance, how critical the problem is for the project (e.g., could we modify the simulator parameters to accommodate a larger binary?)  and if these requirements are temporary (i.e., code size will be less of a problem after a certain development stage) or permanent.
We talked about introducing optimization levels in the compiler, where one of them would optimize for code size. I think in the short term we could only provide an optimization level that strictly optimizes for code size (equivalent to -Oz, disabling all kinds of unrolling and other optimizations that could increase code size), but that would have a severe impact on performance.
What are your thoughts?

One thing that would be good to analyze is why there is so much code - this seems to indicate there are confounding variables. We should have smaller binaries with more fusion if things were operating correctly as we are reducing all of the boilerplate that would have been around each dispatch. If authoring by hand we should be seeing a few dozen extra instructions if these were unrolled totaling only a small N KB increase.
(I don't think this is a new issue - we've always generated way too much code - but most people have strictly cared about execution performance at the cost of code size and I believe there's some larger issues than just the tuning parameters can solve)

(e.g. in the snippet above all those trailing select/cmp/etc ops should be a single SIMD instruction when lowered, but I suspect they are many orders of magnitude more than that :)

Agreed with Ben on this, we probably are hitting the limits of what is currently implemented in MLIR  w.r.t scalar folding. Also the lowering we use for the CPU codegen has been developed primarily with x86 performance in mind, where codesize is not an issue. Optimization flags can only mask the deficiencies, but we need to make sure either vector dialect does a better lowering to LLVM (or figure our if the LLVM IR lowered to by MLIR isnt getting peephole optimized enough within LLVM). We are fusing more for sure, but we also not fusing that much that it leads to several MB of code. The main pain point I see here is the lowering of vector operations to LLVM IR. It has pretty bad code size and compilation time implications.

I can definitely take a deeper look! Note that there are no other differences at MLIR level other than the fusion of concat + trailing ops. There are quite a few instances of them so they are probably dominating the code size. Some instances look even worse than the one above (e.g., trailing ops go from vector<1x16xi32> to vector<12x32xi32>) and I do see register spilling, which is another multiplier (note that some concat ops are unrolled by a factor of 64 already!). Let's see what else I find :)
Quick and unrelated question: Why is tosa.apply_scale not lowered until we lower to the LLVM dialect? We are missing a few optimizations at MLIR level because of that.

Very excited you're looking into this! The amount of potential performance we're sitting on is quite large :)
@rsuderman has the history; I believe it's because the operation it's performing is not easily expanded with the currently available ops - it makes things a bajillion times worse :)
Here's a related issue: https://github.com/google/iree/issues/7781
You can see what it expands to in the current world here:
https://github.com/google/iree/blob/main/third_party/llvm-project/mlir/test/Conversion/TosaToArith/tosa-to-arith.mlir#L15-L49
The reference for it is in the TOSA spec section 1.5.2:
https://developer.mlplatform.org/file/download/tywkdgcadici74tnd67t/PHID-FILE-ygtmfyvgclfmd5vetehh/tosa_spec.pdf
(apply_scale_32 / apply_scale_16).
It really calls for a small number of ops in arith/math/whatever that model the behavior without needing to expand it to that form (and require the 64-bit math to be explicitly modeled). It's kind of like why we have an FMA instead of just mul+add. The examples above with the cmp+selects are similar expansions with no ops to contract into (clamp/saturate/etc).

there's some orthogonal design work specifically for code size that is also critical to offsetting bloat: executables in IREE can have an arbitrary number of functions and we link them together and deduplicate but today we are effectively generating each dispatch function independently in its own hal.executable and fully inlining everything. Ideally we'd be able to extract common code segments out into functions and call them inside our inner loops to operate on registers (we're not limited by any standard calling conventions here as we're only calling amongst our own code).
As an example say there's an 8x or 16x or ... unrolled sequence of instructions for doing something like the apply_scale: we could generate an internal function that does that, emit a call, and then share that across all dispatches that perform the apply_scale. That's effectively what most libraries do: https://github.com/ARM-software/ComputeLibrary/blob/7dcb9fadb98cad05fca72de3273311d570d98b4e/src/core/NEON/kernels/arm_gemm/quantized.cpp#L59
This would let us take any innermost loop operating on fixed sizes and share it among all dispatches that use it - there's a much smaller number of those vs the total number of unique dispatches post-fusion (matmul + add and matmul + mul could share the same matmul section). The goal of fusing dispatches is to keep things in registers and avoid round-tripping into memory between operations and there are lots of ways to do that besides fully inlining everything.
Both are really exciting and meaty areas with lots of potential :)

I can definitely take a deeper look! Note that there are no other differences at MLIR level other than the fusion of concat + trailing ops. There are quite a few instances of them so they are probably dominating the code size. Some instances look even worse than the one above (e.g., trailing ops go from vector<1x16xi32> to vector<12x32xi32>) and I do see register spilling, which is another multiplier (note that some concat ops are unrolled by a factor of 64 already!). Let's see what else I find :)
Quick and unrelated question: Why is tosa.apply_scale not lowered until we lower to the LLVM dialect? We are missing a few optimizations at MLIR level because of that.

I am missing the context on fusion of concat + trailing ops. Do you have a snippet of the IR that shows a little bit more context on whats happening here.

I am missing the context on fusion of concat + trailing ops. Do you have a snippet of the IR that shows a little bit more context on whats happening here.

My bad... I meant "contract" not "concat" :). Contract + trailing ops like the snippet I posted above. Let me know if it's still not clear and I'll clarify further :)

I spent some time looking at the LLVM-IR right before going into the RISC-V backend and the generated assembly. Paradoxically, the major code size problem is related to the issue #7781 that @benvanik mentioned above. We generate i64 instructions for some tosa ops like tosa.apply_scale. For example:
    %79 = sext <16 x i8> %78 to <16 x i64>, !dbg !90
    %80 = shl <16 x i64> <i64 1, i64 1, i64 1, i64 1, i64 1, i64 1, i64 1, i64 1, i64 1, i64 1, i64 1, i64 1, i64 1, i64 1, i64 1, i64 1>, %79, !dbg !90
  	%81 = sext <16 x i8> %77 to <16 x i32>, !dbg !90
  	%82 = add <16 x i64> %80, <i64<PHONE_NUMBER>, i64<PHONE_NUMBER>, i64<PHONE_NUMBER>, i64<PHONE_NUMBER>, i64<PHONE_NUMBER>, i64<PHONE_NUMBER>, i64<PHONE_NUMBER>, i64<PHONE_NUMBER>, i64<PHONE_NUMBER>, i64<PHONE_NUMBER>, i64<PHONE_NUMBER>, i64<PHONE_NUMBER>, i64<PHONE_NUMBER>, i64<PHONE_NUMBER>, i64<PHONE_NUMBER>, i64<PHONE_NUMBER>>, !dbg !90
  	%83 = sub <16 x i64> %80, <i64<PHONE_NUMBER>, i64<PHONE_NUMBER>, i64<PHONE_NUMBER>, i64<PHONE_NUMBER>, i64<PHONE_NUMBER>, i64<PHONE_NUMBER>, i64<PHONE_NUMBER>, i64<PHONE_NUMBER>, i64<PHONE_NUMBER>, i64<PHONE_NUMBER>, i64<PHONE_NUMBER>, i64<PHONE_NUMBER>, i64<PHONE_NUMBER>, i64<PHONE_NUMBER>, i64<PHONE_NUMBER>, i64<PHONE_NUMBER>>, !dbg !90
  	%84 = icmp sge <16 x i32> %76, zeroinitializer, !dbg !90
  	%85 = select <16 x i1> %84, <16 x i64> %82, <16 x i64> %83, !dbg !90
  	%86 = icmp sge <16 x i32> %81, <i32 32, i32 32, i32 32, i32 32, i32 32, i32 32, i32 32, i32 32, i32 32, i32 32, i32 32, i32 32, i32 32, i32 32, i32 32, i32 32>, !dbg !90
  	%87 = select <16 x i1> %86, <16 x i64> %85, <16 x i64> %80, !dbg !90
  	%88 = sext <16 x i32> %76 to <16 x i64>, !dbg !90
  	%89 = extractvalue [1 x [1 x [1 x <16 x i32>]]] %67, 0, 0, 0, !dbg !90
  	%90 = sext <16 x i32> %89 to <16 x i64>, !dbg !90
  	%91 = sext <16 x i8> %77 to <16 x i64>, !dbg !90
  	%92 = mul <16 x i64> %88, %90, !dbg !90
  	%93 = add <16 x i64> %92, %87, !dbg !90
  	%94 = ashr <16 x i64> %93, %91, !dbg !90
  	%95 = trunc <16 x i64> %94 to <16 x i32>, !dbg !90

I64 vectors are not supported in the RISC-V implementation that we use so every single i64 element in a vector is scalarized and going through memory between operations. Taking into account that we sometimes unroll this code by a factor of 64… You can imagine the mess :). This is for sure also penalizing compile time.
If we add the ‘+zve64x’ feature to support i64 vectors, the code size goes from 2.2MB to 186KB :). We have to address #7781 before we can explore anything else. @benvanik, @MaheshRavishankar could we prioritize that work? Do we have an estimate?

Wow thats a great find. @rsuderman will have more context on that. Lets sync up when we are all back to see what the way forward here is.

mmmm an actual order of magnitude - great investigation :)

@rsuderman has been working on a fix to get rid of the i64 instructions related to tosa.apply_scale: https://reviews.llvm.org/D125583. The patch is still WIP but in a preliminary evaluation the code size goes down to 388 KB!

@hcindyl @dcaballe I believe we can mark this as fixed correct as it was submitted in
https://reviews.llvm.org/D125583

I still have to enable it for RISC-V based on specific hardware features. It's in my TODO list but I haven't been able to get at it yet.

https://github.com/google/iree/pull/9407 added the final bits to fix this issue.
Thank you all!


Render .gitignore, black py files

Fixes | Closes | Resolves #1204

Please remove anything marked as optional that you don't need to fill in.
Choose one of the keywords preceding to refer to the issue this PR solves, followed by the issue number (e.g Fixes # 666).
If there is no issue, remove the line. Remove this note after reading.

Changes introduced in this PR:

Add a means of rendering the .gitignore file to avoid committing terraform provider related files.

Types of changes
What types of changes does your PR introduce?
Put an x in the boxes that apply

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds a feature)
[ ] Breaking change (fix or feature that would cause existing features to not work as expected)
[ ] Documentation Update
[ ] Code style update (formatting, renaming)
[ ] Refactoring (no functional changes, no API changes)
[ ] Build related changes
[ ] Other (please describe):

Testing
Requires testing

[x] Yes
[ ] No

In case you checked yes, did you write tests?

[x] Yes
[ ] No

Documentation
Does your contribution include breaking changes or deprecations?
If so have you updated the documentation?

[x] Yes, docstrings
[ ] Yes, main documentation
[ ] Yes, deprecation notices

Further comments (optional)
If this is a relatively large or complex change, kick off the discussion by explaining why you chose the solution you did and what alternatives you considered and more.

Upgraded black due to dependency incompatibility issue:
https://github.com/psf/black/issues/2964

Does this commit justify a new release?


shift-click broken

I just installed the latest Karabiner-Elements on my iMac running the Sierra GM. I love that the "fn" key once again works as left control, but shift-click no longer works to extend the selection. Reasonable trade-off, but slightly annoying.
Here's my karabiner.json:
{
    "profiles": [
        {
            "name": "Default profile",
            "selected": true,
            "simple_modifications": {
                "fn": "left_control"
            }
        }
    ]
}

It also breaks command-click and command-shift-click, which I use all the time in Firefox, to open a link in a new tab. That is more than annoying.

On my end Cmd Click is also not working for Chrome. Event Viewer doesn't give any inside.

Strangely, Karabiner-EventViewer sees the Cmd and Shift flags on mouse clicks, when Karabiner-Elements is NOT running. Start Karabiner-Elements, however, and those flags no longer appear in Karabiner-EventViewer mouse clicks, though the FlagsChanged events still happen for the command and Shift keys.

Indeed, same problem here. Command+Click for opening new tabs stopped working for me too.

Similar issue for me when Ctrl + Mouse swipe up (full screen zoom in). Is it possible that all key + mouse action combinations are broken?

Hi, the majority of modifier keys don't work for me after installing the nightly build and remapping right option -> forward delete on my 2015 MBP. Alt-Tab for task switching works, but Ctrl-Tab in Safari doesn't (just tabs between links on page instead of between tabs, nor does Ctrl + mouse zoom to zoom in screen.

Similar issue: option + click doesn't work. E.g. when clicking on the Airport icon in the menubar to view extra network information

Same issue.

Same issue here, mod keys with click doesn't work.

Where does the karabiner.json go? I can't find anything where one would actually put it...

@tobsn Does this link help?

@talha131 yeah i found that a minute after asking when I decided to look into the code to find the part where it initially loads the json... *facepalm
thanks ;)
btw. combinations are not possible atm right? like command_r+7

btw. combinations are not possible atm right? like command_r+7

Yeah they are not until #16 gets fixed.

Thank you for feedback!
This issue is caused by OS X problem, and it needed large changes to avoid this problem.
I fixed this issue at Karabiner-Elements v0.90.22.
Please upgrade to the latest version from "Misc > Check for updates" or
https://pqrs.org/latest/karabiner-elements-latest.dmg

I can confirm this issue is fixed.

Me too, thanks!

I updated via "Misc > Check for updates". It took a long time, but finally brought up the app and asked me to identify my keyboard by pressing the key to the right of the left "shift" key. I did so, and nothing happened. I closed the dialog with the red circle on its upper-left-hand corner.
Karabiner-Elements then worked as expected, swapping the fn and control keys on my keyboard, as I specified in ~/.karabiner.d/configuration/karabiner.json. And shift and command work again with mouse clicks.
I can't use it, however, since it breaks things I rely on in Emacs. But I'll open a new issue for that, since it will effect only a small audience, and close this one.

I had the same issue as @billstclair. My keyboard was not identified, but after closing the dialog everything worked as expected.

but finally brought up the app and asked me to identify my keyboard by pressing the key to the right of the left "shift" key. I did so, and nothing happened. I closed the dialog with the red circle on its upper-left-hand corner.

I too faced the same issue. I closed the identify keyboard dialog, uninstalled old version of Karabiner-Elements, and did a fresh install.

Can confirm, I had to do a manual re-install. Updated yesterday and still wasn't working, but a fresh reinstall today fixed the issue.

Looks like this is still not fully fixed. Randomly, after many hours of work, it sometimes stops working. Turning Karabiner Elements off and on again fixes it for some time, but it breaks again after a couple of hours.

Was noticing the same thing today ☝

I created new issue #204 for this problem, since discussing it in closed one not seems to be a good idea.


[asana] Revert Version

Description
Revert the asana library version from 5.07 to 3.2.1 due to backward compatibility issues affecting the AsanaConnect integration when using the native image.
Asana forum

lts check that we cant fix the content first

the pack support level is community, so....


Automate the process of configuring True Values so when fonts are updated the config doesn't break

The 'true values' are no longer being reflected in the parametric axes, when user axes are set, and I guess this is because the values config was based on the font file, and when the file changed, the config broke.
It would be great to have that process automated so that (a) when pre-loaded fonts are updates, they continue to work as expected and (b) if a user loads a new font with parametric axes, it also "just works"
Related issues for reference:

https://github.com/TypeNetwork/variable-type-tools/issues/42
https://github.com/TypeNetwork/variable-type-tools/issues/61

Sadly this isn't working today

And... it’s almost impossible to keep it working, but I’ll see what we can do to divert resources to a permanent display of a single unchanging version of the font.
Unless you’d like to use a video, previously provided, have a new one made,  “light a fire” under your recommended hire on this project, that’s all we can do this week.

Amstelvar Alpha is very fixed, so that will work for demonstrations of the idea

I've added .truevalues file processing. Currently this only exists for Amstelvar, and the values don't seem particularly right... but it's working correctly according to the values it has, as far as I can tell.

@graphicore only Amstelvar Roman has TrueValues functioning in the latest appspot instance; as a medium term goal I'd love to see TrueValues working on all the fonts pre-loaded, for all 4-5 user axes (weight, width, optical size, slant, grade)

@davelab6 what's the definition of TrueValues? Also, there's only one .truevalues XML file in the sources, the one for Amstelvar-Roman. If I get the other files, it should be easy.

True values are the actual parametric values of the registered axes all,over the designspace, making it possible for the sliders of the parametric axes to move as the registered axes are changed by the user.
This was only demonstrated to display the relationship. Moving the sliders of parametric axes while in this mode was problematic.
On Apr 28, 2022, at 1:59 PM, Lasse Fister @.***> wrote:

@davelab6 what's the definition of TrueValues? Also, there's only one .truevalues XML file in the sources, the one for Amstelvar-Roman. If I get the other files, it should be easy.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.


Add debugDescription methods to UBSignal and UBSignalObserver

Add some missing pragmas too

Wanna write some tests for the debugDescriptors?

I love debugging

LGTM, can add tests for debugDescriptors later.


Update readme script

Update readme script to save file when building FAISS (exact search) index

Gensim < 4.0 is needed for this version of code


Add user zoom controls

Ctrl + and Ctrl ] for zooming in
Ctrl - and Ctrl [ for zooming out
Very simple

Because I'm exporting on MacOS?? duh

Also for future use

Try adding the action through the g
Godot editor instead of hardcoding them. Go to Project > Project Settings > Input Map.
Also, no macOS preset, and why change the shake strength from 15 to 10?

mb. i forgot to change the shake strength back. you could also remove the macos preset if you want

also it'd be VERY NICE if you said this an hour ago, before i hardcoded them

Closing because it goes against Griddycode's goal


Ubuntu 22.04 + ROCm 5.4.2, RX 480 not detected

Hello,
I am trying to use 4 rx480s for pytorch with rocm 5.4.2. sudo lshw -C display shows all the gpus, but clinfo, rocminfo, and torch.cuda.is_available() show 0 gpus. Is rx480 supported by rocm? If so, how do I get it to be detected? If not, is there any other way to use with pytorch on Ubuntu?
lshw:
  *-display
       description: VGA compatible controller
       product: CoffeeLake-S GT2 [UHD Graphics 630]
       vendor: Intel Corporation
       physical id: 2
       bus info: pci@0000:00:02.0
       logical name: /dev/fb0
       version: 00
       width: 64 bits
       clock: 33MHz
       capabilities: pciexpress msi pm vga_controller bus_master cap_list rom fb
       configuration: depth=32 driver=i915 latency=0 mode=1600x900 resolution=1600,900 visual=truecolor xres=1600 yres=900
       resources: iomemory:600-5ff iomemory:400-3ff irq:130 memory:6071000000-6071ffffff memory:4000000000-400fffffff ioport:6000(size=64) memory:c0000-dffff
  *-display
       description: VGA compatible controller
       product: Ellesmere [Radeon RX 470/480/570/570X/580/580X/590]
       vendor: Advanced Micro Devices, Inc. [AMD/ATI]
       physical id: 0
       bus info: pci@0000:02:00.0
       version: cf
       width: 64 bits
       clock: 33MHz
       capabilities: pm pciexpress msi vga_controller bus_master cap_list rom
       configuration: driver=amdgpu latency=0
       resources: iomemory:410-40f iomemory:400-3ff irq:131 memory:4100000000-41ffffffff memory:4080000000-40801fffff ioport:4000(size=256) memory:50300000-5033ffff memory:50340000-5035ffff
  *-display
       description: VGA compatible controller
       product: Ellesmere [Radeon RX 470/480/570/570X/580/580X/590]
       vendor: Advanced Micro Devices, Inc. [AMD/ATI]
       physical id: 0
       bus info: pci@0000:03:00.0
       version: cf
       width: 64 bits
       clock: 33MHz
       capabilities: pm pciexpress msi vga_controller bus_master cap_list rom
       configuration: driver=amdgpu latency=0
       resources: iomemory:420-41f iomemory:430-42f irq:132 memory:4200000000-42ffffffff memory:4300000000-43001fffff ioport:3000(size=256) memory:50200000-5023ffff memory:50240000-5025ffff
  *-display
       description: VGA compatible controller
       product: Ellesmere [Radeon RX 470/480/570/570X/580/580X/590]
       vendor: Advanced Micro Devices, Inc. [AMD/ATI]
       physical id: 0
       bus info: pci@0000:04:00.0
       version: cf
       width: 64 bits
       clock: 33MHz
       capabilities: pm pciexpress msi vga_controller bus_master cap_list rom
       configuration: driver=amdgpu latency=0
       resources: iomemory:440-43f iomemory:430-42f irq:133 memory:4400000000-44ffffffff memory:4380000000-43801fffff ioport:2000(size=256) memory:50100000-5013ffff memory:50140000-5015ffff
  *-display
       description: VGA compatible controller
       product: Ellesmere [Radeon RX 470/480/570/570X/580/580X/590]
       vendor: Advanced Micro Devices, Inc. [AMD/ATI]
       physical id: 0
       bus info: pci@0000:05:00.0
       version: cf
       width: 64 bits
       clock: 33MHz
       capabilities: pm pciexpress msi vga_controller bus_master cap_list rom
       configuration: driver=amdgpu latency=0
       resources: iomemory:450-44f iomemory:460-45f irq:134 memory:4500000000-45ffffffff memory:4600000000-46001fffff ioport:1000(size=256) memory:50000000-5003ffff memory:50040000-5005ffff

clinfo:
Number of platforms:				 1
  Platform Profile:				 FULL_PROFILE
  Platform Version:				 OpenCL 2.1 AMD-APP (3513.0)
  Platform Name:				 AMD Accelerated Parallel Processing
  Platform Vendor:				 Advanced Micro Devices, Inc.
  Platform Extensions:				 cl_khr_icd cl_amd_event_callback

  Platform Name:				 AMD Accelerated Parallel Processing
Number of devices:				 0

rocminfo:
ROCk module is loaded
=====================
HSA System Attributes
=====================
Runtime Version:         1.1
System Timestamp Freq.:  1000.000000MHz
Sig. Max Wait Duration:<PHONE_NUMBER>3709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model:           LARGE
System Endianness:       LITTLE

==========
HSA Agents
==========
*******
Agent 1
*******
  Name:                    Intel(R) Core(TM) i3-9100 CPU @ 3.60GHz
  Uuid:                    CPU-XX
  Marketing Name:          Intel(R) Core(TM) i3-9100 CPU @ 3.60GHz
  Vendor Name:             CPU
  Feature:                 None specified
  Profile:                 FULL_PROFILE
  Float Round Mode:        NEAR
  Max Queue Number:        0(0x0)
  Queue Min Size:          0(0x0)
  Queue Max Size:          0(0x0)
  Queue Type:              MULTI
  Node:                    0
  Device Type:             CPU
  Cache Info:
    L1:                      32768(0x8000) KB
  Chip ID:                 0(0x0)
  ASIC Revision:           0(0x0)
  Cacheline Size:          64(0x40)
  Max Clock Freq. (MHz):   4200
  BDFID:                   0
  Internal Node ID:        0
  Compute Unit:            4
  SIMDs per CU:            0
  Shader Engines:          0
  Shader Arrs. per Eng.:   0
  WatchPts on Addr. Ranges:1
  Features:                None
  Pool Info:
    Pool 1
      Segment:                 GLOBAL; FLAGS: FINE GRAINED
      Size:                    8042664(0x7ab8a8) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
    Pool 2
      Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
      Size:                    8042664(0x7ab8a8) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
    Pool 3
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED
      Size:                    8042664(0x7ab8a8) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
  ISA Info:
*** Done ***

Hi @mystsec, RX480 is no longer supported.
Link to latest version of ROCm.
https://rocm.docs.amd.com/en/latest/
Link to previous releases of ROCm.
https://rocm.docs.amd.com/en/latest/release/versions.html


Leverage Maven's -Dstyle.color to avoid coloring instead of stripping the ASCII codes in the client

Fix #373

I don't think leveraging the somewhat broken (atm) style.color option is a good idea, see for example https://issues.apache.org/jira/browse/MNG-6380?filter=12350130.  I've raised a bunch of PR in maven to fix the coloring, but I don't think they'll be integrated in 3.x branch, but rather 4.x.  And I don't really want to wait for maven 4.x for those fixes in mvnd.
Also, tying the no-color option to non-buffering does not seem a benefit.  The default buffering mode allows to read the output much more easily (per project logs are grouped together).
What's the problem with the current approach ?

I don't think leveraging the somewhat broken (atm) style.color option is a good idea, see for example https://issues.apache.org/jira/browse/MNG-6380?filter=12350130.

With my current proposal, mvnd -B -Dstyle.color=always package | less -R outputs colors, as expected. Explicit -Dstyle.color is always honored, and -B does not override it in any way.

I've raised a bunch of PR in maven to fix the coloring, but I don't think they'll be integrated in 3.x branch, but rather 4.x. And I don't really want to wait for maven 4.x for those fixes in mvnd.
Also, tying the no-color option to non-buffering does not seem a benefit.

No, I do not tie no-color to noBuffering. You probably saw this location https://github.com/mvndaemon/mvnd/pull/371/files#diff-2e0b63101a0e22feb7df48ea1a5e83e867c4d076bb72ea4eeccfc5a79dca756bR117 but there, noBuffering is i.a. triggered by !stdoutIsTerminal which IMO is the right thing to do. I admit, it might have been a separate commit, because it is not directly related to coloring. I am open to discuss it.

The default buffering mode allows to read the output much more easily (per project logs are grouped together).

I agree, and do not think I did anything against the benefits the default buffering mode brings.

What's the problem with the current approach ?

Originally, I had two motivations for these changes (and when implementing another two appeared):

I did not like that we add the new --color option for something that was solvable by available Maven options. I originally thought we could use -B under the hood (and then learned about -Dstyle.color which is better). Now that I know that --color is proposed for Maven 4, it is actually quite acceptable and if you like, I would not mind adding --color back (keeping -Dstyle.color as its alias).

I found the state after https://github.com/mvndaemon/mvnd/pull/359 a bit ineffective: we instruct the daemon to produce colored messages and then we strip colors in the client https://github.com/mvndaemon/mvnd/pull/359/files#diff-3bfeaf9064d5d02331090c71be3e33960138f55d4fc7cc11fb692a587b30cb91R842 Isn't it more effective to tell the daemon whether we want colored messages upfront using -Dstyle.color=always or -Dstyle.color=never? You seem to imply that -Dstyle.color does not work properly. You have pointed to an example with -B and -Dstyle.color=always combined, but as I mentioned above, that case works well with the current proposal, IMO. Maybe you know more situations where -Dstyle.color does not work as expected?

I have found a related bug that existed both before and after https://github.com/mvndaemon/mvnd/pull/359 : Once the daemon was started with -B option, all subsequent invocations, albeit without -B, were colorless. The current PR solves this problem by always sending an explicit -Dstyle.color=always or -Dstyle.color=never to the daemon.

As noted above, !stdoutIsTerminal should trigger noBuffering.

and 4. are separate problems and I would not mind discussing them separately, after we find an agreement about 1. and 2.

I did not like that we add the new --color option for something that was solvable by available Maven options. I originally thought we could use -B under the hood (and then learned about -Dstyle.color which is better). Now that I know that --color is proposed for Maven 4, it is actually quite acceptable and if you like, I would not mind adding --color back (keeping -Dstyle.color as its alias).

Ok. I'd rather keep the --color option as an alias.

I found the state after #359 a bit ineffective: we instruct the daemon to produce colored messages and then we strip colors in the client https://github.com/mvndaemon/mvnd/pull/359/files#diff-3bfeaf9064d5d02331090c71be3e33960138f55d4fc7cc11fb692a587b30cb91R842 Isn't it more effective to tell the daemon whether we want colored messages upfront using -Dstyle.color=always or -Dstyle.color=never? You seem to imply that -Dstyle.color does not work properly. You have pointed to an example with -B and -Dstyle.color=always combined, but as I mentioned above, that case works well with the current proposal, IMO. Maybe you know more situations where -Dstyle.color does not work as expected?

We'd have to verify the behavior of MNG-6719

I have found a related bug that existed both before and after #359 : Once the daemon was started with -B option, all subsequent invocations, albeit without -B, were colorless. The current PR solves this problem by always sending an explicit -Dstyle.color=always or -Dstyle.color=never to the daemon.

Yes, that one should be fixed.  My idea was to always have coloring on the server side, but it's true that unless the system property is always set explicitly, it could keep the previous values, which is wrong.

As noted above, !stdoutIsTerminal should trigger noBuffering.

I don't think I agree on that one.  I often redirect the output to a log file, but that does not mean I want the project's output to be interleaved.

Points 3. and 4. are separate problems and I would not mind discussing them separately, after we find an agreement about 1. and 2.

Can we move #4 out.  Also, I think constants would be nice to have for always, never and auto values.

I did not like that we add the new --color option for something that was solvable by available Maven options. I originally thought we could use -B under the hood (and then learned about -Dstyle.color which is better). Now that I know that --color is proposed for Maven 4, it is actually quite acceptable and if you like, I would not mind adding --color back (keeping -Dstyle.color as its alias).

Ok. I'd rather keep the --color option as an alias.

OK, let me add it back.

I found the state after #359 a bit ineffective: we instruct the daemon to produce colored messages and then we strip colors in the client https://github.com/mvndaemon/mvnd/pull/359/files#diff-3bfeaf9064d5d02331090c71be3e33960138f55d4fc7cc11fb692a587b30cb91R842 Isn't it more effective to tell the daemon whether we want colored messages upfront using -Dstyle.color=always or -Dstyle.color=never? You seem to imply that -Dstyle.color does not work properly. You have pointed to an example with -B and -Dstyle.color=always combined, but as I mentioned above, that case works well with the current proposal, IMO. Maybe you know more situations where -Dstyle.color does not work as expected?

We'd have to verify the behavior of MNG-6719

I have verified that mvn verify | tee mvn.log when executed on Windows with git (bash) does not write the color escape to the file.

I have found a related bug that existed both before and after #359 : Once the daemon was started with -B option, all subsequent invocations, albeit without -B, were colorless. The current PR solves this problem by always sending an explicit -Dstyle.color=always or -Dstyle.color=never to the daemon.

Yes, that one should be fixed. My idea was to always have coloring on the server side, but it's true that unless the system property is always set explicitly, it could keep the previous values, which is wrong.

Filed https://github.com/mvndaemon/mvnd/issues/373 so that we have it in the change log.

4 As noted above, !stdoutIsTerminal should trigger noBuffering.

I don't think I agree on that one. I often redirect the output to a log file, but that does not mean I want the project's output to be interleaved.

Yes, thanks, that sounds as a valid reason. Let me remove it.

Also, I think constants would be nice to have for always, never and auto values.

+1

All done, could you please review?


Create student.txt

CooneyTech Robotics Team

This seems to be a school district email; this dataset is for post-secondary (higher education) institutions only. Closing.


On Windows, don't attach to current console session

Before, on windows the electron console output may not lead to stdout so electron-rebuilt failed to detect version.
setting the ELECTRON_NO_ATTACH_CONSOLE env variable fix this
Fixes #37

+1 for this

Thanks! Sorry I'm totally spacing on these PRs


Missing cross-certificates from list

Hi,
I was looking at the list of trusted cross-certificates and their expiration date. I'm a bit confused as some seem to be missing from the list. For example, this one:

Issued to: Microsoft Code Verification Root
Issued by: Microsoft Code Verification Root
Expires:   Sat Nov 01 14:54:03 2025
SHA1 hash: 8FBE4D070EF8AB1BCCAF2A9D5CCAE7282A2C66B3

    Issued to: USERTrust RSA Certification Authority
    Issued by: Microsoft Code Verification Root
    Expires:   Tue Jul 22 22:03:49 2025
    SHA1 hash: BEB914182A829D0C69556907E37F2799BFFFD891

        Issued to: Sectigo RSA Code Signing CA
        Issued by: USERTrust RSA Certification Authority
        Expires:   Wed Jan 01 00:59:59 2031
        SHA1 hash: 94C95DA1E850BD85209A4A2AF3E1FB1604F9BB66

I'm not sure if this is intentional, and the list is only partial.
The expiration is also set to Tue Jul 22 22:03:49 2025, unlike the other ones in the list which all expire before July 1, 2021.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 0e326d3e-833a-e312-d35d-11569690c7e6
Version Independent ID: 7a9795cf-093e-a67f-8162-b832e1161ba8
Content: Deprecation of Software Publisher Certificates, Commercial Release Certificates, and Commercial Test Certificates - Windows drivers
Content Source: windows-driver-docs-pr/install/deprecation-of-software-publisher-certificates-and-commercial-release-certificates.md
Product: windows-hardware
Technology: windows-devices
GitHub Login: @tedhudek
Microsoft Alias: tedhudek

Actually, I think this may have already been answered here https://github.com/MicrosoftDocs/windows-driver-docs/issues/2713


Resolve nested fields

When responding to a query, IX-API server does not provide a mechanism to automatically resolve nested field IDs into objects.
Taking the case of an IP, a valid response looks like:
{
  "managing_customer": "238189294",
  "consuming_customer": "2381982",
  "external_ref": "IX:Service:23042",
  "id": "string",
  "version": 0,
  "address": "<IP_ADDRESS>",
  "prefix_length": 23,
  "fqdn": "string",
  "valid_not_before": "2019-08-24T14:15:22Z",
  "valid_not_after": "2019-08-24T14:15:22Z"
}

Both managing_customer and consuming_customer are actually fields pointing to customer objects via IDs.  To get details for each of these objects, we need to perform queries to the customers endpoint filtering the result by IDs.
We may want to also make nested field resolving optional by setting an attribute in the query like depth=<integer> where 0 means "don't resolve nested fields".
Nested fields must be identified for each object as well as the type of those fields.

Closing this as it would have a drastic performance impact. Data sets should be small enough to store them and work in memory.


Replacing 8-byte-long length prefix with varint

@moozzyk you forgot to enable unsafe code.

A few things:

We need to update the protocol spec saying what algorithm we're using for variable sized ints.
We should put a comment in the source code pointing to the spec.

Weird I though I enabled unsafe and build passed locally. Will look into it.

@moozzyk You used the UI and it ended up in the debug only section. never use the UI


Using RFID MFRC522 on SPI1_3CS

Hi,
I am trying to get the RFID-reader RC522 working on SPI1. By default SPI0 is used, but since I am using this display https://www.waveshare.com/wiki/3.5inch_RPi_LCD_(A) the default SPI port is allready used.
I added the following line in the boot/config.txt
dtoverlay=spi1-3cs
and ls /dev/spi* shows
/dev/spidev1.0  /dev/spidev1.1  /dev/spidev1.2
But I cant get it working. Do you have any Idea?

Did you check https://github.com/MiczFlor/RPi-Jukebox-RFID/wiki/Wiring_for_RC522_card_reader , if this helps?


refactor: components plugin (#260)

@mercs600, should I then also rewrite this section and include everything individually?
const components = {
  T3CeMenuSitemap: T3CeMenuPages,
  T3CeMenuSitemapPages: T3CeMenuPages,
  T3CeMenuPages,
  T3CeMenuSubpages: T3CeMenuPages,
  T3CeMenuRelatedPages: T3CeMenuPages,
  T3CeMenuRecentlyUpdated: T3CeMenuPages,
  T3CeMenuSectionPages: T3CeMenuPages,
  T3CeMenuCategorizedPages: T3CeMenuPages,
}

@kubilaymelnikov good point, since we have them as separate files, we can rewrite this and include separately

@mercs600 there is a bug here which is related to (#254).
I'm still fixing that.

@kubilaymelnikov  we can merge it, but before - could you rebase your branch with master and squash your commits into one ?


"Fixed in XX, Affecting YY" should give some numbers in the email

We should add some numbers like:
This email contains all the bugs which has been fixed in nightly but still affected in other channels.
51 - X bugs
52 - Y bugs
53 - Z bugs

This is a bit awkward because it's implemented as a single query rather than one per channel, and the part that renders the email tries to be generic so doesn't "understand" the queries or versions etc.  Guess I'll have to think about this more.


Possible bug in regards to Vault and Standby Redirect

So, I'm using the vault backend, and when I hit the primary vault node, everything works.  However, if confd hits the node that is in "standby mode" confd is reporting the error, "FATAL EOF" when attempting token authentication.  It appears to be a result of the 307 redirect that the standby uses to redirect the request to the active vault node.

@cqcwillard update to the latest version of vault.  Much easier

This issue is not present with the latest version of Vault, closing.


nixos/matrix-synapse: wait for network-online.target

While network.target only guarantees that network devices have been created the network-online.target allows delaying service startup until after a configurable network state has been reached.
This should resolve spurious failures, e.g. when synapse tries to load the discovery information for its OIDC provider from a remote host.
Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

This is the error I saw. It crashed synapse and caused a dependency failure on the worker units.
synapse.http.client: [sentinel] Error sending request to  GET https://idm.lossy.network/oauth2/openid/synapse/.well-known/openid-configuration: ConnectError No route to host
synapse.app._base: [sentinel] Error during startup
Traceback (most recent call last):
  File "/nix/store/9aksrixmnzz2cmljhd7ym77q8z6wkzh6-matrix-synapse-1.92.1/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 140, in load_metadata
    await p.load_metadata()
  File "/nix/store/9aksrixmnzz2cmljhd7ym77q8z6wkzh6-matrix-synapse-1.92.1/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 588, in load_metadata
    return await self._provider_metadata.get()
  File "/nix/store/9aksrixmnzz2cmljhd7ym77q8z6wkzh6-matrix-synapse-1.92.1/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 136, in get
    return await self._cachedcall.get()
  File "/nix/store/9aksrixmnzz2cmljhd7ym77q8z6wkzh6-matrix-synapse-1.92.1/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 106, in get
    self._result.raiseException()
  File "/nix/store/3j645f6l6rkyln6bi8zjfvairwqhh2m2-python3.10-twisted-22.10.0/lib/python3.10/site-packages/twisted/python/failure.py", line 504, in raiseException
    raise self.value.with_traceback(self.tb)
  File "/nix/store/3j645f6l6rkyln6bi8zjfvairwqhh2m2-python3.10-twisted-22.10.0/lib/python3.10/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
    result = context.run(
  File "/nix/store/3j645f6l6rkyln6bi8zjfvairwqhh2m2-python3.10-twisted-22.10.0/lib/python3.10/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/nix/store/9aksrixmnzz2cmljhd7ym77q8z6wkzh6-matrix-synapse-1.92.1/lib/python3.10/site-packages/synapse/util/caches/cached_call.py", line 126, in _wrapper
    return await f()
  File "/nix/store/9aksrixmnzz2cmljhd7ym77q8z6wkzh6-matrix-synapse-1.92.1/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 600, in _load_metadata
    metadata_response = await self._http_client.get_json(url)
  File "/nix/store/9aksrixmnzz2cmljhd7ym77q8z6wkzh6-matrix-synapse-1.92.1/lib/python3.10/site-packages/synapse/http/client.py", line 559, in get_json
    body = await self.get_raw(uri, args, headers=actual_headers)
  File "/nix/store/9aksrixmnzz2cmljhd7ym77q8z6wkzh6-matrix-synapse-1.92.1/lib/python3.10/site-packages/synapse/http/client.py", line 644, in get_raw
    response = await self.request("GET", uri, headers=Headers(actual_headers))
  File "/nix/store/9aksrixmnzz2cmljhd7ym77q8z6wkzh6-matrix-synapse-1.92.1/lib/python3.10/site-packages/synapse/http/client.py", line 411, in request
    response = await make_deferred_yieldable(request_deferred)
twisted.internet.error.ConnectError: An error occurred while connecting: 113: No route to host.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nix/store/9aksrixmnzz2cmljhd7ym77q8z6wkzh6-matrix-synapse-1.92.1/lib/python3.10/site-packages/synapse/app/_base.py", line 254, in wrapper
    await cb(*args, **kwargs)
  File "/nix/store/9aksrixmnzz2cmljhd7ym77q8z6wkzh6-matrix-synapse-1.92.1/lib/python3.10/site-packages/synapse/app/homeserver.py", line 362, in start
    await oidc.load_metadata()
  File "/nix/store/9aksrixmnzz2cmljhd7ym77q8z6wkzh6-matrix-synapse-1.92.1/lib/python3.10/site-packages/synapse/handlers/oidc.py", line 144, in load_metadata
    raise Exception(
Exception: Error while initialising OIDC provider 'oidc-kanidm'

Wondering if we should do the same on 23.05.
IIRC network.target is not what you need in 99% of all cases anyways.

And, apart from after, we should also add a want dependency, per https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/


Support multiple CNI implementations

Let's start with Flannel and Calico

Done (the implementation is spread across many commits...)


client:settimeout doesn't work

First of all, thanks for the great work.  I am able to convert my single connection server into multi-connection in no time.
My server is kind of reverse of traditional servers. This is a simulator to send hardware messages to the connected clients where  server sends messages and occasionally receives messages the clients.
In my Handler, I have SendMessage() and ReceiveMessage() with timeout 0.1 sec just to make sure I read pending messages if any.
If I don't wrap the socket in the copas, time out on client:receive() works fine but if I wrap the socket in copas, timeout doesn't work. It waits there indefinably.  I even tried swapping the socket wrap and socket:settimeout function but didn't help.
Any help would be appreciated.

-------------------------------------------------------------------------
-- ReceiveMessage : Receive the message from client if available
-- @param client: socket
-------------------------------------------------------------------------
function ReceiveMessage(client)

     --set the timeout
     client:settimeout(0.1, 't')

    -- wraps socket into copas
     client = copas.wrap(client)

   --read the header of 8 bytes
    local header, status = client:receive(8)

     if(header== nil) then
       return
    else
      -- get the payload size from the header
    local length = tonumber(string.byte(header,3))

      -- read the payload
    local payload, status, partial = client:receive(length)
    end
end

Is this issue still alive or can it be closed (considering it's from 2012)?

It is still an issue. The settimeout function of Copas wrapped sockets works different to bare sockets.
It is more a boolean thing:
0: don't wait, return what has been received so far
any other value: wait until you get data; this is equivalent to settimeout(-1) on bare sockets.
It is not possible to set a limited time-out value on Copas sockets.
I am going to implement a version that implements the setttimeout() of Copas sockets functionally equivalent to the Luasocket sockets. (without the 'b' / 't' flag).
Is this interesting to anyone else?

@okroth I've been thinking about this as well see #33 just never came around to implement it (or had an immediate need for it)
I think copas should handle its own internal timeouts using timers. Main problem here is a performant timer implementation (extremely performant when setting or cancelling, and might be slow when executing) something like a timerwheel.

I think there was a way to get the current timeout from a LuaSocket socket, maybe that can be reused.

I think eventually we should modify the Copas library to always wrap the
LuaSockt object int a Copas socket and work on these objects only.

it already does that, or didn't I get your message right?

As well one could safely change UDP timeouts. Currently, if one thread
changes the timeout on "its" Copas.UDP socket, it changes it for all.

This would mean that the timeouts are not related to the sockets, but to the threads. Not sure though. I like the concept that the Copas sockets are a drop-in replacement for luasocket sockets. So we might just want to mimic that behaviour

Regarding an efficient timer implementation, I like to refer to the
binary heap implementation, with an inverted index

That is not efficient. Modifications on a binary tree always result in a bubble-up-down being executed to resort the tree. It's very efficient for timers that eventually execute. But in socket operations the timeouts are just fallbacks, so in best case scenario's they never get executed, only scheduled and cancelled.
For a few connections it wouldn't really matter, but when scaling, a timer-wheel will really out perform the binary tree.

Hi Thijs,
I made a few comments:

I think there was a way to get the current timeout from a LuaSocket
socket, maybe that can be reused.

Unfortunately not. LuaSocket implements settimeout() but not gettimeout().
Additionally, Copas calls settimeout( 0 ) on all sockets it wraps, so we
can't store any value there.

I think eventually we should modify the Copas library to always
wrap the
LuaSockt object int a Copas socket and work on these objects only.

it already does that, or didn't I get your message right?

It wraps the sockets, but also leaves access to, e.g. copas.send(
socket, ...) with socket being a bare LuaSocket socket.
This should be hidden to avoid bypassing the copas.socket.send() with
copas.send( copas.socket.socket,...)

As well one could safely change UDP timeouts. Currently, if one thread
changes the timeout on "its" Copas.UDP socket, it changes it for all.

This would mean that the timeouts are not related to the sockets, but
to the threads. Not sure though. I like the concept that the Copas
sockets are a drop-in replacement for luasocket sockets. So we might
just want to mimic that behaviour

In the mean time I found that it is meaningless for UDP anyway. It is
always the same socket on UDP so it's OK to change the timeout.
For TCP the problem did not exist.

Regarding an efficient timer implementation, I like to refer to the
binary heap implementation, with an inverted index

That is not efficient. Modifications on a binary tree always result in
a bubble-up-down being executed to resort the tree. It's very
efficient for timers that eventually execute. But in socket operations
the timeouts are just fallbacks, so in best case scenario's they never
get executed, only scheduled and cancelled.
For a few connections it wouldn't really matter, but when scaling, a
timer-wheel will really out perform the binary tree.

The binary tree is very fast on checking the next timer to expire: O(1)
So if one frequently has to test, the performance gain is here. Also the
bubble and sink operations have O(log n), so scaling is good here as well.
Actually, one has two bubble+sink operations per send/receive/sleep
operation, regardless of whether it timed out or not.
Plus one or more lookups per operation (if several bare receives are
required to fulfil the Copas receive operation)
As the time-out times are absolute, no reordering happens here.
A time wheel works best on a interval base, where the scheduler looks at
every interval onto the wheel at the current time slot and then executes
all registered tasks.
This requires that time is rounded to ticks.
And it requires that at every tick the wheel is checked.
Thus it needs a regular tick based scheduling.
Copas works different.
It goes to sleep until something happens on I/O using the LuaSockets
select() function.
So it does not work until some I/O happens or the time-out of select()
expires.
Currently the time-out time is computed as the smaller of the next to
wake up sleeping task and the parameter given to copas.step().
So the granularity of the network time-outs is the parameter of copas.step()
I think that should be changed to that it takes the next to wake up
sleeping task or next due time-out or the timeout parameter, whatever
comes first.
Possibly, I will try to make a evaluation sample and find out...
--
Oliver

I think eventually we should modify the Copas library to always wrap the
LuaSockt object int a Copas socket and work on these objects only.

it already does that, or didn't I get your message right?

It wraps the sockets, but also leaves access to, e.g. copas.send(
socket, ...) with socket being a bare LuaSocket socket.
This should be hidden to avoid bypassing the copas.socket.send() with
copas.send( copas.socket.socket,...)

I don't like that amount of magic, if folks use the copas sockets, it should work, if they circumvent, they are on their own...

The binary tree is very fast on checking the next timer to expire: O(1)
So if one frequently has to test, the performance gain is here. Also the
bubble and sink operations have O(log n), so scaling is good here as well.
Actually, one has two bubble+sink operations per send/receive/sleep
operation, regardless of whether it timed out or not.
Plus one or more lookups per operation (if several bare receives are
required to fulfil the Copas receive operation)
As the time-out times are absolute, no reordering happens here.

Assume I have 5 timeouts, in 10, 20, 30, 40 and 50 seconds from now.
Now the socket for the 30 second timeout completes its operation, so Copas must cancel that timer. Removing the 30 second timer from the binary tree will cause a bubble sequence.
After reading, that same socket decides to return an answer by writing to the socket. Copas now insert the same 30 seconds timeout again in the binary tree. Hence it bubbles again.
And this will be the main operation; scheduling and cancelling, again and again. I'm not very much into O notations but I think a timerwheel does this a O(1), where binary tree does O(n).

A time wheel works best on a interval base, where the scheduler looks at
every interval onto the wheel at the current time slot and then executes
all registered tasks.
This requires that time is rounded to ticks.
And it requires that at every tick the wheel is checked.
Thus it needs a regular tick based scheduling.
Copas works different.
It goes to sleep until something happens on I/O using the LuaSockets
select() function.
So it does not work until some I/O happens or the time-out of select()
expires.
Currently the time-out time is computed as the smaller of the next to
wake up sleeping task and the parameter given to copas.step().
So the granularity of the network time-outs is the parameter of copas.step()

No, copas already supports timers. So the select call it does has a timeout maximized at the next timer that expires, or step size, whichever comes first.

Hi Thijs,

I don't like that amount of magic, if folks use the copas sockets, it
should work, if they circumvent, they are on their own...

It's about the same amount of magic as LuaSSL adds to LuaSocket. Just
enough. :-)

As the time-out times are absolute, no reordering happens here.

Assume I have 5 timeouts, in 10, 20, 30, 40 and 50 seconds from now.
Now the socket for the 30 second timeout completes its operation, so
Copas must cancel that timer. Removing the 30 second timer from the
binary tree will cause a bubble sequence.
After reading, that same socket decides to return an answer by writing
to the socket. Copas now insert the same 30 seconds timeout again in
the binary tree. Hence it bubbles again.
And this will be the main operation; scheduling and cancelling, again
and again. I'm not very much into O notations but I think a timerwheel
does this a O(1), where binary tree does O(n).

The timer wheel, at least the implementations I came across, are based
on fixed interval sizes. And on each interval, the wheel has to be
checked for scheduled events.
Very nice if the scheduler is based on something like a timer interrupt
(e.g. every 10ms).
If one has a scheduler with variable timing, the scheduler has to look
to all slots from last to current tick, so this is not O(1), which means
constant effort.
It's more O(n), so linearly with number of events. Inserting and
removing events is dependent on the data structure within the slots,
maybe O(log n)
There were some discussions on the Lua list about the efforts on adding
a table element (not table.insert!).
O(log n) is one of the best implementations that one can get. It means
that you can scale from 10 to 100 and only need twice the time, go to
10000 and use four times the time.
With O(n) you can go only to 40 with same time.
With O( n*log n) you can manage 28 items (this is the effort of quicksort)
With O(n²) you reach the limit at 20 items
So if one thing scales well, it's O(log n)
BTW, the algorithm that assembles the tables for socket.select(), is O(n)...
--
Oliver

as the timeouts are fallbacks, the tick can be like 0.1 second, doesn't need to be precise.
Allocate tables with subtables. First table holds slots with 30 second precision, 240 entries for 2 full hours. Each subtable contains 300 entries (30 timespan / 0.1 tick size). Final (3rd) table holds all threads timing out there, in that time slot.
Set the step to ticksize, and now every cancel/schedule operation is reduced to 3 table lookups. No matter how you scale.

Hi,
I am using copas library and set timeout still doesn't work. Is there any fix to this?
Thanks,
Aditya

@Aditya23456 unfortunately not, I've written a PoC timer library for this purpose but ran out of time when actually implementing it with Copas.

I am using copas library and set timeout still doesn't work. Is there
any fix to this?

Thanks,
Aditya

Did you try to call copas.step() with a smaller timeout value?
Copas uses the smaller of this and the time to the next thread to wake
from sleep as timeout value to socket.select().
--
Oliver

Hi, I am using this example
local copas = require("copas")
h=require("copas.http")
h.TIMEOUT=5 --made time out to 5 seconds
local asynchttp = h.request
local URLlist = {
"http://www.google.com",
"http://www.microsoft.com",
"http://www.apple.com",
}
local handler = function(host)
res, err = asynchttp(host)
end
for _, host in ipairs(list) do copas.addthread(handler, host)
end
copas.loop()
Here, URLlist is list of devices. If one of my device goes offline copas.loop takes like 2 minutes to exit. I don't know how to use step in this condition. I see this function in copas
function copas.loop(timeout)
while not copas.finished() do copas.step(timeout) end
end
So i changed copas.loop() to copas.loop(5) but nothing worked. I see copas.finished() returns false when all 3 -
next(_reading) or next(_writing) or _sleeping:getnext()) are 0. With some print statements I found when a device goes offline, _writing is stuck with that device for longer time than needed. I think I can simply  write my own function copas.loop2() which breaks while finished and pops/makes next(_reading) or next(_writing) or _sleeping:getnext())=0 after a timeout. Does this sound reasonable?
Thanks,
Aditya