Disable bookend optimization in nvfuser

🚀 Feature
Bookend optimization was implemented when nvfuser makes unnecessary copy of tensor going through meta operations (view, reshape, transpose) that could be resolved through view.
Since nvfuser now does support alias operations, we would want to turn bookend optimization off by default.
Motivation
It is beneficial to have meta operations being inside fusion. reshape is a good example here.
  # t_2: "cuda:0 f32[2, 16, 8, 16]"
  t1 = torch.permute(t_2, [0, 2, 1, 3])  # t1: "cuda:0 f32[2, 8, 16, 16]"
    # t1 = ltorch.permute(t_2, [0, 2, 1, 3])  # t1: "cuda:0 f32[2, 8, 16, 16]"
      # t1 = prims.transpose(t_2, (0, 2, 1, 3))  # t1: "cuda:0 f32[2, 8, 16, 16]"
  del t_2
  t3 = torch.reshape(t1, (2, 8, 256))  # t3: "cuda:0 f32[2, 8, 256]"
    # t3 = ltorch.reshape(t1, (2, 8, 256))  # t3: "cuda:0 f32[2, 8, 256]"
      # t3 = prims.reshape(t1, (2, 8, 256))  # t3: "cuda:0 f32[2, 8, 256]"

e.g. Reshape could ended up being a real data copy, we would want to fuse that with other operations. If t3 is input to a fusion, we can fuse the permute/reshape with other operations; similarly, if t_2 is an output from fusion, codegen could make a better decision on whether to fuse the reshape with previous operation or to produce t_2 in a format that the following permute+reshape does't trigger memory copy.
Pitch
The promise we (@wujingyue) made earlier:

we'll bring more clarity to CPU latency and disable bookending with blast radius controlled.

Actionable item:

[ ] check functional issue with bookend disabled;
[ ] track einsum performance after disabling bookend optimization #193 ;
[ ] nvfuser backend host latency.

cc @tfogal @apaz-cli

For the first action item, https://github.com/Lightning-AI/lightning-thunder/pull/206 triggers the following CI errors.
FAILED thunder/tests/test_examine_memory.py::test_nanogpt_block_nvfuser_cuda_float32 - AssertionError: assert 235985920 == 242277376
 +  where 242277376 = sum(dict_values([6291456, 3072, 9216, 3072, 3072, 12288, 3072, 7077888, 9437184, 2359296, 9437184, 3072, 3072, 12582912, 18874368, 18874368, -18874368, 100663296, 100663296, -100663296, 6291456, 6291456, -6291456, 6291456, 12599296, -6291456, 25165824, 25165824, 6291456, 6291456, -6291456]))
 +    where dict_values([6291456, 3072, 9216, 3072, 3072, 12288, 3072, 7077888, 9437184, 2359296, 9437184, 3072, 3072, 12582912, 18874368, 18874368, -18874368, 100663296, 100663296, -100663296, 6291456, 6291456, -6291456, 6291456, 12599296, -6291456, 25165824, 25165824, 6291456, 6291456, -6291456]) = <built-in method values of dict object at 0x7fc8e07a5600>()
 +      where <built-in method values of dict object at 0x7fc8e07a5600> = {'del t16': -18874368, 'del t29': -100663296, 'del t49': -6291456, 'del t53': -6291456, ...}.values
FAILED thunder/tests/test_examine_memory.py::test_view_ops_nvfuser_cuda_float32 - assert 128 == 144
FAILED thunder/tests/test_ops.py::test_core_vs_torch_consistency_getitem_nvfuser_cuda_float64 - ValueError: Trying to extract a dtype from object slice(3, 1, None) with unknown type <class 'slice'>!
FAILED thunder/tests/test_ops.py::test_core_vs_torch_consistency_getitem_nvfuser_cuda_float16 - ValueError: Trying to extract a dtype from object slice(3, 1, None) with unknown type <class 'slice'>!
FAILED thunder/tests/test_ops.py::test_core_vs_torch_consistency_getitem_nvfuser_cuda_bool8 - ValueError: Trying to extract a dtype from object slice(3, 1, None) with unknown type <class 'slice'>!
FAILED thunder/tests/test_ops.py::test_core_vs_torch_consistency_getitem_nvfuser_cuda_int64 - ValueError: Trying to extract a dtype from object slice(3, 1, None) with unknown type <class 'slice'>!
FAILED thunder/tests/test_ops.py::test_core_vs_torch_consistency_getitem_nvfuser_cuda_bfloat16 - ValueError: Trying to extract a dtype from object slice(3, 1, None) with unknown type <class 'slice'>!
FAILED thunder/tests/test_ops.py::test_core_vs_torch_consistency_getitem_nvfuser_cuda_int32 - ValueError: Trying to extract a dtype from object slice(3, 1, None) with unknown type <class 'slice'>!
FAILED thunder/tests/test_ops.py::test_core_vs_torch_consistency_getitem_nvfuser_cuda_float32 - ValueError: Trying to extract a dtype from object slice(3, 1, None) with unknown type <class 'slice'>!
FAILED thunder/tests/test_jit_general.py::test_litgpt_variants[cuda-falcon-40b-like] - RuntimeError: inp->definition() && inp->definition()->isA<PadOp>() INTERNAL ASSERT FAILED at "/Fuser/csrc/preseg_passes/remove_empty.cpp":256, please report a bug with repro script to NVFuser at https://github.com/NVIDIA/Fuser/issues. Inputs to CatOp must be outputs of PadOps

Go figure...

getitem_nvfuser tests failed for a similar reason to https://github.com/Lightning-AI/lightning-thunder/blob/54bb6146ff757905925f8d9ea2197870c4971011/thunder/tests/opinfos.py#L3113-L3115. I can again create a wrapper so slice objects don't get passed to FusionDefinitionWrapper. But I'd love to hear thoughts from @jjsjann123 and @kevinstephano who may have a better fix.

FAILED thunder/tests/test_examine_memory.py::test_view_ops_nvfuser_cuda_float32 - assert 128 == 144 is due to golden testing. 128 is less memory than 144, so it's in fact an improvement.

FAILED thunder/tests/test_jit_general.py::test_litgpt_variants[cuda-falcon-40b-like] - RuntimeError: inp->definition() && inp->definition()->isA<PadOp>() INTERNAL ASSERT FAILED at "/Fuser/csrc/preseg_passes/remove_empty.cpp":256, please report a bug with repro script to NVFuser at https://github.com/NVIDIA/Fuser/issues. Inputs to CatOp must be outputs of PadOps  is gone after I rebase.
However, on my workstation, the tolerance 1e-5 seems to be too small for both TOT and this PR (1e-3 seems to be large enough). I'll see what CI says.
$ pytest thunder/tests/test_jit_general.py -k test_litgpt_variants[cuda-falcon-40b-like]
========================================================================================================================================================================================================================================= test session starts =========================================================================================================================================================================================================================================
platform linux -- Python 3.10.12, pytest-8.1.1, pluggy-1.5.0
Test order randomisation NOT enabled. Enable with --random-order or --random-order-bucket=<bucket_type>
benchmark: 4.0.0 (defaults: timer=torch.utils.benchmark.utils.timer.timer disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=True warmup_iterations=100000)
rootdir: /opt/pytorch/lightning-thunder
configfile: pyproject.toml
plugins: timestamper-0.0.10, xdist-3.5.0, random-order-1.1.1, cov-4.1.0, benchmark-4.0.0, hypothesis-6.100.0, timeout-2.2.0, anyio-4.3.0, shard-0.1.2
timeout: 900.0s
timeout method: signal
timeout func_only: False
collected 68 items / 67 deselected / 1 selected
Running 1 items in this shard

thunder/tests/test_jit_general.py F                                                                                [100%]

============================================================================================================================================================================================================================================== FAILURES ===============================================================================================================================================================================================================================================
_____________________________________________________________________________________________________________________________________________________________________________________________________________________________ test_litgpt_variants[cuda-falcon-40b-like] ______________________________________________________________________________________________________________________________________________________________________________________________________________________________

name = 'falcon-40b-like', device = device(type='cuda')

    @skipif_not_pytorch_2_1
    @pytest.mark.parametrize(
        "name",
        (
            "gpt-neox-like",
            "llama1-like",
            "long-context-like",
            "llama2-like",
            "falcon-7b-like",
            "falcon-40b-like",
            "codellama2-like",
            pytest.param(
                "mixtral-like",
                marks=pytest.mark.xfail(raises=(NotImplementedError, TypeError), reason="topk and where", strict=True),
            ),
        ),
    )
    @pytest.mark.parametrize(
        "device",
        ("cpu", "cuda", "meta"),
    )
    def test_litgpt_variants(name, device):
        if device == "cuda" and not torch.cuda.is_available():
            pytest.skip("CUDA not available")

        device = torch.device(device)

        x = torch.randint(0, 200, (5, 5), device=device)
        config = litgpt_model.Config.from_name(name)

        with device:
            reference = litgpt_model.GPT(config)
        expected_logits = reference(x)

        expected_logits.sum().backward()

        with device:
            model = litgpt_model.GPT(config)
        model.load_state_dict(reference.state_dict())
        tom = thunder.jit(model, executors=nvfuserex if device.type == "cuda" else torchex)
        actual_logits = tom(x)
>       assert_close(actual_logits, expected_logits)
E       AssertionError: Tensor-likes are not close!
E
E       Mismatched elements: 2153 / 12700 (17.0%)
E       Greatest absolute difference: 0.0003186464309692383 at index (3, 3, 33) (up to 1e-05 allowed)
E       Greatest relative difference: 0.4005630612373352 at index (1, 4, 143) (up to 1.3e-06 allowed)

thunder/tests/test_jit_general.py:654: AssertionError
======================================================================================================================================================================================================================================= short test summary info =======================================================================================================================================================================================================================================
FAILED thunder/tests/test_jit_general.py::test_litgpt_variants[cuda-falcon-40b-like] - AssertionError: Tensor-likes are not close!
============================================================================================================================================================================================================================ 1 failed, 67 deselected, 6 warnings in 7.70s =============================================================================================================================================================================================================================

Filed another blocker: https://github.com/Lightning-AI/lightning-thunder/issues/549

Yet-another blocker: https://github.com/NVIDIA/Fuser/issues/2362

These are all blockers that I can tell from the recent CI run. 🤞

The previous blockers have all been fixed. However, the most recent CI run failed with new errors -- number mismatches this time...
=========================== short test summary info ============================
FAILED thunder/tests/test_grad.py::test_vjp_correctness_sdpa_manual_grad_forward_scaled_dot_product_attention_nvfuser_cuda_float16 - AssertionError: Tensor-likes are not close!

Mismatched elements: 139 / 114688 (0.1%)
Greatest absolute difference: 0.001220703125 at index (0, 0, 15, 53) (up to 1e-05 allowed)
Greatest relative difference: 0.115966796875 at index (2, 0, 17, 11) (up to 0.001 allowed)
FAILED thunder/tests/test_grad.py::test_vjp_correctness_sdpa_manual_grad_forward_scaled_dot_product_attention_nvfuser_cuda_bfloat16 - AssertionError: Tensor-likes are not close!

Mismatched elements: 210005 / 212992 (98.6%)
Greatest absolute difference: 688.0 at index (0, 0, 48, 81) (up to 1e-05 allowed)
Greatest relative difference: inf at index (7, 0, 110, 32) (up to 0.016 allowed)
FAILED thunder/tests/test_grad.py::test_vjp_correctness_sdpa_manual_grad_forward_scaled_dot_product_attention_torch_cuda_bfloat16 - AssertionError: Tensor-likes are not close!

Mismatched elements: 209468 / 212992 (98.3%)
Greatest absolute difference: 848.0 at index (5, 1, 56, 19) (up to 1e-05 allowed)
Greatest relative difference: inf at index (4, 1, 109, 70) (up to 0.016 allowed)
FAILED thunder/tests/test_grad.py::test_vjp_correctness_sdpa_manual_grad_forward_scaled_dot_product_attention_torch_cuda_float16 - AssertionError: Tensor-likes are not close!

Mismatched elements: 210696 / 212992 (98.9%)
Greatest absolute difference: 704.0 at index (0, 1, 49, 13) (up to 1e-05 allowed)
Greatest relative difference: inf at index (0, 0, 2, 8) (up to 0.001 allowed)
FAILED thunder/tests/test_cudnn_executor.py::test_vjp_correctness_cudnn_sdpa[bfloat16-never-cat-grad-qkv] - AssertionError: Tensor-likes are not close!

Mismatched elements: 916 / 1310720 (0.1%)
Greatest absolute difference: 1.21875 at index (7, 1, 0, 96) (up to 0.2 allowed)
Greatest relative difference: inf at index (1, 1, 0, 2) (up to 0.02 allowed)
FAILED thunder/tests/test_cudnn_executor.py::test_vjp_correctness_cudnn_sdpa[float16-never-cat-grad-qkv] - AssertionError: Tensor-likes are not close!

Mismatched elements: 1582 / 1310720 (0.1%)
Greatest absolute difference: 1.0703125 at index (5, 0, 1, 39) (up to 0.2 allowed)
Greatest relative difference: inf at index (0, 1, 0, 2) (up to 0.02 allowed)
FAILED thunder/tests/test_cudnn_executor.py::test_vjp_correctness_cudnn_sdpa[bfloat16-may-cat-grad-qkv] - AssertionError: Tensor-likes are not close!

Mismatched elements: 1372 / 1310720 (0.1%)
Greatest absolute difference: 1.0859375 at index (6, 0, 1, 24) (up to 0.2 allowed)
Greatest relative difference: inf at index (1, 1, 0, 0) (up to 0.02 allowed)
FAILED thunder/tests/test_cudnn_executor.py::test_vjp_correctness_cudnn_sdpa[float16-may-cat-grad-qkv] - AssertionError: Tensor-likes are not close!

Mismatched elements: 1474 / 1310720 (0.1%)
Greatest absolute difference: 1.400390625 at index (2, 0, 1, 45) (up to 0.2 allowed)
Greatest relative difference: inf at index (0, 1, 0, 1) (up to 0.02 allowed)
= 8 failed, 4548 passed, 790 skipped, 91 xfailed, 93 xpassed, 110694 warnings in 975.39s (0:16:15) =
/usr/local/lib/python3.10/dist-packages/coverage/control.py:888: CoverageWarning: No data was collected. (no-data-collected)
  self._warn("No data was collected.", slug="no-data-collected")

##[error]Bash exited with code '1'.
Finishing: Testing: regular

Good news: these number mismatches no longer show up after I resync.
Bad news: distributed tests start to fail.
One error is NVDIA/Fuser#2395.
The other error seems to be that https://github.com/Lightning-AI/lightning-thunder/blob/c21533c12a2a826aee84e011c415b216cb6f779d/thunder/tests/distributed/test_ddp.py#L771-L777 expects slices and pads to be in the top-level trace, which are however fused into an nvFusion. cc @crcrpar

Thanks for identifying latest status, Jingyue!

Good news: these number mismatches no longer show up after I resync.

🎉

Bad news: distributed tests start to fail.

😢

One error is NVIDIA/Fuser#2395.
The other error seems to be that
https://github.com/Lightning-AI/lightning-thunder/blob/c21533c12a2a826aee84e011c415b216cb6f779d/thunder/tests/distributed/test_ddp.py#L771-L777
expects slices and pads to be in the top-level trace, which are however fused into an nvFusion. cc @crcrpar

Ahh, yeah that may have made sense before but less so now.
@crcrpar can we beg from your help here? Could you update the test to assert that the associated slice/pad are top level symbols or a fused executor took those ops? From thunder's point of view, either is acceptable.

I'll push more fixes tomorrow to tie up loose ends for CI.
Despite these pending CI failures, I was able to put some perf readings in https://github.com/Lightning-AI/lightning-thunder/pull/731, but I've yet to digest them. Caveat: I had to skip many benchmarks that OOM as is.

Closed by #731


Fix namespacing issues for links in render helpers

d3c0e0d8 removed the ability to use namespaces on the sortable tree helper. This reinstates that capability, but also adds consistent rendering and namespacing for all render helpers.

ce696f3a meant to clarify that a show link was not being used for a delete link, but neglected to update in all the helpers. This PR also addresses that issue.

Actually, this PR makes each renderer consistent in how it builds links, and allows you to explicitly pass what controller you want to use for building said links.

It was good PR. I have to keep it in mind if I will improve the gem.
Thanks again!


fix(repo): ensure index generates in the right order

fixes #1411

This change is

@technosophos strangely the ordering was correct before #1402 got merged, but I can't see what could have caused that in that PR.

@prydonius I know what happened there. I'm doing a quick look at the codebase to see if it makes more sense to sort here, or to sort in cmd/helm/repo_index.go

Okay, I think we should do this as implemented here in the change. Originally, I viewed sort as an extra step, but since we have made a strong comittment regarding ordering, it makes sense to have that commitment consistently enforced.

After discussing over a call, @technosophos decided it would make more sense to explicitly call sort in cmd/helm/repo_index.go. It will be more of a pain to consistently enforce an in-memory sorted index, since each method that modified the index would need to call sort.
PTAL

Yeah, this feels like the best direction for now. Thanks for updating all those tests so I don't accidentally do this again.


IDVA6-696 Health check end point

Link to Jira:
https://companieshouse.atlassian.net/browse/IDVA6-696
Changes:
Adds health check end point
Small update to our authentication middleware to allow whitelisted urls

CI: No security warnings found


Failures when testing the googleapis generated package (starting after the 10th test)

The generated tests in the generated googleapis package fails when using test_runner.
There are the steps to run the tests and see the failures:
$ git clone https://github.com/dart-lang/googleapis.git
$ cd googleapis
$ pub global activate test_runner
$ cd generated/googleapis
$ pub global run test_runner

The tests will now run, but only the first 10 will succeed.
Checking Dart SDK binaries...
Dart SDK binaries OK.

Looking for Dart project in "./"...
Found project "googleapis".

Looking for test suites...
Found 62 test suites (62 Standalone VM, 0 Dartium).

Running all tests...
Test suite passed: admin/email_migration_v2_test.dart
Test suite passed: adexchangeseller/v2_0_test.dart
Test suite passed: adexchangeseller/v1_1_test.dart
Test suite passed: adsensehost/v4_1_test.dart
Test suite passed: adexchangebuyer/v1_3_test.dart
Test suite passed: admin/reports_v1_test.dart
Test suite passed: adsense/v1_4_test.dart
Test suite passed: admin/directory_v1_test.dart
Test suite passed: appsactivity/v1_test.dart
Test suite failed: appstate/v1_test.dart
Detailed results of test suite appstate/v1_test.dart:
┌────────────────────────────────────────────────────
│
│ Unhandled exception:
│ No top-level getter 'main' declared.
│
│ NoSuchMethodError: method not found: 'main'
│ Receiver: top-level
│ Arguments: [...]
│ #0      NoSuchMethodError._throwNew (dart:core-patch/errors_patch.dart:176)
│ #1      _getMainClosure (dart:_builtin:97)
└────────────────────────────────────────────────────
Test suite failed: androidpublisher/v2_test.dart
Detailed results of test suite androidpublisher/v2_test.dart:
┌────────────────────────────────────────────────────────────
│
│ Unhandled exception:
│ No top-level getter 'main' declared.
│
│ NoSuchMethodError: method not found: 'main'
│ Receiver: top-level
│ Arguments: [...]
│ #0      NoSuchMethodError._throwNew (dart:core-patch/errors_patch.dart:176)
│ #1      _getMainClosure (dart:_builtin:97)
└────────────────────────────────────────────────────────────
Test suite failed: bigquery/v2_test.dart
Detailed results of test suite bigquery/v2_test.dart:
┌────────────────────────────────────────────────────
│
│ Unhandled exception:
│ No top-level getter 'main' declared.
│
│ NoSuchMethodError: method not found: 'main'
│ Receiver: top-level
│ Arguments: [...]
│ #0      NoSuchMethodError._throwNew (dart:core-patch/errors_patch.dart:176)
│ #1      _getMainClosure (dart:_builtin:97)
└────────────────────────────────────────────────────

From here it continues with No top-level getter 'main' declared. errors. All tests have main methods.
This is using Dart SDK 1.9.0-dev.8.0 and test_runner 0.2.15+2 on OS X.
Testing the googleapis_beta package have the same issue - tests after test number 10 fails.

Hey Soren,
i'm investigating. Question: are the failure consistant? i.e. always the same tests fail at each run?

Weird, just went through the steps above to try to replicate. I got this:
Summary: ALL 62 TEST SUITE(S) PASSED.
Trying again...

Actually on my second run 2 tests failed so it seems a bit random... I'm looking into it.

It's weird: it's random. I can't reproduce when running the tests individually with the generated files... Looks like a problem happening only when running lots of instances of dart in parallel (maybe...)

I think I fixed it. There was a place where we were not waiting for the temporary test files to be created before running the VM tests... You can update to 0.2.15+3

Thanks again for the report!

It all works now! Thanks!


Appstudio update mvp-test-component

Pipelines as Code configuration proposal

Pipelines as Code CI/mvp-test-component-on-pull-request has successfully validated your commit.

StatusDurationName

✅ Succeeded
6 seconds
init

✅ Succeeded
15 seconds
clone-repository

✅ Succeeded
1 minute
build-container

✅ Succeeded
15 seconds
clair-scan

✅ Succeeded
17 seconds
sanity-inspect-image

✅ Succeeded
18 seconds
deprecated-base-image-check

✅ Succeeded
56 seconds
clamav-scan

✅ Succeeded
8 seconds
sbom-json-check

✅ Succeeded
7 seconds
sanity-label-check

✅ Succeeded
8 seconds
sanity-optional-label-check

✅ Succeeded
5 seconds
show-summary

Pipelines as Code CI/mvp-test-component-on-pull-request has successfully validated your commit.

StatusDurationName

✅ Succeeded
6 seconds
init

✅ Succeeded
15 seconds
clone-repository

✅ Succeeded
1 minute
build-container

✅ Succeeded
17 seconds
sanity-inspect-image

✅ Succeeded
56 seconds
clamav-scan

✅ Succeeded
15 seconds
clair-scan

✅ Succeeded
18 seconds
deprecated-base-image-check

✅ Succeeded
8 seconds
sbom-json-check

✅ Succeeded
7 seconds
sanity-label-check

✅ Succeeded
8 seconds
sanity-optional-label-check

✅ Succeeded
5 seconds
show-summary


Naming collision detected

when I execute   ./packager/packager.sh, I see the error  below!
Error building DepdendencyGraph:
Error: Naming collision detected: /Users/guo/Desktop/react-native/node_modules/react-tools/src/shared/vendor/core/dom/isTextNode.js collides with /Users/guo/Desktop/react-native/AwesomeProject/node_modules/react-native/node_modules/react-tools/src/shared/vendor/core/dom/isTextNode.js
at HasteMap._updateHasteMap (/Users/guo/Desktop/react-native/packager/react-packager/src/DependencyResolver/DependencyGraph/HasteMap.js:123:13)
at /Users/guo/Desktop/react-native/packager/react-packager/src/DependencyResolver/DependencyGraph/HasteMap.js:95:28
at tryCallOne (/Users/guo/Desktop/react-native/node_modules/promise/lib/core.js:37:12)
at /Users/guo/Desktop/react-native/node_modules/promise/lib/core.js:103:15
at flush (/Users/guo/Desktop/react-native/node_modules/promise/node_modules/asap/raw.js:50:29)
at doNTCallback0 (node.js:417:9)
at process._tickCallback (node.js:346:13)

@foghina @martinbigio I got the same error.
Some observations:

react-native init projectName and then npm start. npm start if broken however react-native start works.
err:
sh: node_modules/react-native/packager/packager.sh: No such file or directory

name collision detected error after doing react-native start.

I took master branch for generating the new project(changed the cli for this).

I'm getting similar error but its erroring out on react-tools, any ideas?
Error building DepdendencyGraph:
 Error: Naming collision detected: /Users/corysmith/Apps/indieshuffle/node_modules/react-native/node_modules/react-tools/package.json collides with /Users/corysmith/Apps/indieshuffle/node_modules/react-native/packager/package.json
    at HasteMap._updateHasteMap (/Users/corysmith/Apps/indieshuffle/node_modules/react-native/packager/react-packager/src/DependencyResolver/DependencyGraph/HasteMap.js:127:13)
    at /Users/corysmith/Apps/indieshuffle/node_modules/react-native/packager/react-packager/src/DependencyResolver/DependencyGraph/HasteMap.js:104:32
    at tryCallOne (/Users/corysmith/Apps/indieshuffle/node_modules/react-native/node_modules/promise/lib/core.js:37:12)
    at /Users/corysmith/Apps/indieshuffle/node_modules/react-native/node_modules/promise/lib/core.js:103:15
    at flush (/Users/corysmith/Apps/indieshuffle/node_modules/react-native/node_modules/promise/node_modules/asap/raw.js:50:29)
    at doNTCallback0 (node.js:407:9)
    at process._tickCallback (node.js:336:13)

I'm getting similar error too after adding  react-native-router
Error building DepdendencyGraph:
Error: Naming collision detected: /Users/andrej/projects/test/testProject/node_modules/react-native-router/node_modules/react-native/Libraries/Promise.js collides with /Users/andrej/projects/test/testProject/node_modules/react-native/Libraries/Promise.js
at HasteMap._updateHasteMap (/Users/andrej/projects/test/testProject/node_modules/react-native/packager/react-packager/src/DependencyResolver/DependencyGraph/HasteMap.js:123:13)
at /Users/andrej/projects/test/testProject/node_modules/react-native/packager/react-packager/src/DependencyResolver/DependencyGraph/HasteMap.js:95:28
at tryCallOne (/Users/andrej/projects/test/testProject/node_modules/react-native/node_modules/promise/lib/core.js:37:12)
at /Users/andrej/projects/test/testProject/node_modules/react-native/node_modules/promise/lib/core.js:103:15
at flush (/Users/andrej/projects/test/testProject/node_modules/react-native/node_modules/promise/node_modules/asap/raw.js:50:29)
at doNTCallback0 (node.js:407:9)
at process._tickCallback (node.js:336:13)

If your are stranded here
So have a solution for this but not forever
you can delete you react-native folder from node_module/react-native-router/node_module/react-native
and you can use react-native from node_module/react-native

I fixed a similar issue to the duplicated module one on dee96fbedcd600e8819019005ae75ffbee2246f0. In regards the packager.sh, that one should have been fixed on 32aeeea3135fbb504b1481c0e7cbad1ddf1ae057.
Please reopen if any of this still repro :)
cc @mkonicek in case you've seen the naming colision mentioned on the task

This Promise issue looks legimit, sorry for closing. @mkonicek we don't have a strategy for dealing with this inside the packager, it just doesn't support having the same haste module defined twice. Fortunately, the new CLI brings the getBlacklistedRE function to the user on rn-cli.config.js so people should be able to blacklist the ones they want to get rid. Not sure if we can do anything in the mean time.

@martinbigio I have an idea for how to handle duplicate Haste modules by scoping them. I don't have the bandwidth to build it myself but will describe my vision for it in a new issue.

@ide, I think part of this will be solved by the issue we were just chatting about, which is that haste modules outside of react-native, react-tools, and currently, due to AsyncStorage, parse, shouldn't be used at all. Duplicate Haste module scoping would be cool, but I think fixing this other issue (which I plan to do ASAP) will solve most cases.

@skevy so just to be clear -- you want to fix deep dependencies so that we don't run javascript for Node on a platform.. that's not Node?

Wait, what? I don't think I said that. I'm confused by your response.

All I was saying was that the issue described here would be resolved by @providesModule not being used by the packager for packages outside of the FB ecosystem.

Oh i see. I thought you were referring to honoring the "browser" entry in the JSON.

No no...we should have the 'react-native' directive.

What things should I know before adding to the blacklist file? as it is Im getting
Error building DepdendencyGraph:
Error: Naming collision detected:
.../main_application/node_modules/react-native/node_modules/react-tools/src/shared/vendor/core/warning.js collides with .../main_application/node_modules/react-native/node_modules/react-tools/docs/js/react.js
at HasteMap._updateHasteMap (.../main_application/node_modules/react-native/packager/react-packager/src/DependencyResolver/DependencyGraph/HasteMap.js:123:13)
at .../main_application/node_modules/react-native/packager/react-packager/src/DependencyResolver/DependencyGraph/HasteMap.js:95:28
at tryCallOne (.../main_application/node_modules/react-native/node_modules/promise/lib/core.js:37:12)
at .../main_application/node_modules/react-native/node_modules/promise/lib/core.js:103:15
at flush (.../main_application/node_modules/react-native/node_modules/promise/node_modules/asap/raw.js:50:29)
at doNTCallback0 (node.js:417:9)
at process._tickCallback (node.js:346:13)
when I add warning.js and also one other file to the blacklist the packager starts but I run into issues once the application starts

@mangogogos i got the same situation,
Error building DepdendencyGraph:
Error: Naming collision detected: E:\Andriod\AwesomeProject\node_modules\react-native\node_modules\react-tools\src\shared\vendor\core\warning.js collides with E:\Andriod\AwesomeProject\node_modules\react-native\node_modules\react-tools\docs\js\react.js
    at HasteMap._updateHasteMap (E:/Andriod/AwesomeProject/node_modules/react-native/packager/react-packager/src/DependencyResolver/DependencyGraph/HasteMap.js:123:13)
    at E:/Andriod/AwesomeProject/node_modules/react-native/packager/react-packager/src/DependencyResolver/DependencyGraph/HasteMap.js:95:28
    at tryCallOne (E:\Andriod\AwesomeProject\node_modules\react-native\node_modules\promise\lib\core.js:37:12)
    at E:\Andriod\AwesomeProject\node_modules\react-native\node_modules\promise\lib\core.js:103:15
    at flush (E:\Andriod\AwesomeProject\node_modules\react-native\node_modules\promise\node_modules\asap\raw.js:50:29)
    at doNTCallback0 (node.js:417:9)
    at process._tickCallback (node.js:346:13)

PS E:\Andriod\AwesomeProject> react-native -v
0.1.7
PS E:\Andriod\AwesomeProject>

in my node_modules,there is only react-native, no more others. so who can help?

I have the same issue

Same problem here. I found a workaround: npm prune && npm i

It happened to me when I changed some dependencies in my project. Running node_modules/react-native/packager/packager.sh --resetCache fixed the dependencies issue.

Is there a way to exclude a folder from the packaging?

@jpapillon solution don't works for me

Removing global react-native-cli package worked for me:
npm uninstall -g react-native-cli

I'm trying to put a fairly complex react application into a webview in react native (to buy myself a little bit of time to do the "Real" port to RN.  But I'm running into this issue.  I've tried everything I can think of, but it seems that my compiled app (in dist/assets/main.js is legitimately colliding with the version of invariant that react-tools is trying to pull in.
Am I just totally stuck here until someone figures out how to namespace/scope the Haste modules?
[9:18:47 AM] <START> Building Haste Map
Error building DepdendencyGraph:
 Error: Naming collision detected: /Users/litch/code/Wallet/dist/assets/main.js collides with /Users/litch/code/Wallet/node_modules/react-tools/src/shared/vendor/core/invariant.js
    at HasteMap._updateHasteMap (/Users/litch/code/Wallet/node_modules/react-native/packager/react-packager/src/DependencyResolver/DependencyGraph/HasteMap.js:123:13)
    at /Users/litch/code/Wallet/node_modules/react-native/packager/react-packager/src/DependencyResolver/DependencyGraph/HasteMap.js:95:28
    at tryCallOne (/Users/litch/code/Wallet/node_modules/promise/lib/core.js:37:12)
    at /Users/litch/code/Wallet/node_modules/promise/lib/core.js:103:15
    at flush (/Users/litch/code/Wallet/node_modules/asap/raw.js:50:29)
    at doNTCallback0 (node.js:428:9)
    at process._tickCallback (node.js:357:13)
Process terminated. Press <enter> to close the window

The packager has a way to configure a blacklist (grep around for the details), which would let you exclude main.js from the JS bundle since RN doesn't need it anyway.

Oh - ok.  That is interesting, thank you.

None of the "solutions" work for me. Anyone have a way around this?

There were too many naming collisions after my upgrading...

The same issue. Previous suggestions don't work for me

Error building DependencyGraph:
Error: Naming collision detected: /Users/vkhazin/Projects/Samples/ReactNative/package.json collides with /Users/vkhazin/Projects/Samples/ReactNative/node_modules/react-native/Libraries/ReactNative/ReactNative.js
at HasteMap._updateHasteMap (HasteMap.js:123:13)
at HasteMap.js:103:32
at tryCallOne (/Users/vkhazin/Projects/Samples/ReactNative/node_modules/promise/lib/core.js:37:12)
at /Users/vkhazin/Projects/Samples/ReactNative/node_modules/promise/lib/core.js:123:15
at flush (/Users/vkhazin/Projects/Samples/ReactNative/node_modules/asap/raw.js:50:29)
at nextTickCallbackWith0Args (node.js:456:9)
at process._tickCallback (node.js:385:13)
~

as @ide mentioned you can blacklist files/regexps on the packager. Take a look at packager/blacklist.js :)

thank you for your comment martinbigio!
Could you please elaborate a bit more as my experience with React.js is 6 hours old and with React Mobile 2 hours old?

@vkhazin absolutely, I'm glad you're giving React Native a try. My comment was for @yarax tough. There's something wrong on your setup. The steps posted on https://facebook.github.io/react-native/docs/getting-started.html should work. Could you give it a try? :)

Also, in general it's better to ask these type of questions on Stack Overflow: http://stackoverflow.com/questions/tagged/react-native

Hi Martin,
Thank you for your comments!
I did tried twice twice using node 4x and node 5x to make sure the issue is
not related to version of node.
In both cases created the project using the instructions mentioned:
https://facebook.github.io/react-native/docs/getting-started.html and in
both cases ended up with the same error.
On Tue, Jan 12, 2016 at 11:20 AM, Martín Bigio<EMAIL_ADDRESS>wrote:

Also, in general it's better to ask these type of questions on Stack
Overflow: http://stackoverflow.com/questions/tagged/react-native
—
Reply to this email directly or view it on GitHub
https://github.com/facebook/react-native/issues/3558#issuecomment-170962404
.

--
Sincerely yours,
Vlad Khazin
Email<EMAIL_ADDRESS>Skype: vladimir.khazin
Cell:<PHONE_NUMBER>
Fax:<PHONE_NUMBER>
http://www.linkedin.com/in/vkhazin

I think the problem is related to the fact the project name is ReactNative. cc @mkonicek

Here is something to laugh about:

named my project ReactNative by calling: 'react-native init ReactNative'
project created:  /Users/vkhazin/Projects/Samples/ReactNative/
try to build:
Error: Naming collision detected
Error: Naming collision detected: /Users/vkhazin/Projects/Samples/ReactNative/package.json collides with /Users/vkhazin/Projects/Samples/ReactNative/node_modules/react-native/Libraries/ReactNative/ReactNative.js

Repeat all of the above steps with 'react-native init AwesomeProject' and it runs!

I had a similar issue when downgrading from 21 to 18, and uninstalled the react peer dependency and collision error never occurred again.
npm uninstall react --save

Failed to build DependencyGraph: Naming collision detected: /Users/erwai/Documents/app/MyApp/node_modules/react/node_modules/fbjs/lib/warning.js collides with /Users/erwai/Documents/app/MyApp/node_modules/fbjs/lib/warning.js
Error: Naming collision detected: /Users/erwai/Documents/app/MyApp/node_modules/react/node_modules/fbjs/lib/warning.js collides with /Users/erwai/Documents/app/MyApp/node_modules/fbjs/lib/warning.js
at HasteMap._updateHasteMap (HasteMap.js:132:13)
at HasteMap.js:103:28
at tryCallOne (/Users/erwai/Documents/app/MyApp/node_modules/promise/lib/core.js:37:12)
at /Users/erwai/Documents/app/MyApp/node_modules/promise/lib/core.js:123:15
at flush (/Users/erwai/Documents/app/MyApp/node_modules/asap/raw.js:50:29)
at _combinedTickCallback (node.js:376:9)
at process._tickCallback (node.js:407:11)
npm ERR! Darwin 15.2.0
npm ERR! argv "/Users/erwai/.nvm/versions/node/v5.9.1/bin/node" "/Users/erwai/.nvm/versions/node/v5.9.1/bin/npm" "start"
npm ERR! node v5.9.1
npm ERR! npm  v3.8.3
I tried all methods suggested above and none of them works...
I updated my Node & npm versions to the latest versions but it still not working.

Do yourselves a favor:

watchmen watch-del-all
rm -fr $TMPDIR/react-*
npm cache clean
npm start -- --reset-cache

If you had/have a problem resolving modules:

Follow steps 1-3 above and In your package.json file under scripts add:
"start": "node node_modules/react-native/local-cli/cli.js start --root ../path-to-root/  -- --reset-cache"

Create a rn-cli.config.js file where your package.json file is and put in:

'use strict';

const path = require('path');

module.exports = {
    getProjectRoots() {
        return this._getRoots();
    },

    getAssetRoots() {
        return this._getRoots();
    },

    _getRoots() {
        // match on either path separator
        if (__dirname.match(/node_modules[\/\\]react-native[\/\\]packager$/)) {
            // packager is running from node_modules of another project
            return [path.resolve(__dirname, '../../..')];
        } else if (__dirname.match(/Pods\/React\/packager$/)) {
            // packager is running from node_modules of another project
            return [path.resolve(__dirname, '../../..')];
        } else {
            return [
                path.resolve(__dirname, '../'),
                path.resolve(__dirname, '.'),
                path.resolve(__dirname, '../path-to-root/')
            ];
        }
    },
};

Starting with the --reset-cache flag is fairly key. I had it in at one point when attempting to resolve modules then took it out thinking I didn't need it anymore. Once the project blew up for whatever reason the Name collision error popped up. The only thing I actually changed that made it work was re-inputting the --reset-cache flag  in the start script after deleting modules, watchmen, and even npm cache cleaning.
Credit goes to @johnknaack for finding the rn-cli-config.js option found in react-native/local-cli/detail.config.js

It works for me. @hyphenbash Remove react.

Thanks @hyphenbash Also resolved this problem on Ubuntu Linux 15.10 by removing react depency. (I had downgraded from react native 0.24.1 to 0.20.0)

All, use react-native version 0.26.0. That'll solve your problem.
package.json file

@yarax Apparently having NPM version > 3.0 was the fix for me when i had the exact same issue. Hope this helps if you are still looking.

intenta con este metodo:
http://stackoverflow.com/a/33278101

I had this problem because I had a node_modules and a node_modules_bk folder.
It worked after deleting the second one.

I was getting errors like
Failed to build DependencyGraph: @providesModule naming collision:
Failed to build DependencyGraph: @providesModule naming collision:
Duplicate module name: accepts
Duplicate module name: accepts

I solved it by deleting the folder .npm_cache and rerunning packager with --reset-cache
rm -rf .npm_cache

For me the naming collision error was actually caused by using --reset-cache. Running packager without resetting the cache fixes it.

If anyone's using react-native-router-flux and ending up here for help check:
https://github.com/aksonov/react-native-router-flux/issues/1803

it might seem crazy what i'm about to say. I have old source code which required small fix & new app is on the way already, so no need to fix this one properly.
I've removed react_native from the dependency which was throwing this exception, renamed in it's source paths from react_native local to upper folder, like import ... from '../react-native' which is in main node_modules and also created symlink to node_modules/react_native for the binary file used by same dependency and it all built well :-) i would not recommend to have anything like this in production, but for one-time fix it worked out.


Unable to Build for Windows on OS X when sourcedir is "."

I tried all possible arguments in CLI yet I still get  "... and 1 more" while building for Windows on OS X. I have also install rcedit, wine. Can anyone clearly document building for Windows on OS X.

Could you please answer the questions in the issue reporting documentation so that we may better help you with your problem?

Electron Packager - 5.1.1
CLI
electron-packager . FooBar --platform=win32 --arch=ia32 --version=0.36.0 --icon=assets/win/icon.ico

What's the exact console output when you run the electron-packager command?

I have never seen that error before. Have you tried building a package for OS X instead? (I want to know if it's specific to Windows as the target platform.)

For OS X it is working perfectly, is there any requirements for building windows package I mean dependencies other than Wine and rcedit for OS X ? There is any other better documentation/tutorial for the same.

same error here, on OSX if using electron-packager ./ ElectronTest --out=dist --platform=darwin --arch=x64 --version=0.36.7 --asar --overwrite --ignore=node_modules/electron-prebuilt --ignore=node_modules/electron-packager it works fine, but if I use electron-packager . ElectronTest --out=dist --all --asar --overwrite --ignore=node_modules/electron-prebuilt --ignore=node_modules/electron-packager,then it shows
electron-packager . ElectronTest --out=dist --all --asar --overwrite --ignore=node_modules/electron-prebuilt --ignore=node_modules/electron-packager

... and 1 more

and stucks.
I didn't install anything like Wine or rcredit though.

Have almost same issue:
➜  universal-sms electron-packager . --all
Unable to determine application name or Electron version. Please specify an application name and Electron version.
Cannot find module 'package.json' from '/Users/drakmail/Develop/drakmail'(Properties not found yet: dependencies.electron-prebuilt,devDependencies.electron-prebuilt)
➜  universal-sms pwd
/Users/drakmail/Develop/drakmail/universal-sms

➜  universal-sms ls
bin          db.gdb       index.html   js           main.js      node_modules package.json sender.js    storage.js

@drakmail this is a different error, you need to either add the electron-prebuilt module as a dependency in package.json or specify the Electron version as a parameter. (See readme for details)

@malept Thanks, it's really helps!

When I followed the instructions by using:
electron-packager . --all
It didn't work! I figured I need to do it manually..
This works in terminal:
electron-packager ./ App --out=dist --platform=all --arch=all --asar --overwrite --ignore=node_modules/electron-prebuilt --ignore=node_modules/electron-packager

@gvzq I have a comment and a question:

Per the docs, --ignore=node_modules/electron-prebuilt --ignore=node_modules/electron-packager is already ignored by default.
Are you saying that you need to type ./ and not .  to correctly specify the app's source directory?

Sorry if I wasn't clear. Running electron-packager ./ --all does not execute properly due to a EISDIR error.
electron-app usr$ electron-packager ./ --all
Packaging app for platform linux ia32 using electron v1.3.3
EISDIR: illegal operation on a directory, rename '/var/folders/jk/3cmm8ls96bnc00gfys3vqky80000gn/T/electron-packager/linux-ia32/-linux-ia32/electron' -> '/var/folders/jk/3cmm8ls96bnc00gfys3vqky80000gn/T/electron-packager/linux-ia32/-linux-ia32'

@gvzq this is a different error than this GitHub issue, could you please file a separate issue? (Making sure you read our docs on filing bugs and follow the directions in the GitHub issue template) It would be much appreciated.

ok @malept

The ... and 1 more message is a bug in nugget, the library used to download Electron, https://github.com/maxogden/nugget/issues/16
If you make your terminal window bigger then you should see more output about the download happening.

The ... and 1 more message is a bug in nugget, the library used to download Electron, https://github.com/maxogden/nugget/issues/16
If you make your terminal window bigger then you should see more output about the download happening.

I'm inclined to close this since this is a dependency bug, then.

I had the same problem with the same error "... and 1 more" trying to package for win32 from Mac. In my case the solution was to add Wine to PATH.


Retrieve Oauth2 additionalInformations while not using JWT

Hi, I'm implementing Oauth2 in our applications. I'm using jdbc store for storing tokens.
I would like to add some additional informations to authentication on UAA server so I've implemented TokenEnhancer:
public class CustomTokenEnhancer implements TokenEnhancer {

private static final Logger logger = LoggerFactory.getLogger(CustomTokenEnhancer.class);

@Override
public OAuth2AccessToken enhance(OAuth2AccessToken accessToken, OAuth2Authentication authentication) {
	if (authentication.getUserAuthentication().getPrincipal().getClass().isAssignableFrom(CustomUserDetails.class)) {
		logger.debug("Enhancing token with custom metadata");
		final Map<String, Object> additionalInfo = new HashMap<>();
		CustomUserDetails customUserDetails = (CustomUserDetails) authentication.getPrincipal();
		additionalInfo.put(C.METADATA_CURRENT_USER_ID, customUserDetails.getId());
		additionalInfo.put(C.METADATA_CURRENT_USER_DEFAULT_LANGUAGE, customUserDetails.getDefaultLanguage());
		((DefaultOAuth2AccessToken) accessToken).setAdditionalInformation(additionalInfo);
	}
	return accessToken;
}

}
On the client side I'd like to access those informations, but while I'm not using JWT I don't know how to access OAuth2AccessToken.
I was trying with this method
Principal principal = SecurityContextHolder.getContext().getAuthentication();
if (principal.getClass().isAssignableFrom(OAuth2Authentication.class)) {
	OAuth2Authentication oAuth2Authentication = (OAuth2Authentication) SecurityContextHolder.getContext().getAuthentication();
	OAuth2AuthenticationDetails auth2AuthenticationDetails = (OAuth2AuthenticationDetails) oAuth2Authentication.getDetails();
	OAuth2AccessToken accessToken = remoteTokenServices.readAccessToken(auth2AuthenticationDetails.getTokenValue());
	Map<String, Object> additionalInformations = accessToken.getAdditionalInformation();
}

but at the end I realised that RemoteTokenServices.readAccessToken() reading is not supported (?due to security reasons?).
So is it possible somehow access additionalInformations through accessToken and remoteServices or do I need to read it through database?
Or is there any other way how can I access CustomUserDetails from Authentication in client application?
Thanks

@dsyer can you help me with this? Thanks

Where did the OAuth2Authentication come from (probably from a ResourceServerTokenServices, possibly already a  RemoteTokenServices)? Isn't that where you should be looking to modify something, if anything? I believe there's a TokenConverter strategy you can plug in, and you also might have to look at the auth server (if you control it) to see if it is actually sending you the additional info.

@dsyer that should work. However currently my RemoteTokenServices is always replaced with one from org/springframework/boot/autoconfigure/security/oauth2/resource/ResourceServerTokenServicesConfiguration. Is there any simple way how to "reconfigure" the RemoteTokenServices?
this is my config class on resource application
@Configuration
public class Oauth2Configuration {

	@Autowired
	private ResourceServerProperties resource;

	@Bean
	public ResourceServerTokenServices remoteTokenServices() {
		RemoteTokenServices services = new RemoteTokenServices();
		services.setCheckTokenEndpointUrl(this.resource.getTokenInfoUri());
		services.setClientId(this.resource.getClientId());
		services.setClientSecret(this.resource.getClientSecret());
		services.setAccessTokenConverter(docuTokenConverter());
		return services;
	}

	@Bean
	DocuAccessTokenConverter docuTokenConverter() {
		return new DocuAccessTokenConverter();
	}
}

Overriding bean definition for bean 'remoteTokenServices' with a different definition: replacing [Root bean: class [null]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=oauth2Configuration; factoryMethodName=remoteTokenServices; initMethodName=null; destroyMethodName=(inferred); defined in class path resource [com/github/bilak/test/security/configuration/Oauth2Configuration.class]] with [Root bean: class [null]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=org.springframework.boot.autoconfigure.security.oauth2.resource.ResourceServerTokenServicesConfiguration$RemoteTokenServicesConfiguration$TokenInfoServicesConfiguration; factoryMethodName=remoteTokenServices; initMethodName=null; destroyMethodName=(inferred); defined in class path resource [org/springframework/boot/autoconfigure/security/oauth2/resource/ResourceServerTokenServicesConfiguration$RemoteTokenServicesConfiguration$TokenInfoServicesConfiguration.class]]

I figured it out by adding @Import(value = { ResourceServerTokenServicesConfiguration.class }) to the Oauth2Configuration (don't know if this is correct).

That's one way, I guess. If it was me I'd look for a way just switch off the autconfig by not configuring the relevant bits of security.oauth2.*.


MOE pharmacophores

Description
Export and import pharmacophores to MOE ph4 format.

Codecov Report

Merging #26 (f310030) into main (809adb7) will not change coverage.
The diff coverage is 0.00%.


Script does no longer work

python .\print-local-keys.py
Please put your Tuya/Ledvance username<EMAIL_ADDRESS>Please put your Tuya/Ledvance password:
device name:    Smart Plug 1
device id:      bf94c09abf90xxxxxxx
local key:      hGwPLA)T_Z%F>{PL
device name:    Smart Plug 2
device id:      bf9772ae0daxxxxxxx
local key:      '!}{g82vEr^E`4&f
Press Enter to continue...

I can control my smart plugs via the app

Yes but the keys are encrypted in some way, and homeassistant won't accept the keys as the script print them.

They look pretty similar to what I would get for my lamps (16 chars, alphanumeric with special characters) and for me it still works. Any error message or hints in the logs of home assistant?

works just fine for me

Okay, let me try again tomorrow.

Hey, you know what? It was my home assistant that was bad! Not your program, thanks so much👍


custom URL scheme upper case letters not working

Summary

System Console > Customization > Posts
In Custom URL Schemes field enter a value like "file, DMS, dms"

I did so and the upper case one is not working. We have a software which is exporting links as
DMS:<GUID>

inserting this link into a message is not working. Manually replace DMS to
dms:<GUID>

is working as a clickable link.
I don't know if this is the only place, but I didn't find any other... If this is only rendered on client side, the windows desktop is also affected (have not tested other apps).
Webapp -> uppercase not working
windows desktop app -> uppercase not working
Android App -> uppercase partially working (DMS does, HTTP does not)

Ticket Link
none
Checklist

[x] just a little regex

Setup Test Server label detected. Spinmint test server created if build succeeds (checks pass and no conflicts with base branch).

Spinmint test server created at: https://i-0351d232adb71ef9f.test.spinmint.com
Test Admin Account: Username: sysadmin | Password: sysadmin
Test User Account: Username: user-1 | Password: user-1
Instance ID: i-0351d232adb71ef9f

Spinmint test server destroyed

Setup Test Server label detected. Spinmint test server created if build succeeds (checks pass and no conflicts with base branch).

Spinmint test server created at: https://i-01c2fec700f429a60.test.spinmint.com
Test Admin Account: Username: sysadmin | Password: sysadmin
Test User Account: Username: user-1 | Password: user-1
Instance ID: i-01c2fec700f429a60

Definitely not a problem! Let us know if you have any questions along the way :)

Setup Test Server label detected. Spinmint test server created if build succeeds (checks pass and no conflicts with base branch).

Hey @Hobby-Student,
The Spinmint test server is used to test a PR. For developing a feature please use a local dev environment. You can find an instruction for this on https://developers.mattermost.com/contribute/webapp/developer-setup/. If you have problems feel free as them on the Mattermost Community Server.

@hanzei
did read the dev instructions. will try to get a dev machine asap.

Hey @Hobby-Student,
Did you manage to set up a dev environment? Is there anything I can help you with?

Did you manage to set up a dev environment?

I'm currently waiting for some RAM. Should arrive the day after tomorrow.
Setting up the dev machine itself seems to be easy.

In Custom URL Schemes field enter a value like "file, DMS, dms"

Instead of having the admin specify an uppercase custom URL scheme, would it make sense to make all of the custom URL schemes case-insensitive?

Instead of having the admin specify an uppercase custom URL scheme, would it make sense to make all of the custom URL schemes case-insensitive?

exactly what I'm trying to do. If you could point me to the right direction, where custom schemes are parsed...?
Android app seems to work case insensitive.

I think this line is the one that needs to be changed since it currently looks for the exact string in the array. Using a .findIndex would let you do a case insensitive comparison instead.

This issue has been automatically labelled "stale" because it hasn't had recent activity.
A core team member will check in on the status of the PR to help with questions.
Thank you for your contribution!
/cc @jasonblais @hanzei

Hey @Hobby-Student - let us know, if we can help or have any questions on the latest note from @hmhealey :)

@jasonblais you could buy me some free time ;)
dev machine is currently setting up. hope to get my hands on the code this weekend

I think I got it. it now recognizes lower and upper case letter before ":" and then checks the message with "toLowerCase()" against the default and custom schemes.

check url also with upper case protocol (href is converted to lower case)
check scheme in lower case against list of schemes (default and custom) <- without this, default schemes are only recognized as is, e. g. HTTP won't be recognized

the only thing it needs, is to also edit marked.js. Without this, the message will only be checked against lower case protocols.

should I make a pull request for mattermost/marked?

Great work @Hobby-Student!! :tada:
@hmhealey will have insight on potential changes for mattermost/marked.

Yes, please submit a PR against https://github.com/mattermost/marked for that, and when you do, can you link the PR back here so we definitely see it? It's easy for us to miss PRs against forked libraries somestimes.
After that's merged, this PR can be updated to point to the new version of marked

please submit a PR

done

FYI, this will conflict with https://github.com/mattermost/mattermost-webapp/pull/2516 depending on which gets merged first. The commit hash in this PR is the newer one

Setup Test Server label detected. Spinmint test server created if build succeeds (checks pass and no conflicts with base branch).

Spinmint test server created at: https://i-0c6b9278d359a6d7e.test.spinmint.com
Test Admin Account: Username: sysadmin | Password: sysadmin
Test User Account: Username: user-1 | Password: user-1
Internal IP: <IP_ADDRESS>
Instance ID: i-0c6b9278d359a6d7e

tested. results:

URL with title is working as expected. URL with quotes is working in lowercase only.
it seems, that there are some more changes needed to get it work in all circumstances...

Thank you @Hobby-Student!
@hmhealey apart from what's shown in the screenshot above, any additional tests that we should do? If so, I can ask our QA, but in general it appears to work as expected.

I don't think there's anything specific to test. These changes are mostly around making it so that the URL scheme can contain uppercase letters

Sounds good.
Thanks @Hobby-Student for this! I requested a second dev review and then this should be ready to merge :)

Spinmint test server destroyed

Please rebase and fix conflicts so we can merge it.
done.

Glad that I was able help 😃

Thank you for the contribution @Hobby-Student 👍


v0.11.0 - Uncaught SyntaxError Buffer

Uncaught SyntaxError: The requested module
'/@fs/home/dev/repo/node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js?v=0b1a4bd8' does not provide an export named 'Buffer' (at index.js?v=0b1a4bd8:1:10)
9,10,etc worked before.
I create a module which uses buffer, build it and use it in a vite app. Everything is built with vite, including other packages in turbo repo.
In v0.11.0 the resulting library has this:
import { Buffer as BufferPolyfill } from '/home/dev/repo/node_modules/.pnpm/vite-plugin-node-polyfills<EMAIL_ADDRESS>window.Buffer = BufferPolyfill
import { global as globalPolyfill } from '/home/dev/repo/node_modules/.pnpm/vite-plugin-node-polyfills<EMAIL_ADDRESS>window.global = globalPolyfill
import { process as processPolyfill } from '/home/dev/repo/node_modules/.pnpm/vite-plugin-node-polyfills<EMAIL_ADDRESS>window.process = processPolyfill

In v0.10.0 the resulting library has this:
/*!
 * The buffer module from node.js, for the browser.
 *
 * @author   Feross Aboukhadijeh <https://feross.org>
 * @license  MIT
 */
(function(e) {
  const a = Ca, t = ar, n = typeof Symbol == "function" && typeof Symbol.for == "function" ? Symbol.for("nodejs.util.inspect.custom") : null;
  e.Buffer = i, e.SlowBuffer = P, e.INSPECT_MAX_BYTES = 50;
  const l =<PHONE_NUMBER>;
  e.kMaxLength = l, i.TYPED_ARRAY_SUPPORT = d(), !i.TYPED_ARRAY_SUPPORT && typeof console < "u" && typeof console.error == "function" && console.error(
    "This browser lacks typed array (Uint8Array) support which is required by `buffer` v5.x. Use `buffer` v4.x if you require old browser support."
  );
  function d() {
    try {
      const s = new Uint8Array(1), r = { foo: function() {
        return 42;
      } };
      return Object.setPrototypeOf(r, Uint8Array.prototype), Object.setPrototypeOf(s, r), s.foo() === 42;
    } catch {
      return !1;
    }
  }

My vite config;
import { defineConfig } from 'vite';
import { nodePolyfills } from 'vite-plugin-node-polyfills';

import path from 'path';

// https://vitejs.dev/config/
export default defineConfig({
  plugins: [nodePolyfills()],
  resolve: {
    alias: {
      '@/': new URL('./src/', import.meta.url).pathname,
    },
  },
  build: {
    target: 'esnext',
    lib: {
      entry: path.resolve(__dirname, 'src/index.ts'),
      formats: ['cjs', 'es'],
      fileName: (format) => `index.${format}.js`,
    },
  },
});

Specifically, in a production build there is no issue that I've seen as my site deploys. This happens when running in dev mode.

Duplicate of #29 (fixed in v0.11.1).


HPX 2

I'm opening this as a placeholder for discussion on long term plans for HPX (which we may call 2). See this as a wishlist of things that should be finally deprecated, cleaned up, or implemented.
I'll start (please extend this list):

Follow semver
Complete modularization
CMake modernization
Namespace and header location cleanup (clear policy on what's part of the API, what's experimental, what's private) (#4364)
Executors; there should be a clear story about what executor to use in which situations
Libfabric improvements in master: this is one of the strongest selling points of distributed HPX to existing applications using MPI (features have to be amazing for them to care, performance is a much better selling point)
ROCm support (minimal future support, not full parallel algorithms)

Many of these things are already in progress but moving to 2 would allow us to "officially" clean up and break compatibility without feeling too bad about ourselves. If needed we can keep 1.X in lifesupport mode for users who can't take all the breaking changes yet.
The namespace cleanup is partly motivated by having e.g. an experimental namespace (this has been discussed before, but should be formalized) so that we can introduce new features and if needed break or remove them without deprecation warnings as long as they are experimental.
The alternative to this is of course keep incrementally adding and removing features as we've been doing now, but I think going for a new major release would be good for both visibility and for allowing us to clean up things properly.

We had a brief discussion (@aurianer, @biddisco, @hkaiser, and myself) yesterday on how to proceed towards 2.0.0. I've tried to write down here the rough consensus from that discussion. 1.6.0 and further 1.X releases do not have to come 6 months apart. We can do them as quickly as a significant change is there. There are a few things from the above list (libfabric cleanup, ROCm) that I didn't put in a specific release. We can squeeze them in to 1.6.0 or any later 1.X release as things are ready. Please add the major things you think are missing.
1.5.0

(already there: CMake targets, local runtime)
All local modules in place (#4606, #4592)
API headers and namespaces for local functionality in place (without this users will have to first change their headers to the replacements we have now, and then again to the final headers we define in #4626)

1.6.0

Split up main library into three parts (according to #4593)

1.7.0

If needed

2.0.0

All modules in place, including distributed
API headers and namespaces for distributed functionality in place


Speed up DuplicateFlowTest

This test takes about 15 seconds to run on my machine. Most of the time appears
to be taken in loading the Smartdown flows, notably the pay-leave-for-parents
and the pay-leave-for-parents-adoption flows. These are being (re-)loaded,
because the singleton instance is being reset. Unfortunately I couldn't tell
from the commit note why this reset is necessary, so I didn't feel comfortable
just removing it.
However, I noticed that the Smartdown::Registry#available_methods method being
tested returns a list of flow names which are only based on filenames - it
does not rely on instantiating the flows themselves. So it seems as if it
should be safe to temporarily disable flow preloading for the duration of just
this test.
With this change in place, the test takes much less than 1 second to run.

Unfortunately I couldn't tell from the commit note why this reset is necessary, so I didn't feel comfortable just removing it.

Perhaps something similar to https://github.com/alphagov/smart-answers/pull/1579 ?
I do not feel good about how we solve this problem in smartanswers (see here) but in this particular case I don't see how it could cause any problems :+1:

@tadast: I have some ideas about how to improve the whole registry singleton/options thing, but I think I'd still prefer to make this change now. Thanks for the feedback.


Error: validation failed: unable to recognize "": no matches for kind "StatefulSet" in version "apps/v1beta2"

i am running the below command i am getting error
helm install --name nfs-server-provisioner stable/nfs-server-provisioner --set persistence.enabled=true,persistence.size=8Gi
i am getting this error Error: validation failed: unable to recognize "": no matches for kind "StatefulSet" in version "apps/v1beta2"
So how to reslove this issue and how to change apps/v1beta2 to apps/v1beta1

What's your kubernetes version?It seems like StatefulSet has been removed from your version.I think you need to change apps/v1beta2 to apps/v1.

i am using 1.16.1 version
how to change apps/v1beta2 to apps/v1
where i need to change can you give me command or procedure

What's your kubernetes version?It seems like StatefulSet has been removed from apps/v1beta2 in your version.I think you need change apps/v1beta2 to apps/v1.

i am using 1.16.1 version
how to change apps/v1beta2 to apps/v1
where i need to change can you give me command or procedure

It seem like this file：
https://github.com/helm/charts/blob/master/stable/nfs-server-provisioner/templates/statefulset.yaml
But i don't know how to change this local if not create a pr to helm.

Maybe you can try follow this:
cd ~/.helm/cache/archive
tar -xvf nfs-server-provisioner-0.3.1.tgz
vi nfs-server-provisioner/templates/statefulset.yaml
#change apiversion apps/v1
helm package nfs-server-provisioner
helm serve &
helm repo add local http://<IP_ADDRESS>:8879
helm repo update
helm install local/nfs-server-provisioner --name my-nfs\

if this is not ok, do this:
cd ~/.helm/repository/local
tar -xvf nfs-server-provisioner-0.3.1.tgz
vi nfs-server-provisioner/templates/statefulset.yaml
helm package nfs-server-provisioner
Successfully packaged chart and saved it to: /root/.helm/repository/local/nfs-server-provisioner-0.3.1.tgz
# helm install local/nfs-server-provisioner --name my-nfs
NAME:   my-nfs
LAST DEPLOYED: Tue Nov  5 02:32:23 2019
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/ClusterRole
NAME                           AGE
my-nfs-nfs-server-provisioner  0s

==> v1/ClusterRoleBinding
NAME                           AGE
my-nfs-nfs-server-provisioner  0s

==> v1/Pod(related)
NAME                             READY  STATUS             RESTARTS  AGE
my-nfs-nfs-server-provisioner-0  0/1    ContainerCreating  0         0s

==> v1/Service
NAME                           TYPE       CLUSTER-IP      EXTERNAL-IP  PORT(S)                                 AGE
my-nfs-nfs-server-provisioner  ClusterIP  <IP_ADDRESS>  <none>       2049/TCP,20048/TCP,51413/TCP,51413/UDP  0s

==> v1/ServiceAccount
NAME                           SECRETS  AGE
my-nfs-nfs-server-provisioner  1        0s

==> v1/StatefulSet
NAME                           READY  AGE
my-nfs-nfs-server-provisioner  0/1    0s

==> v1/StorageClass
NAME  PROVISIONER                                  AGE
nfs   cluster.local/my-nfs-nfs-server-provisioner  0s

NOTES:
The NFS Provisioner service has now been installed.

A storage class named 'nfs' has now been created
and is available to provision dynamic volumes.

You can use this storageclass by creating a `PersistentVolumeClaim` with the
correct storageClassName attribute. For example:

    ---
    kind: PersistentVolumeClaim
    apiVersion: v1
    metadata:
      name: test-dynamic-volume-claim
    spec:
      storageClassName: "nfs"
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 100Mi

I tried and this was ok.

StatefulSet is only in apps/v1 in 1.16.1.
It seem like this file：
https://github.com/helm/charts/blob/master/stable/nfs-server-provisioner/templates/statefulset.yaml
But i don't know how to change this local if not create a pr to helm.

Hi i have installed kubernetes 1.14.6 version
I am installing mongodb using helm command in aws instance i am using below command but mongodb pod and pvc pod status showing pending
helm install --set mongodbRootPassword=hello,mongodbUsername=mongo,mongodbDatabase=mydata --set volumePermissions.enabled=true --name mongo stable/mongodb
in PVC pod logs showing no persistent volumes available for this claim and no storage class is set
error
in mongodb pod logs showing pod has unbound immediate PersistentVolumeClaims error
So can you please suggest me how to resolve this issue.

/triage support

@veera2019 Try using this command to change apps/v1beta2 to apps/v1
 kubectl convert -f <file> --output-version apps/v1.

@RA489 convert is deprecated:

kubectl convert is DEPRECATED and will be removed in a future version.
In order to convert, kubectl apply the object to the cluster, then kubectl get at the desired version.
error: error parsing stable/ipfs/templates/statefulset.yaml: error converting YAML to JSON: yaml: line 6: did not find expected key


Uncaught TypeError: Cannot read property 'maximize' of null

[Enter steps to reproduce below:]

...
...

Atom Version: 1.11.2
Electron Version: 0.37.8
System: Mac OS X 10.11.6
Thrown From: Termination package, v0.1.4
Stack Trace
Uncaught TypeError: Cannot read property 'maximize' of null
At /Users/mannyhagman/.atom/packages/Termination/lib/status-bar.coffee:49

TypeError: Cannot read property 'maximize' of null
    at atom-workspace.subscriptions.add.atom.commands.add.termination:fullscreen (/Users/mannyhagman/.atom/packages/Termination/lib/status-bar.coffee:49:51)
    at CommandRegistry.module.exports.CommandRegistry.handleCommandEvent (/Applications/Atom.app/Contents/Resources/app.asar/src/command-registry.js:260:29)
    at CommandRegistry.handleCommandEvent (/Applications/Atom.app/Contents/Resources/app.asar/src/command-registry.js:3:61)
    at CommandRegistry.module.exports.CommandRegistry.dispatch (/Applications/Atom.app/Contents/Resources/app.asar/src/command-registry.js:160:19)
    at AtomEnvironment.module.exports.AtomEnvironment.dispatchApplicationMenuCommand (/Applications/Atom.app/Contents/Resources/app.asar/src/atom-environment.js:1127:28)
    at EventEmitter.outerCallback (/Applications/Atom.app/Contents/Resources/app.asar/src/application-delegate.js:341:25)
    at emitTwo (events.js:100:13)
    at EventEmitter.emit (events.js:185:7)

Commands
     -2:02.4.0 platformio-ide-terminal:new (ul.list-inline.tab-bar.inset-panel)
     -1:46 termination:new (ol.tree-view.full-menu.list-tree.has-collapsable-children.focusable-panel)
     -0:28.7.0 termination:close-all (atom-text-editor.editor.is-focused)
     -0:24.2.0 termination:new (atom-text-editor.editor.is-focused)
     -0:11.3.0 core:save (atom-text-editor.editor.is-focused)
     -0:06.7.0 platformio-ide-terminal:toggle (atom-text-editor.editor.is-focused)
     -0:00.0 termination:fullscreen (atom-text-editor.editor.is-focused)

Config
{
  "core": {
    "disabledPackages": [
      "atom-ternjs",
      "highlight-selected",
      "activate-power-mode",
      "welcome"
    ],
    "telemetryConsent": "no",
    "themes": [
      "atom-dark-ui",
      "atom-dark-syntax"
    ],
    "useCustomTitleBar": true
  }
}

Installed Packages
# User
activate-power-mode, v1.1.0 (inactive)
atom-beautify, v0.29.13 (active)
atom-material-syntax, v0.4.6 (inactive)
atom-material-syntax-dark, v0.2.5 (inactive)
atom-material-ui, v1.3.6 (inactive)
atom-ternjs, v0.15.0 (inactive)
highlight-selected, v0.11.2 (inactive)
markdown-preview-plus, v2.4.0 (active)
native-ui, v0.19.1 (inactive)
nucleus-dark-ui, v0.10.12 (inactive)
selection-highlight, v0.1.5 (active)
Termination, v0.1.4 (active)
unity-ui, v2.1.6 (inactive)
atom-dark-syntax, v0.27.0 (active)
atom-dark-ui, v0.52.0 (active)
atom-light-syntax, v0.28.0 (inactive)
atom-light-ui, v0.44.0 (inactive)
base16-tomorrow-dark-theme, v1.2.0 (inactive)
base16-tomorrow-light-theme, v1.2.0 (inactive)
one-dark-ui, v1.6.0 (inactive)
one-light-ui, v1.6.0 (inactive)
one-dark-syntax, v1.3.0 (inactive)
one-light-syntax, v1.3.0 (inactive)
solarized-dark-syntax, v1.0.2 (inactive)
solarized-light-syntax, v1.0.2 (inactive)
about, v1.7.0 (active)
archive-view, v0.61.1 (active)
autocomplete-atom-api, v0.10.0 (active)
autocomplete-css, v0.11.2 (active)
autocomplete-html, v0.7.2 (active)
autocomplete-plus, v2.31.1 (active)
autocomplete-snippets, v1.11.0 (active)
autoflow, v0.27.0 (inactive)
autosave, v0.23.1 (active)
background-tips, v0.26.1 (active)
bookmarks, v0.42.0 (active)
bracket-matcher, v0.82.1 (active)
command-palette, v0.38.0 (inactive)
deprecation-cop, v0.54.1 (active)
dev-live-reload, v0.47.0 (active)
encoding-selector, v0.22.0 (active)
exception-reporting, v0.40.0 (active)
find-and-replace, v0.201.1 (inactive)
fuzzy-finder, v1.4.0 (active)
git-diff, v1.1.0 (active)
go-to-line, v0.31.0 (inactive)
grammar-selector, v0.48.2 (active)
image-view, v0.59.0 (active)
incompatible-packages, v0.26.1 (active)
keybinding-resolver, v0.35.0 (active)
line-ending-selector, v0.5.0 (active)
link, v0.31.1 (inactive)
markdown-preview, v0.158.0 (active)
metrics, v1.0.0 (active)
notifications, v0.65.1 (active)
open-on-github, v1.2.0 (inactive)
package-generator, v1.0.0 (inactive)
settings-view, v0.242.2-hotfix1 (active)
snippets, v1.0.2 (active)
spell-check, v0.68.2 (active)
status-bar, v1.4.1 (active)
styleguide, v0.47.0 (active)
symbols-view, v0.113.1 (inactive)
tabs, v0.101.0 (active)
timecop, v0.33.2 (active)
tree-view, v0.209.3 (active)
update-package-dependencies, v0.10.0 (active)
welcome, v0.35.1 (inactive)
whitespace, v0.33.0 (active)
wrap-guide, v0.38.2 (active)
language-c, v0.52.1 (active)
language-clojure, v0.21.0 (active)
language-coffee-script, v0.47.2 (active)
language-csharp, v0.12.1 (active)
language-css, v0.37.1 (active)
language-gfm, v0.88.0 (active)
language-git, v0.15.0 (active)
language-go, v0.42.1 (active)
language-html, v0.45.1 (active)
language-hyperlink, v0.16.0 (active)
language-java, v0.23.0 (active)
language-javascript, v0.119.0 (active)
language-json, v0.18.2 (active)
language-less, v0.29.5 (active)
language-make, v0.22.2 (active)
language-mustache, v0.13.0 (active)
language-objective-c, v0.15.1 (active)
language-perl, v0.35.0 (active)
language-php, v0.37.2 (active)
language-property-list, v0.8.0 (active)
language-python, v0.45.0 (active)
language-ruby, v0.69.0 (active)
language-ruby-on-rails, v0.25.0 (active)
language-sass, v0.56.0 (active)
language-shellscript, v0.22.4 (active)
language-source, v0.9.0 (active)
language-sql, v0.23.0 (active)
language-text, v0.7.1 (active)
language-todo, v0.28.0 (active)
language-toml, v0.18.0 (active)
language-xml, v0.34.9 (active)
language-yaml, v0.26.0 (active)

# Dev
No dev packages

Thanks for the bug! (Should I really be thanking you for a bug? :stuck_out_tongue_winking_eye: ) I'm afraid I'm horrid with coffescript so it might take a while to get this straightened out.

No worries! I basically just clicked on the error window in atom and it automatically created the issue. Good luck with the coffee script!

@mannyhagman: I've got a potential fix in on a "dummy" Atom package. Could you install faketerm2 in Atom and let me know if it works?

Yes for sure. I'll update you soon.

@Fred-Barclay
Wow. fakeTerm2 is awesome. So far the best terminal package I have used on Atom.

Great! I'll be getting a fix in as soon as possible, then. You may see some weird stuff going on with termination or faketerm2 - just ignore it and don't update! :smile: I'll comment back here once the situation has stabilised.

Alright, hopefully termination is fixed. :tada: :  You'll need to completely uninstall it (if you haven't already) since I've reset the version numbers.

Thanks Fred, I'll be able to do that a bit later today and let you know what he results are.

Sounds good. Let me know when you find out (plus any suggestions, etc.) :smile:


Security issue in dependency

@ottovw Thanks for reporting! Will get this fixed in the next couple of days. I'm thinking about doing a complete re-write with Typescript.

@ottovw I've published a new release (v1.0.3). Please try the latest version, it should be good now.


Add a new monitor for Kubernetes.

Add a new monitor for Kubernetes, to be used directly by projects plugging into Kubernetes such as Trireme-Kubernetes and the Enforcer launched on Kubernetes

implemented by #495


jobsite: manager and builder

WIP
Has to be done:

[ ] How to handle the exports, currently both -pipeline and -export have to be defined and it feels very awkward.
[ ] Check subdir for export on builder side.
[ ] Only one export is allowed.
[ ] Comments and documentation.

Can be followup:

[ ] Logs need to be sent over in the /progress endpoint on the builder and handled in the manager.
[ ] Defining proper responses for all the routes in the builder.

This PR implements a new concept in osbuild-composer: the jobsite. The jobsite is an isolated set of services to run builds in. They can be separated from the rest of the system through network segmentation or other means. The jobsite and what's in it are only available to things inside it. One jobsite is used for each build, after which all resources used in it are removed again.
In the case of osbuild-composer the worker can set up a network, start a separate virtual machine containing the builder and then spawn the manager to point at it.
This implements the jobsite-manager and jobsite-builder. The builder is a thin wrapper around osbuild itself, it takes in a manifest; build options and allows to exports things.
The manager is responsible for sending the manifest and retrieving exports from the builder.
Because these commands are potentially running in untrusted environments they are quite picky about what they accept. Both will exit on any kind of error. The builder can only be used in the correct order, any out-of-order access will terminate the builder. This pickiness extends to file paths not existing or already existing when they're not expected to exist.
Steps
The manager and builder know a concept of "states". These are the steps that they both go through in lockstep. The steps are:

claim
provision
populate
build
progress
export

During the claim-step the manager takes ownership of the builder, this is currently a no-op but can contain some jobsite-lifetime secrets. After this the manager sends the manifest to the builder in the provision step. In the populate step the manager can send any required sources and other inputs to the builder, this is currently a no-op but something we want to do soon. The manager then instructs the builder to start through the build step, where it also tells the builder what things to export and some other tidbits related to the build.
The manager can now start polling the progress step. This endpoint will contain logs and progress information as it becomes available during build. Once the progress is done we transition into the export step where the manager retrieves the built artifacts from the builder. When this is done both exit and the jobsite can be torn down.

@croissanne I've addressed the initial concerns; I'll create a separate commit to add these to the specfile.

Thanks for the PR! This is a very thorough approach that also comes with some complexity. I wonder if we could simplify it a little bit. Part of the complexity seems to be coming from the fact that multiple endpoints have to be called in a specific order and this means a relativ complex API for both client and internal management. If this could be simplfied into a single endpoint that takes all required data it would reduce some of this maybe?
As an idea maybe we could instead just have a single endpoint, e.g. "/build" that would take a POST multipart/form-data [0]. In there we would have the manifest.json coming as a file, pipelines and environments as form data and (later) sources as a additional (tar) file. Once all that is uploaded the bulder would return a (e.g. randomly generated) URL with the progress of the build for the client. The client then just keeps polling on this endpoint until the result is ready for download. Is there any downside with this simpler approach (we can always add more later if we discover we need it)? Or any other things I'm not considering here?
This is cool stuff! The core is really good, I'm just wondering if we can reduce the complexity of the interface somehow.
[0] It's a bit ugly but it provides what we need and is a standard and can be driven from curl trivially which can be useful for e.g. testing.

Documentation is indeed sorely lacking; I'll be adding it in this PR and the commit message and PR start are not yet complete. A README is a good idea, perhaps it can be included in the website documentation as well @ochosi?
@mvo5 @ondrejbudai the split came naturally, it's possible to do all of this in a single big request especially since this initial implementation doesn't do much more than that. Perhaps that is a better approach.
My idea for splitting them out was to provision some secrets initially and to allow other routes to validate inputs according to the schema without having to receive all the data in a possibly very big and very long-to-transfer request.

Yup - I can pull in the readme as soon as something has landed. Should be a one-liner🤞 (unless you hide some extra markdown fun for me 😂)

However since we're not currently doing any of that it might be better to remove them again and then replace them as they're being implemented and as needed.

Should we block on this or will it be done in a followup?

However since we're not currently doing any of that it might be better to remove them again and then replace them as they're being implemented and as needed.

Should we block on this or will it be done in a followup?

I don't think we had a concrete outcome if we actually want to do that so let's put it in a followup if we want to simplify.


Can I parse HTML5 with this crate?

I've successfully used xmlparser to parse HTML, however I noticed that xmlparser does not accept attributes without quotes (and then some other things). Is this project willing to accept patches that are at least avoiding errors on html5 or is this entirely out of scope?

No. And this is not planed.
AFAIK, HTML5 is far more complex than simply omitting quotes around attribute values. It would be easier to simply fork the xmlparser and add an HTML specific stuff.

thanks, just as an update I have since created https://github.com/untitaker/html5gum which solves my needs.


Fixes #3127 - View Layout improvements (e.g. Remove CheckAbsolute)

Fixes:

Fixes #3127

Todo:

[x] Removes CheckAbosolute for Computed
[x] Merge fix for #3135 (Toplevel is confused about dimension - Should use Computed and Dim.Fill )
[x] Fix DialogTests.Location_When_Not_Application_Top_Not_Default
[x] Fix AutoSizeTextTests
[x] Fix LabelTests.Label_Draw_Vertical_Wide_TextAlignments
[x] Fix RadioGroupTests.DisplayMode_Width_Height_Vertical_Horizontal_Space
[x] Fix ScrollBarViewTests.Hosting_ShowBothScrollIndicator_Invisible
[x] Fix ScrollBarViewTests.InternalTests
[x] Fix ScrollViewTests.AutoHideScrollBars_False_ShowHorizontalScrollIndicator_ShowVerticalScrollIndicator
[x] Fix TextViewTests.UnwrappedCursorPosition_Event

@BDisp & @tznind: as I expected, a big challenge in all this will be getting AutoSize working again.... Esp vertical.
Right now almost all unit tests pass.
Those that don't:

A bunch of AutoSize = true pass. Many of the failing tests have to do with vertical orientation. Shocking, I know.
A Dialogs test that is odd
LabelTests, RadioGroupTests, TextViewTests - I suspect these are related to Autosize and will pass once that's figured out
ScrollBar/ViewTests - odd.

However, I have almost everything else working! I only had to change a small number of unit tests; and as I did so I tried to be very careful to ensure the tests tested the new "correct" behavior.
To be clear, significant behavior has changed as a result of what I've done here:
OnResizeNeeded no longer changes Frame
But calls SetRelativeLayout. This reduces the # of places Frame is mucked with and actually makes more sense: That is the whole point of SetRelativeLayout.
Toplevel now defaults to Width/Height = Dim.Fill (0)
And thus LayoutStyle.Computed.
SetRelativeLayout now works regardless of LayoutStyle.
It also, treats each of X, Y, Width, and Height as independent w.r.t. Computed:
var r = new Rect (newX, newY, newW, newH);
if (Frame != r) {
	// Set the frame. Do NOT use `Frame` as it overwrites X, Y, Width, and Height, making
	// the view LayoutStyle.Absolute.
	_frame = r;
	if (X is Pos.PosAbsolute) {
		_x = Frame.X;
	}
	if (Y is Pos.PosAbsolute) {
		_y = Frame.Y;
	}
	if (Width is Dim.DimAbsolute) {
		_width = Frame.Width;
	}
	if (Height is Dim.DimAbsolute) {
		_height = Frame.Height;
	}

	if (IsInitialized) {
		// TODO: Figure out what really is needed here. All unit tests (except AutoSize) pass as-is
		//LayoutFrames ();
		//TextFormatter.Size = GetTextFormatterSizeNeededForTextAndHotKey ();
		SetNeedsLayout ();
		//SetNeedsDisplay ();
	}

	// BUGBUG: Why is this AFTER setting Frame? Seems duplicative.
	if (!SetFrameToFitText ()) {
		TextFormatter.Size = GetTextFormatterSizeNeededForTextAndHotKey ();
	}
}

Frame
public virtual Rect Frame {
	get => _frame;
	set {
		_frame = new Rect (value.X, value.Y, Math.Max (value.Width, 0), Math.Max (value.Height, 0));

		// If Frame gets set, by definition, the View is now LayoutStyle.Absolute, so
		// set all Pos/Dim to Absolute values.
		_x = _frame.X;
		_y = _frame.Y;
		_width = _frame.Width;
		_height = _frame.Height;

		// TODO: Figure out if the below can be optimized.
		if (IsInitialized /*|| LayoutStyle == LayoutStyle.Absolute*/) {
			LayoutFrames ();
			TextFormatter.Size = GetTextFormatterSizeNeededForTextAndHotKey ();
			SetNeedsLayout ();
			SetNeedsDisplay ();
		}
	}
}

I'm sure more is broken as this is a DEEP change, but I think you'll agree it is far more "correct" and will simplify things moving forward so it's worth it... 😶

All unit tests pass.
Now, what's needed is:

[ ] A code review from @BDisp  and @tznind
[ ] A bunch of manual testing to find out what I broke that's not tested by unit tests.

The SyncrhonizationContextTests is failing.

The ColorPicker View is producing exactly the results I would expect it to. It has a bug in how it is trying to implement it's own "AutoSize" given the new design of Frame that I've implemented in this PR. Specifically,
private void SetInitialProperties ()
{
	CanFocus = true;
	AddCommands ();
	AddKeyBindings ();
	LayoutStarted += (o, a) => {
		Bounds = new Rect (Bounds.Location, new Size (_cols * BoxWidth, _rows * BoxHeight));
	};
}

Setting Bounds has the same effect as setting Frame.
To fix this, ColorPicker should change its logic for positioning the background color picker:
void SetInitialProperties ()
{
	CanFocus = true;
	AddCommands ();
	AddKeyBindings ();
	LayoutStarted += (o, a) => {
		var thickness = GetFramesThickness ();
		Width  = _cols * BoxWidth + thickness.Vertical;
		Height = _rows * BoxHeight + thickness.Horizontal;
	};
}

The SyncrhonizationContextTests is failing.

I can't reproduce this. Weird as it should have nothing to do with my changes.

I can't reproduce this. Weird as it should have nothing to do with my changes.

I'm using your PR and only happens when I run all the unit tests. But I also can't pull from your remote because I get some git errors and I don't know why.

I can't reproduce this. Weird as it should have nothing to do with my changes.

I'm using your PR and only happens when I run all the unit tests. But I also can't pull from your remote because I get some git errors and I don't know why.

What's the error?
All tests pass for me with:

VS 2022's Test Explorer - both Windows and Ubuntu
Resharpers' Unit Test Explorer
Github action on Push
dotnet clean & dotnet test

What's the error?
All tests pass for me with:

VS 2022's Test Explorer - both Windows and Ubuntu
Resharpers' Unit Test Explorer
Github action on Push
dotnet clean & dotnet test

I deleted your remote from the .git and fetch again and now it isn't fail anymore. I think the culprit was two branches where you submit 2 PR to my branch with the same name but one start with capital letter and the other with lower letter.
bDisp-v2_vkeypacketsimulator-fix_3054
BbDisp-v2_vkeypacketsimulator-fix_3054

Waiting for the Dim.Auto to see the AutoSize behavior.

So am I. 😜
Don't hold your breath. It's a lot of work and I don't know when I can get to it again. But I'm confident this PR will make it easier.

I think this is ready to merge.
@BDisp ok?
@tznind ok?


exclude InfraID from immutability check

What this PR does / why we need it:
HostedCluster spec.infraID can be defaulted by the HO if not set at create time.
Also incorporated feedback from https://github.com/openshift/hypershift/pull/1262#issuecomment-1097136978
Which issue(s) this PR fixes (optional, use fixes #<issue_number>(, fixes #<issue_number>, ...) format, where issue_number might be a GitHub issue, or a Jira story:
Fixes #
Checklist

[ ] Subject and description added to both, commit and PR.
[ ] Relevant issues have been referenced.
[ ] This change includes docs.
[ ] This change includes unit tests.

/test-e2e-aws-all

/lgtm
/hold

Whoops typo before...
/test e2e-aws-all

/test e2e-aws-all

/lgtm

/hold cancel


Improve README.rst

Fixed anchors that did not work on GitHub as mentioned in #92.
Additionally included info on the command_path settings variable and let users know the demo opens a Youtube video.

thanks a lot.


Problems with upgrade check

I ran dxdiag on my Windows 10 laptop. It reported no problems and found WDDM 2.1 and DirectX, but that doesn't appear to be sufficient for a Win11 upgrade on the machine. (Everything else passed OK on WhyNotWin11 except that my CPU is 6th generation, not the current 11th.)
[Enter feedback here]

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: db554047-b9ca-f5c9-d79f-4a70686a66db
Version Independent ID: e05bf4bc-ab29-9d32-bd53-5ca3e70aba1b
Content: Windows 11 requirements - What's new in Windows
Content Source: windows/whats-new/windows-11-requirements.md
Product: w11
GitHub Login: @greg-lindsay
Microsoft Alias: greglin

@twasserman . We cannot give comments on Microsoft windows OS policy, I am a normal user like you having a laptop with intel i5 7thgen 7200U, even though all requirements are met, but not in the windows 11 CPU list.
We can do something as of now, please write your concern as feedback in feedback hub.


notmuch -- respect +afew feature flag, fix sync buffer kill

When #3933 was closed, part of it's documentation snuck in. The part about the +afew feature flag is in the docs, but it's not respected.
This change will make it be respected. It also edits the notmuch sync function to use compile in order to address the problems described in https://github.com/hlissner/doom-emacs/issues/3188
Please consider this for merge.

Thanks!


boto3 support

Is there a moto3, that will support boto3?

+1

Moto implements the AWS APIs and it should be library agnostic. If any service or endpoint doesn't work with boto3, please open an issue. This is a duplicate of https://github.com/spulec/moto/issues/459.


BUG : Start game button not working

When opening the deployed link the start game button not working. I want to solve this issue. Please assign me this issue.

I really like your app and want to work on your project ,please assign this issue to me under gssoc-ext 2024


When will sharp support esm?

Feature request
What are you trying to achieve?
I want to use sharp in my netlify app's serverless function. But it cannot work with require is not defined error. So I think if sharp support esm it maybe get rid of this situation.
When you searched for similar feature requests, what did you find that might be related?

What would you expect the API to look like?

What alternatives have you considered?

Please provide sample image(s) that help explain this feature

Please see https://github.com/lovell/sharp/issues/2981


Fix pydantic related errors on FirecrawlScrapeWebsiteTool

Fix #131

Disclaimer: This review was made by a crew of AI Agents.
Code Review Comment for PR #132
Overview
This pull request introduces significant modifications to the firecrawl_scrape_website_tool.py file aimed at addressing Pydantic-related errors, enhancing import structure, and bolstering error handling mechanisms. Below are detailed observations and suggestions regarding the changes made.
Key Improvements

New Import: The addition of ConfigDict from Pydantic allows for more nuanced model configurations and accommodates arbitrary types.
Error Handling: Improved handling of the FirecrawlApp import provides more clarity on package requirements, enhancing user guidance when the dependency is missing.

Suggested Improvements
1. Import Organization
To adhere to Python's best practices for import organization, consider structuring the imports as follows:
# Standard library imports
from typing import TYPE_CHECKING, Any, Dict, Optional

# Third-party imports
from pydantic import BaseModel, Field, ConfigDict

# Local imports
from crewai_tools.tools.base_tool import BaseTool

# Conditional imports should be at the top
try:
    from firecrawl import FirecrawlApp
except ImportError:
    raise ImportError(
        "`firecrawl` package not found, please run `pip install firecrawl-py`"
    )

2. Enhance Import Error Check Placement
Moving the import error handling to the top of the file promotes an early failure strategy, which can prevent unnecessary execution of code when critical dependencies are absent.
3. Improve Type Hints in the _run Method
Adding more specific type hints can enhance clarity and aid in development:
async def _run(self, url: str, max_depth: Optional[int] = None, max_pages: Optional[int] = None, timeout: Optional[int] = None) -> Dict[str, Any]:

4. Explicit Model Configuration
The model configuration should be more explicit to clarify the expected behavior:
class FirecrawlScrapeWebsiteTool(BaseTool):
    model_config = ConfigDict(
        arbitrary_types_allowed=True,
        validate_assignment=True,
        frozen=False
    )

Additional Recommendations

Documentation Enhancements: Additional docstrings for the FirecrawlScrapeWebsiteToolSchema class should clarify the purpose of each attribute, which aids maintainability and user understanding.

Refine Error Handling: It’s advisable to enhance error handling within the _run method to ensure that specific exceptions are raised, providing clearer feedback to users in case of scraping failures.

Comments on Complex Logic: Implementing comments where code may not be immediately clear will help future developers understand the reasoning behind certain implementations.

Conclusion
The enhancements made in PR #132 effectively address specific Pydantic-related errors and improve import handling in firecrawl_scrape_website_tool.py. However, following the recommendations for import organization, type hints, documentation, and error handling would further improve the robustness and maintainability of the code.
Implementing these changes will yield a more professional and dependable codebase. Thank you for your work on this PR, and I look forward to seeing the updates based on this feedback!

Is this ready to merge? 👀
Caleb from Firecrawl


Setup website for docs

Perceived Problem

README will not scale for the kind of detailed and verbose documentation that Nexus Prisma needs
Many large code blocks needed to span the stack that Nexus Prisma sits across (db<>seed<>api<>client<>query<>json result)

Ideas / Proposed Solution(s)

https://github.com/shuding/nextra
https://github.com/remorses/dokz
integrate with GH Pages
Setup custom domain at https://nexus.prisma.io

How about (Docusaurus)[https://docusaurus.io/]?

@iddan would prefer nextra or dokz, at least last I liked.

:tada: This issue has been resolved in version 1.0.0 :tada:
The release is available on:

npm package (@next-major dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


rename main executable to something less ambiguous

Comment:
Hi!
Thanks for your work!
Should we rename the final executable from main to llama-main with something like script: mv $PREFIX/bin/main $PREFIX/bin/llama-main?
We could also patch the Makefile.
Would that interfere with dependent packages like e.g. llama-cpp-python?

we can definitely do that. I did not rename it at first because I was trying to keep the build as close to upstream as possible before I started making changes.
With that being said, the llama-cpp-python does not depend on llama.cpp-feedstock. At least not yet.

My next goal is to get the windows build to work. We will follow from there


Warning while compiling hpx

While compiling the latest master branch of hpx, invalid attribute warnings are shown.
The full log of the compilation is linked below.
https://gist.github.com/justwagle/c1a16b6d2dc5bacc8924396bb96bf562

This is a side-effect of the deprecation of hpx_fwd.hpp. Presumably those forward declarations mentioned in the warnings are seen by the compiler after the class was already defined (with the correct export attribute).

I encountered a similar issue recently where the thread class is already defined.
Output:
https://gist.github.com/W4RH4WK/abf89baf6e03a9c0ba0f6cba170680f9
Minimum Example:
#include <hpx/include/serialization.hpp>
#include <hpx/runtime/serialization/list.hpp>
#include <hpx/include/threads.hpp>

#include <iostream>

int main(void) {
        std::cout << "Hello World" << std::endl;
        return 0;
}

Currently on:
b603449e6b97ced17adb829b07413f41b2ab6131

@W4RH4WK Thanks for this report. This should be fixed by merging #2429.


Enhancing scenario_simulator_v2 to print or visualize parameters/modifiers used for test iterations

Is your feature request related to a problem? Please describe.
While debugging scenarios with different values (list) for modifiers, it is not clear which values is selected and used between different combinations in a specific test iteration.
For example, a scenario with modifiers like that ...
    - name: __tier4_modifier_an
      list:
        - 0.2
        - 2
    - name: __tier4_modifier_D
      list:
        - 5
        - 10

it becomes confusing which values of __tier4_modifier_an is used with which value of  __tier4_modifier_D in an iteration
Describe the solution you'd like
It will be helpful if the parameters and modifiers used for each iteration is visualized in RViz.
Describe alternatives you've considered
It could a valid alternative to have it only printed in the terminal instead of text visualization in RViz
Additional context

@yamacir-kit Do you have any opinion about how these information should be output to the user? (in Rviz, in terminal, make the verbosity configurable, etc.)


Exp14: NVP_4 vs NVP

Run NVP_4 for 90 epochs with a lsdim of 500.

Script has been written.  Waiting to run until we decide how many epochs to test it over.

Script has been written and ready to run.

Running

Experiment terminated after ~60 epochs. Loss failed to reduce past 140 consistently while NVP succeeds at a significantly reduced run time. Considerations for improvements on NVP_4 are welcome. It's worth noting that we may want to revisit this experiment once learning-rate/batch-size scheduling is up and running. I'm inclined to believe we may get different results upon either increasing the learning rate, or decreasing the batch size at the start and scheduling them. Another approach is to implement #39.

Running again with increased learning rate just for comparison's sake

Significant improvement has been observed in NVP_4 when utilizing learning rate of 1e-4. For future considerations we ought to test 1e-3, as well as learning rate scheduling strategies.