Doc download

The content of the whole topbar is compared when setting use_download_button on or off.
Tests re inspired from a similar tests for the edit issue button.

very nice! this looks great - one thought, can you collapse them into a single test_topbar_download_button test, rather than having one for "on" and one for "off"? Just add a comment before each block to describe what you're testing for

I tried this:
def test_topbar_download_button(sphinx_build_factory, file_regression):
    """Test building with download button on or off."""

    # Button on
    confoverrides = {
        "html_theme_options.use_download_button": True,
    }
    sphinx_build = sphinx_build_factory("base", confoverrides=confoverrides).build(
        assert_pass=True
    )

    source_btns = sphinx_build.html_tree("section1", "ntbk.html").find_all(
        "div", attrs={"class": "topbar-main"}
    )[0]
    file_regression.check(source_btns.prettify(), extension="_on.html", encoding="utf8")

    # Button off
    confoverrides = {
        "html_theme_options.use_download_button": True,
    }
    sphinx_build = sphinx_build_factory("base", confoverrides=confoverrides).build(
        assert_pass=True
    )

    source_btns = sphinx_build.html_tree("section1", "ntbk.html").find_all(
        "div", attrs={"class": "topbar-main"}
    )[0]
    file_regression.check(source_btns.prettify(), extension="_off.html", encoding="utf8")

But I then get an error message during the copytree operation for the second test. The folder test_topbar_download_button0 already exists. Is there a way to create a different test folder for the second test?

ah hmmm - how about instead we only test for when the topbar is set to "off", since it is "on" by default and we are implicitly testing this in the default book build tests?

@choldgraf The tests now only check the option to set the download button to off...

Great - thanks very much!! 🎉


use capitalized suffixes

Currently the project mixes three styles:

lower-case prefix (e.g. R.pPipe)
lower-case suffix (e.g. R.foldl)
upper-case suffix (e.g. R.liftN)

foldL/foldR just feels wrong

If this change were to be made, please make sure that the API is not broken between two minor versions without any deprecation notice.

@TheLudd Until the 1.0.0 is released, the API can be changed without any deprecation notice.

@honzabrecka That is very correct. But I would say it is still a very bad thing to do which is why we have #769

:+1: @TheLudd it would be nice to help those early adopters, even though it is not required

This is starting to make me think again about a mailing list or forum tool where we could announce such changes before they go live.   Anyone have recent experience setting something up?  Would we want the extras hassle?

expanding on my uneasiness about foldl: internal consistency is fine, but we should beware of breaking with expectations from the outside world as well (see reduce). I have never seen foldl with a capital L and would find that very surprising indeed. The other changes don't strike me as problematic.

expanding on my uneasiness about foldl: internal consistency is fine, but we should beware of breaking with expectations from the outside world as well (see reduce). I have never seen foldl with a capital L and would find that very surprising indeed.

I find it hard to justify R.foldl alongside R.scanL, R.partialL, and R.mapAccumL. Although conforming to initial expectations is a good idea, if every API were to do so religiously each would, at best, be as consistent as its predecessor. Entropy would increase over time.
We are not wedded to the names of JavaScript built-ins or Haskell functions or anything else. Ramda's names should be right in the context of Ramda. I don't mind learning an API which doesn't conform to my initial expectations if it's internally consistent. After learning a portion of the system I'll be able to make assumptions—correct assumptions—about how the rest of the system works. Ramda's argument ordering is a good example of this phenomenon.

@davidchambers I agree with you -- 99% :smile: foldL is giving me heartburn. i think it makes more sense to drop foldl for reduce than go with foldL.

it makes more sense to drop foldl for reduce than go with foldL.

Works for me! :grinning:

Would we also change R.unfoldr to R.unreduceRight? :stuck_out_tongue_winking_eye:

Would we also change R.unfoldr to R.unreduceRight?

good yet whimsical question!
i'd be happy with unfoldr becoming simply  unfold

Where do you stand, @CrossEye?

unfold sounds good to me.  And I think reduce is the winner here.  That would save us the alias heartburn for another day.  Besides that, LGTM.

Before I make any changes, let's clarify what you're suggesting. Here's my understanding:

R.foldl → R.reduce
R.foldr → R.reduceRight
R.foldlIndexed → R.reduceIndexed
R.foldrIndexed → R.reduceRightIndexed
R.unfoldr → R.unfold
R.lPartial → R.partial
R.rPartial → R.partialRight
R.mapAccumL → R.mapAccum
R.mapAccumR → R.mapAccumRight
R.scanl → R.scan
R.pCompose → R.composeP
R.pPipe → R.pipeP

For consistency R.unfold should be R.unfoldRight. From the Haskell docs:

In some cases, unfoldr can undo a foldr operation

We're already breaking the fold–unfold symmetry, though, so perhaps this isn't necessary.
I prefer the naming scheme employed by this pull request as it currently stands. If others feel strongly that the verb/verbRight scheme is preferable I will update this pull request. Consistency is valuable even if the names are not quite to my liking. :)

_http://stackoverflow.com/questions/23779494/why-the-r-in-unfoldr
I think unfold is fine. otherwise, :cow2:

:herb:

@CrossEye, does the :herb: apply to my comment above or to the pull request in its current form?

I was responding to your comment; and somehow I missed the final paragraph of it.  I like the names in the comment better than those in the PR.  But I agree that consistency is better than any particular set of names.  I think we learned recently that users are much more invested in reduce than fold, and if we're not going to have the alias, then the reduce versions in this list are probably for the best.

Okay. I'll update the pull request now.

As an alternative to .pipeP and .composeP, perhaps .pipeAsync and .composeAsync?

@yuri: I think .pipeP/.composeP and/or pipePromise/composePromise is more clear.  These functions handle only Promises, not the full range of ways in which one can do asynchronous processing in Javascript, including WebWorkers, Server Push, and setTimeout.  (Yes, I know that Promises can be wrapped around any of these, but then we're back to dealing with Promises.)

@CrossEye The downside of of "composeP" is that it's not immediately clear what "P" stands for. "composePromise" is more explicit, but seems inaccurate, since this function does not "compose a promise". Based on how "Async" suffix is normally used in JavaScript today, I think "composeAsync" will lead people to expect a function that either works with callbacks or with promises. I don't think anyone would expect any of the other behaviours from a functions named this way. Since "compose" and "pipe" are variadic, there is no straightforward way of using callbacks with them, so I think the fact that composeAsync works only with promises shouldn't surprise anyone.

"composePromise" is more explicit, but seems inaccurate, since this function does not "compose a promise"

Neither does reduceRight reduce a right, but it might still be the best fit for our naming conventions.
However, I don't feel strongly about this, and I don't really object to Async.  I really want to get our naming settled down soon, though!
Do others have opinions?

Do others have opinions?

Not on this one. All three options seem reasonable.

:no_mouth:

Let's leave them for now and see if we get questions about them.  If they seem confusing, we can address it again.


rely on udev to manage symlinks, closes #2 (with suggestions)

Based on feedback from @brahman81's PR: https://github.com/oogali/ebs-automatic-nvme-mapping/pull/3

Error out from script early, rather than silently fail.
Test that input is a valid block device early.
Update README with script changes.

Hopefully I've addressed all the suggestions, questions and concerns you have. Please let me know if there's anything else required for this PR.


[move-prover] Sort of fixing a bytecode verification error which broke CI

It appears #3633 broke Ci for the move prover via bytecode verification error in the older copy of the stdlib we are using. The culprit has been identified by inserting debug prints into the bytecode verifier. Because its not obvious why its failing, the according code has been commented out, given that we retire this copy of the stdlib soon anyway.
Motivation
Unblock CI.
Have you read the Contributing Guidelines on pull requests?
Yes
Test Plan
Existing tests
Related PRs
#3633

@bors-libra r=DavidLDill

:pushpin: Commit 1aea70f has been approved by DavidLDill

:hourglass: Testing commit 1aea70f48f5eb6e077d9fed1d2d6d2a0876c3263 with merge 74df1b07982572455cfddb2bef0da55c99ad0d48...

:sunny: Test successful - checks-actions_land_blocking_test, checks-circle_commit_workflow
Approved by: DavidLDill
Pushing 74df1b07982572455cfddb2bef0da55c99ad0d48 to master...


Valid moves?

How to represent that a movement is valid? Prefer CSS solution

Maybe showing the possible movements with a "greener" background?

How to do this using pure css selection?
I mean, how to inspect if a neighbor td has active true? And of vertical, how to inspect a td in a "uncle" tr?


Explicit init command

Would be nice to have an explicit command for creating a Pipfile, at the moment I think I discovered that you do it with pipenv --three via the animated gif.

Hey @rapilabs, the intended workflow is that your Pipfile will be created when you start your project. Any command you run with pipenv will create a new Pipfile and virtual environment if one doesn't already exist for the project.
If you're starting a new project, pipenv --three, pipenv --two, or pipenv --python python-x.x are the commands you would use. These are documented in multiple places in the README and on the home page of the documentation website, pipenv.org.
I'm not sure I see a point in having a command that only creates the Pipfile. This is primarily because it's an additional step most people will find cumbersome, and you can't do anything with the Pipfile until the virtualenv is initialized anyway. For that reason, I'm going to close this out, but please let us know if you have any further feedback or thoughts. Thanks!


Divide keyboard into keycode

keyboard.rb is too long, so I split it.

Sorry, I don't merge it.
I agree with that keyboard.rb is long but it doesn't make me want to split the file.


docs: Issue and PR templates

Codecov Report
Patch and project coverage have no change.

Comparison is base (0edc5ca) 90.29% compared to head (5dccd38) 90.29%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
Additional details and impacted files
@@           Coverage Diff           @@
##             main     #307   +/-   ##
=======================================
  Coverage   90.29%   90.29%
=======================================
  Files          29       29
  Lines        5410     5410
  Branches      640      640
=======================================
  Hits         4885     4885
  Misses        524      524
  Partials        1        1

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Routinator won't listen on all IPv6 enabled interfaces

Hi, I'm trying to run the routinator process to listen on all IPv4 and IPv6 addresses.  I'm able to get it to listen to all interfaces for IPv4 but not IPv6.  I'm running the following routinator command:
/home/routinator/.cargo/bin/routinator -v -b /opt/routinator server --http <IP_ADDRESS>:8080 --rtr <IP_ADDRESS>:8323 --rtr [::]:8323
but it's only listening on IPv4:
sudo lsof -i -n -P | grep 8323
routinato 23541 routinator    3u  IPv4 756991      0t0  TCP *:8323 (LISTEN)

On Linux, there is some kind of overlap between IPv4 and IPv6. You can’t bind to all interfaces on both on the same port. There should be an ‘address already in use’ warning in your log.
However – and again: on Linux – IPv6 includes IPv4, so you only need to bind to --rtr [::]:8323.
Note that starting with 0.7.0, Routinator will exit on these kinds of errors instead of happily running with what it got.

Thanks for the reply.  I validated that this is working.


Github plugin diff comment optimization

Last updated: 2017-06-10 21:29:57
Diff plugin outputs test

Last updated: 2017-06-10 21:39:27
Diff plugin outputs test

Last updated: 2017-06-10 21:40:05
Diff plugin outputs test

Last updated: 2017-06-10 21:40:28
Diff plugin outputs test

Last updated: 2017-06-10 21:40:51
Diff plugin outputs test
No fixture or plugin files were changed in this PR.

Last updated: 2017-06-10 21:42:22
Diff plugin outputs test
No fixture or plugin files were changed in this PR.


Added Roberta model with base and large configs for question answering

Roberta now running on IPU :)
Tested base and large on squad v1.
Base:
wandb:                       train/loss 0.6118
wandb:              train/learning_rate 0.0
wandb:                      train/epoch 3.0
wandb:                train/global_step 516
wandb:                         _runtime 458
wandb:                       _timestamp<PHONE_NUMBER>
wandb:                            _step 52
wandb:              train/train_runtime 268.7952
wandb:   train/train_samples_per_second 988.5
wandb:     train/train_steps_per_second 1.92
wandb:                 train/total_flos 6.627121224537342e+18
wandb:                 train/train_loss 1.0056
wandb:                 eval/exact_match 85.75213
wandb:                          eval/f1 92.08115

Large:
wandb:                       train/loss 0.5226
wandb:              train/learning_rate 0.0
wandb:                      train/epoch 3.0
wandb:                train/global_step 261
wandb:                         _runtime 983
wandb:                       _timestamp<PHONE_NUMBER>
wandb:                            _step 27
wandb:              train/train_runtime 636.8869
wandb:   train/train_samples_per_second 417.192
wandb:     train/train_steps_per_second 0.41
wandb:                 train/total_flos 9.531261908893355e+19
wandb:                 train/train_loss 0.97289
wandb:                 eval/exact_match 88.69442
wandb:                          eval/f1 94.4119

Nice work @jimypbr !


Flash-attention under Triton 2.0

Hi all,
Thanks for your excellent work. I met the following problem when using triton == 2.0.0. Forward succeed, but backward failed. How can I solve it, thanks.
error: 'scf.for' op expects region #0 to have 0 or 1 blocks
And when I comment out these lines, the error disappers.
https://github.com/HazyResearch/flash-attention/blob/main/flash_attn/flash_attn_triton.py#L322-L327
The full error message is:
UserWarning: While `attn_impl: triton` can be faster than `attn_impl: flash` it uses more memory. When training larger models this can trigger alloc retries which hurts perfor
mance. If encountered, we recommend using `attn_impl: flash` if your model does not use `alibi` or `prefix_lm`.
  warnings.warn('While `attn_impl: triton` can be faster than `attn_impl: flash` ' + 'it uses more memory. When training larger models this can trigger ' + 'alloc retries which hurts performance. If encountered, we recommend ' + 'using
`attn_impl: flash` if your model does not use `alibi` or `prefix_lm`.')

error: 'scf.for' op expects region #0 to have 0 or 1 blocks
Traceback (most recent call last):
  File "<string>", line 21, in _bwd_kernel
KeyError: ('2-.-0-.-0-7d1eb0d2fed8ff2032dccb99c2cc311a-d6252949da17ceb5f3a278a70250af13-3b85c7bef5f0a641282f3b73af50f599-3d2aedeb40d6d81c66a42791e268f98b-3498c340fd4b6ee7805fd54b882a04f5-e1f133f98d04093da2078dfc51c36b72-b26258bf01f83919
9e39d64851821f26-d7c06e3b46e708006c15224aac7a1378-f585402118c8a136948ce0a49cfe122c', (torch.float16, torch.float16, torch.float16, None, torch.float16, torch.float32, torch.float16, torch.float16, torch.float32, torch.float32, 'fp32', '
i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32'), ('none', True, 64,
False, False, False, True, 128, 128), (True, True, True, (False,), True, True, True, True, True, True, (False,), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (Tr
ue, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False),
 (False, False), (False, False), (False, False), (True, False), (True, False), (False, False), (False, False)))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "xxxx/attention.py", line 308, in <module>
    l.backward()
  File "~/anaconda3/lib/python3.10/site-packages/torch/_tensor.py", line 487, in backward
    torch.autograd.backward(
  File "~/anaconda3/lib/python3.10/site-packages/torch/autograd/__init__.py", line 200, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
  File "~/anaconda3/lib/python3.10/site-packages/torch/autograd/function.py", line 274, in apply
    return user_fn(self, *args)
  File "~/anaconda3/lib/python3.10/site-packages/flash_attn-0.2.8-py3.10-linux-x86_64.egg/flash_attn/flash_attn_triton.py", line 830, in backward
    _flash_attn_backward(do, q, k, v, o, lse, dq, dk, dv,
  File "~/anaconda3/lib/python3.10/site-packages/flash_attn-0.2.8-py3.10-linux-x86_64.egg/flash_attn/flash_attn_triton.py", line 698, in _flash_attn_backward
    _bwd_kernel[grid](
  File "~/anaconda3/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 77, in run
    timings = {config: self._bench(*args, config=config, **kwargs)
  File "~/anaconda3/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 77, in <dictcomp>
    timings = {config: self._bench(*args, config=config, **kwargs)
  File "~/anaconda3/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 65, in _bench
    return do_bench(kernel_call)
  File "~/anaconda3/lib/python3.10/site-packages/triton/testing.py", line 143, in do_bench
    fn()
  File "~/anaconda3/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 63, in kernel_call
    self.fn.run(*args, num_warps=config.num_warps, num_stages=config.num_stages, **current)
  File "~/anaconda3/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 199, in run
    return self.fn.run(*args, **kwargs)
  File "<string>", line 41, in _bwd_kernel
  File "~/anaconda3/lib/python3.10/site-packages/triton/compiler.py", line 1621, in compile
    next_module = compile(module)
  File "~/anaconda3/lib/python3.10/site-packages/triton/compiler.py", line 1550, in <lambda>
    lambda src: ast_to_ttir(src, signature, configs[0], constants)),
  File "~/anaconda3/lib/python3.10/site-packages/triton/compiler.py", line 963, in ast_to_ttir
    return optimize_triton_ir(mod)
  File "~/anaconda3/lib/python3.10/site-packages/triton/compiler.py", line 957, in optimize_triton_ir
    pm.run(mod)
RuntimeError: PassManager::run failed

And when I comment out https://github.com/HazyResearch/flash-attention/blob/main/flash_attn/flash_attn_triton.py#L322-L327.
If I run something like following code and set head dim = 128, (for example, the size of q is (2, 128, 2, 128) ) the procedure will hang up there. And head dim = 64 (2, 128, 2, 64) works fine.
flash_attn_triton.flash_attn_func(
      q, k, v, None, True/False, None
)

It's the Triton version. As mentioned at the beginning of the file:
Tested with triton==2.0.0.dev20221202.
Triton 2.0 has a new backend (MLIR) but seems like it doesn't yet work for head dimensions
other than 64:
https://github.com/openai/triton/blob/d376020f90002757eea3ea9475d4f7cfc2ec5ead/python/triton/ops/flash_attention.py#L207
We'll update this implementation with the new Triton backend once this is fixed.

It's the Triton version. As mentioned at the beginning of the file:
Tested with triton==2.0.0.dev20221202.
Triton 2.0 has a new backend (MLIR) but seems like it doesn't yet work for head dimensions
other than 64:
https://github.com/openai/triton/blob/d376020f90002757eea3ea9475d4f7cfc2ec5ead/python/triton/ops/flash_attention.py#L207
We'll update this implementation with the new Triton backend once this is fixed.

Hey, @tridao. How do you install this version, though? It seems that PyTorch 2.0.0 and 2.0.1 both require triton 2.0.0. If I try to run pip install triton==2.0.0.dev20221202, I see that it starts downloading an older PyTorch version without CUDA (Downloading torch-1.13.1-cp310-cp310-manylinux1_x86_64.whl (887.5 MB)). If I install it with --no-deps, I get this error:
fatal error: cuda.h: No such file or directory
    2 | #include "cuda.h"
      |          ^~~~~~~~
compilation terminated.

So, I wonder how I should install this specific triton version.

Yeah idk, it works fine with the Pytorch container from Nvidia (e.g. 23.05 has Pytorch 2.0) by pip install triton==2.0.0.dev20221202.

Yeah idk, it works fine with the Pytorch container from Nvidia (e.g. 23.05 has Pytorch 2.0) by pip install triton==2.0.0.dev20221202.

I tried to install it using conda instead of pipenv with the following environment.tml:
name: page-tagging-svc
channels:
  - pytorch
  - nvidia
  - conda-forge
dependencies:
  - python=3.10
  - pytorch=2.0
  - pytorch-cuda=11.7
  - transformers=4.29
  - datasets=2.12
  - pytorch-lightning=2.0
  - torchmetrics=0.11
  - pydantic=1.10
  - python-json-logger=2.0
  - scikit-learn=1.2
  - xgboost=1.7
  - pandas=1.5
  - beautifulsoup4=4.12
  - sentencepiece=0.1
  - fastcore=1.5
  # Dev
  - black=23.3
  - isort=5.12
  - mypy=1.3
  - pylint=2.17
  - pytest=7.3
  - pytest-cov=4.1
  - pytest-mock=3.10
  - click=8.1
  - google-cloud-storage=2.9
  - ultimate-sitemap-parser=0.5
  - pycountry=22.3
  - pillow=9.5
  - jupyterlab=4.0
  - matplotlib=3.7
  - scaleapi=2.14
  - google-cloud-aiplatform=1.26
  - pip
  - pip:
    - jsonargparse[signatures]==4.20.1
    - bentoml==1.0.19
    - starlette==0.24.*
    - iterative-stratification==0.1.7
    - triton==2.0.0.dev20221202

I checked that the correct version of triton was installed. However, I still get:
/tmp/tmpz4gpqr27/main.c:2:10: fatal error: cuda.h: No such file or directory
    2 | #include "cuda.h"
      |          ^~~~~~~~
compilation terminated.
Traceback (most recent call last):
  File "<string>", line 21, in _fwd_kernel
KeyError: ('2-.-0-.-0--2b0c5161c53c71b37ae20a9996ee4bb8-c1f92808b4e4644c1732e8338187ac87-d962222789c30252d492a16cca3bf467-12f7ac1ca211e037f62a7c0c323d9990-5c5e32ff210f3b7f56c98ca29917c25e-06f0df2d61979d629033f4a22eff5198-0dd03b0bd512a184b3512b278d9dfa59-d35ab04ae841e2714a253c523530b071', (torch.float16, torch.float16, torch.float16, torch.float32, torch.float16, torch.float32, torch.float32, 'fp32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32'), ('vector', False, 64, False, False, True, 128, 128), (True, True, True, True, True, True, True, (False,), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (False, False), (True, False), (True, False), (True, False), (True, False), (False, False), (False, False)))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 44, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 559, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 935, in _run
    results = self._run_stage()
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 976, in _run_stage
    self._run_sanity_check()
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1005, in _run_sanity_check
    val_loop.run()
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/pytorch_lightning/loops/utilities.py", line 174, in _decorator
    return loop_run(self, *args, **kwargs)
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/pytorch_lightning/loops/evaluation_loop.py", line 115, in run
    self._evaluation_step(batch, batch_idx, dataloader_idx)
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/pytorch_lightning/loops/evaluation_loop.py", line 375, in _evaluation_step
    output = call._call_strategy_hook(trainer, hook_name, *step_kwargs.values())
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 288, in _call_strategy_hook
    output = fn(*args, **kwargs)
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/pytorch_lightning/strategies/strategy.py", line 378, in validation_step
    return self.model.validation_step(*args, **kwargs)
  File "/home/fernando/klue_workspace/page-tagging-svc/src/torch/lightning.py", line 120, in validation_step
    logits, loss = self._step(batch)
  File "/home/fernando/klue_workspace/page-tagging-svc/src/torch/lightning.py", line 73, in _step
    logits = self._model(batch)
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/fernando/klue_workspace/page-tagging-svc/src/torch/model/markup_lm.py", line 161, in forward
    markup_lm_outputs = self.markup_lm(
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/transformers/models/markuplm/modeling_markuplm.py", line 915, in forward
    encoder_outputs = self.encoder(
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/transformers/models/markuplm/modeling_markuplm.py", line 665, in forward
    layer_outputs = layer_module(
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/transformers/models/markuplm/modeling_markuplm.py", line 549, in forward
    self_attention_outputs = self.attention(
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/transformers/models/markuplm/modeling_markuplm.py", line 507, in forward
    self_outputs = self.self(
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/fernando/klue_workspace/page-tagging-svc/src/torch/model/markup_lm.py", line 310, in forward
    context_layer = flash_attn_func(query_layer, key_layer, value_layer, attention_mask)
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/torch/autograd/function.py", line 506, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/home/fernando/klue_workspace/page-tagging-svc/src/torch/flash_attn_triton.py", line 810, in forward
    o, lse, ctx.softmax_scale = _flash_attn_forward(
  File "/home/fernando/klue_workspace/page-tagging-svc/src/torch/flash_attn_triton.py", line 623, in _flash_attn_forward
    _fwd_kernel[grid](
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/triton/runtime/jit.py", line 106, in launcher
    return self.run(*args, grid=grid, **kwargs)
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 200, in run
    return self.fn.run(*args, **kwargs)
  File "<string>", line 41, in _fwd_kernel
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/triton/compiler.py", line 1239, in compile
    so = _build(fn.__name__, src_path, tmpdir)
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/triton/compiler.py", line 1169, in _build
    ret = subprocess.check_call(cc_cmd)
  File "/home/fernando/anaconda3/envs/python_310_cuda_113/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/gcc', '/tmp/tmpz4gpqr27/main.c', '-O3', '-I/usr/local/cuda/include', '-I/home/fernando/anaconda3/envs/python_310_cuda_113/include/python3.10', '-I/tmp/tmpz4gpqr27', '-shared', '-fPIC', '-lcuda', '-o', '/tmp/tmpz4gpqr27/_fwd_kernel.cpython-310-x86_64-linux-gnu.so', '-L/usr/lib', '-L/usr/lib32']' returned non-zero exit status 1.
python-BaseException
Traceback (most recent call last):
  File "<string>", line 21, in _fwd_kernel
KeyError: ('2-.-0-.-0--2b0c5161c53c71b37ae20a9996ee4bb8-c1f92808b4e4644c1732e8338187ac87-d962222789c30252d492a16cca3bf467-12f7ac1ca211e037f62a7c0c323d9990-5c5e32ff210f3b7f56c98ca29917c25e-06f0df2d61979d629033f4a22eff5198-0dd03b0bd512a184b3512b278d9dfa59-d35ab04ae841e2714a253c523530b071', (torch.float16, torch.float16, torch.float16, torch.float32, torch.float16, torch.float32, torch.float32, 'fp32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32'), ('vector', False, 64, False, False, True, 128, 128), (True, True, True, True, True, True, True, (False,), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (False, False), (True, False), (True, False), (True, False), (True, False), (False, False), (False, False)))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/fernando/.local/share/JetBrains/Toolbox/apps/PyCharm-P/ch-0/223.8836.43/plugins/python/helpers/pydev/pydevd.py", line 2195, in <module>
    main()
  File "/home/fernando/.local/share/JetBrains/Toolbox/apps/PyCharm-P/ch-0/223.8836.43/plugins/python/helpers/pydev/pydevd.py", line 2177, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/home/fernando/.local/share/JetBrains/Toolbox/apps/PyCharm-P/ch-0/223.8836.43/plugins/python/helpers/pydev/pydevd.py", line 1489, in run
    return self._exec(is_module, entry_point_fn, module_name, file, globals, locals)
  File "/home/fernando/.local/share/JetBrains/Toolbox/apps/PyCharm-P/ch-0/223.8836.43/plugins/python/helpers/pydev/pydevd.py", line 1496, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/home/fernando/.local/share/JetBrains/Toolbox/apps/PyCharm-P/ch-0/223.8836.43/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/fernando/klue_workspace/page-tagging-svc/src/torch/trainer.py", line 106, in <module>
    cli_main()
  File "/home/fernando/klue_workspace/page-tagging-svc/src/torch/trainer.py", line 97, in cli_main
    cli = PageTaggingLightningCLI(
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/pytorch_lightning/cli.py", line 353, in __init__
    self._run_subcommand(self.subcommand)
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/pytorch_lightning/cli.py", line 642, in _run_subcommand
    fn(**fn_kwargs)
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 520, in fit
    call._call_and_handle_interrupt(
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 44, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 559, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 935, in _run
    results = self._run_stage()
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 976, in _run_stage
    self._run_sanity_check()
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1005, in _run_sanity_check
    val_loop.run()
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/pytorch_lightning/loops/utilities.py", line 174, in _decorator
    return loop_run(self, *args, **kwargs)
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/pytorch_lightning/loops/evaluation_loop.py", line 115, in run
    self._evaluation_step(batch, batch_idx, dataloader_idx)
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/pytorch_lightning/loops/evaluation_loop.py", line 375, in _evaluation_step
    output = call._call_strategy_hook(trainer, hook_name, *step_kwargs.values())
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 288, in _call_strategy_hook
    output = fn(*args, **kwargs)
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/pytorch_lightning/strategies/strategy.py", line 378, in validation_step
    return self.model.validation_step(*args, **kwargs)
  File "/home/fernando/klue_workspace/page-tagging-svc/src/torch/lightning.py", line 120, in validation_step
    logits, loss = self._step(batch)
  File "/home/fernando/klue_workspace/page-tagging-svc/src/torch/lightning.py", line 73, in _step
    logits = self._model(batch)
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/fernando/klue_workspace/page-tagging-svc/src/torch/model/markup_lm.py", line 161, in forward
    markup_lm_outputs = self.markup_lm(
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/transformers/models/markuplm/modeling_markuplm.py", line 915, in forward
    encoder_outputs = self.encoder(
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/transformers/models/markuplm/modeling_markuplm.py", line 665, in forward
    layer_outputs = layer_module(
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/transformers/models/markuplm/modeling_markuplm.py", line 549, in forward
    self_attention_outputs = self.attention(
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/transformers/models/markuplm/modeling_markuplm.py", line 507, in forward
    self_outputs = self.self(
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/fernando/klue_workspace/page-tagging-svc/src/torch/model/markup_lm.py", line 310, in forward
    context_layer = flash_attn_func(query_layer, key_layer, value_layer, attention_mask)
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/torch/autograd/function.py", line 506, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/home/fernando/klue_workspace/page-tagging-svc/src/torch/flash_attn_triton.py", line 810, in forward
    o, lse, ctx.softmax_scale = _flash_attn_forward(
  File "/home/fernando/klue_workspace/page-tagging-svc/src/torch/flash_attn_triton.py", line 623, in _flash_attn_forward
    _fwd_kernel[grid](
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/triton/runtime/jit.py", line 106, in launcher
    return self.run(*args, grid=grid, **kwargs)
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 200, in run
    return self.fn.run(*args, **kwargs)
  File "<string>", line 41, in _fwd_kernel
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/triton/compiler.py", line 1239, in compile
    so = _build(fn.__name__, src_path, tmpdir)
  File "/home/fernando/.local/share/virtualenvs/page-tagging-svc-6EE2Yu73/lib/python3.10/site-packages/triton/compiler.py", line 1169, in _build
    ret = subprocess.check_call(cc_cmd)
  File "/home/fernando/anaconda3/envs/python_310_cuda_113/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/gcc', '/tmp/tmpz4gpqr27/main.c', '-O3', '-I/usr/local/cuda/include', '-I/home/fernando/anaconda3/envs/python_310_cuda_113/include/python3.10', '-I/tmp/tmpz4gpqr27', '-shared', '-fPIC', '-lcuda', '-o', '/tmp/tmpz4gpqr27/_fwd_kernel.cpython-310-x86_64-linux-gnu.so', '-L/usr/lib', '-L/usr/lib32']' returned non-zero exit status 1.
python-BaseException
wandb: Waiting for W&B process to finish... (failed 1).
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /tmp/wandb/offline-run-20230608_230948-5b856dbf-a5e4-47b3-ad20-6570054489f7
wandb: Find logs at: /tmp/wandb/offline-run-20230608_230948-5b856dbf-a5e4-47b3-ad20-6570054489f7/logs

Process finished with exit code 1

Do you know what it could be, @tridao? Thank you very much in advance.

Idk what's wrong, I use the pytorch docker container and haven't seen this error.

To help other people that might have this problem:
I needed to add two other dependencies in conda: cuda-nvcc and cudatoolkit-dev. And I also needed to include an environment variable C_INCLUDE_PATH=$CONDA_HOME/envs/<env_name>/include.

error: 'scf.for' op expects region #0 to have 0 or 1 blocks for q torch.Size([1120, 280, 4, 32]) (1120, 280, 4, 32) True torch.bfloat16
for triton nightly triton_nightly-2.1.0.dev20230822000928-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

It's the Triton version. As mentioned at the beginning of the file:
Tested with triton==2.0.0.dev20221202.
Triton 2.0 has a new backend (MLIR) but seems like it doesn't yet work for head dimensions
other than 64:
https://github.com/openai/triton/blob/d376020f90002757eea3ea9475d4f7cfc2ec5ead/python/triton/ops/flash_attention.py#L207
We'll update this implementation with the new Triton backend once this is fixed.

Hey, @tridao. How do you install this version, though? It seems that PyTorch 2.0.0 and 2.0.1 both require triton 2.0.0. If I try to run pip install triton==2.0.0.dev20221202, I see that it starts downloading an older PyTorch version without CUDA (Downloading torch-1.13.1-cp310-cp310-manylinux1_x86_64.whl (887.5 MB)). If I install it with --no-deps, I get this error:
fatal error: cuda.h: No such file or directory
    2 | #include "cuda.h"
      |          ^~~~~~~~
compilation terminated.

So, I wonder how I should install this specific triton version.

try to use pip install --no-deps to install trition without update torch


Points unevenly distributed with tries < 5

I know a low number of tries may lead to uneven spacing but I'm trying to understand why it seems like the gaps are always in the lower right (x > 500, y > 500):
function redraw() {
  let points = new FastPoissonDiskSampling({
    shape: [1000, 1000],
    radius: 5,
    tries: 4,
  }).fill();

  let ctx = document.querySelector('canvas').getContext('2d');
  ctx.clearRect(0, 0, 1000, 1000);
  ctx.fillStyle = "hsl(0, 50%, 50%)";
  for (let p of points) {
    ctx.fillRect(p[0], p[1], 2, 2);
  }
}

redraw();

jsfiddle
I will use tries=5 for my project to avoid this, but was curious what causes it. :-) Example outputs:

Good catch, I'll investigate this when I have some time.

Finally come around to investigating this. It seems to be related to a floating point precision issue when calculating the distance between the points to validate the candidates. But I don't really understand why only the points in this specific quadrant seems to be affected. The workaround is really simple, just changing the "epsilon" value (here) from 2e-14 to 2e-12 fix the issue. I'll continue looking into it later today.

(broken english warning 😩)
Figured it out. I was almost sure this was related to the angles, because the test case with the 1000x1000 shape made it look like a quadrant thing (and because most of the algorithm implemented here is based on angles). This was off the mark. Testing the same thing in a 2000x2000 shape shows that the issue still happens when both coordinates are above 500 (most likely above 512).
This is just a matter of compounded float precision losses. The greater the absolute value of the float is, the lesser its precision.

2 + Number.EPSILON * 7 // => 2.0000000000000018
4 + Number.EPSILON * 7 => 4.000000000000002
16 + Number.EPSILON * 7 => 16

Adding the cosinus of an angle to 512 "drastically" reduces its precision.

000 + Math.cos(1)  // =><PHONE_NUMBER>058681398
512 + Math.cos(1) // =><PHONE_NUMBER>058682

This happens everytime the coordinates of a point is calculated based on the coordinates of the previous point and its angle/orientation from it (here). This may end up with a new point which is ever so slightly not far enough from the previous point.
The purpose of the existing "epsilon" variable was to mitigate the precision loss by slightly increasing the distance between the points.  But this "epsilon" value should be calculated based on the size of the space we work with, not naively hardcoded. Increasing the "epsilon" value like I mentionned in my previous message will just offset where the issue starts to manifest itself. Sure it may not happen after (512, 512) but after (2048, 2048) instead.
The fix will probably be something around those lines:
var floatPrecisionMitigation = Math.max(1, Math.log2(Math.max(this.width, this.height)) - 8) | 0;
var epsilon = 2e-14 * Math.pow(2, floatPrecisionMitigation);

with an additional recommendation in the documentation to avoid using extremely large shape (10000x10000)

[I haven't looked closely at the algorithm, but speculating here] With floating point, it kind of feels like if I multiply the radius by 100 and also multiply the shape size by 100, I should get the same results but multiplied by 100, except for that epsilon affecting things. What if epsilon were relative to the shape size? Would we then get size independence?

What if epsilon were relative to the shape size? Would we then get size independence?

Yes, that what I tried to prototype in the little blurb of code with "floatPrecisionMitigation". I'll still have to experiment to find the correct formula but the idea is to multiply epsilon by a power of two based on the log2 of the maximum coordinate of the shape. Say the shape is [1024, 256], we multiply epsilon by $2^{finetune(log2(1024))}$.

Ahh, you're right, I wasn't reading the code closely. Does epsilon have to be a round number, or would something like epsilon = Math.max(this.width, this.height) / 2.5e15 work as well?

I went with
var floatPrecisionMitigation = Math.max(1, Math.max(width, height) / 64 | 0);
var radiusEpsilon = 1e-14 * floatPrecisionMitigation;

(roughly similar to)
var floatPrecisionMitigation = Math.max(1, Math.log2(Math.max(width, height)) - 6);
var radiusEpsilon = 1e-14 * Math.pow(2, floatPrecisionMitigation);

Tested shapes' bounds from 512 to 512000000 without any issue. This change also allows the library to output resultsets with higher point density without making it any slower. Several false negatives due to FP precision issues are avoided.
While I was debugging the issue, I noticed some easy targets for performance optimization (leftovers of the arbitrary dimension handling from poisson-disk-sampling), so the library is about ~10% faster with a higher point density now.
-- PREVIOUS (1.0.2) --

[1024x1024 radius 8 tries 4]: 11.007ms for ~11158 points
[1024x1024 radius 8 tries 15]: 28.878ms for ~13371 points
[1024x1024 radius 8 tries 30]: 44.346ms for ~14054 points
[1024x1024 radius 8 tries 60]: 77.559ms for ~14566 points

[1024x1024 radius 4 tries 4]: 44.730ms for ~43765 points
[1024x1024 radius 4 tries 15]: 114.411ms for ~53324 points
[1024x1024 radius 4 tries 30]: 186.775ms for ~56019 points
[1024x1024 radius 4 tries 60]: 324.102ms for ~58025 points

[1024x1024 radius 2 tries 4]: 195.887ms for ~164328 points
[1024x1024 radius 2 tries 15]: 486.990ms for ~212774 points
[1024x1024 radius 2 tries 30]: 783.791ms for ~223538 points
[1024x1024 radius 2 tries 60]: 1316.438ms for ~231541 points

-- FIX EPSILON --

[1024x1024 radius 8 tries 4]: 12.438ms for ~12210 points
[1024x1024 radius 8 tries 15]: 28.727ms for ~13688 points
[1024x1024 radius 8 tries 30]: 42.580ms for ~14361 points
[1024x1024 radius 8 tries 60]: 76.977ms for ~14786 points

[1024x1024 radius 4 tries 4]: 44.944ms for ~48684 points
[1024x1024 radius 4 tries 15]: 106.644ms for ~54461 points
[1024x1024 radius 4 tries 30]: 178.169ms for ~57189 points
[1024x1024 radius 4 tries 60]: 306.427ms for ~58803 points

[1024x1024 radius 2 tries 4]: 203.596ms for ~194453 points
[1024x1024 radius 2 tries 15]: 452.430ms for ~217548 points
[1024x1024 radius 2 tries 30]: 741.757ms for ~228146 points
[1024x1024 radius 2 tries 60]: 1263.042ms for ~234724 points

-- FIX EPSILON + OPTIMIZATIONS (1.0.3) --

[1024x1024 radius 8 tries 4]: 9.938ms for ~12214 points
[1024x1024 radius 8 tries 15]: 25.298ms for ~13703 points
[1024x1024 radius 8 tries 30]: 37.872ms for ~14354 points
[1024x1024 radius 8 tries 60]: 68.991ms for ~14765 points

[1024x1024 radius 4 tries 4]: 37.334ms for ~48697 points
[1024x1024 radius 4 tries 15]: 91.493ms for ~54517 points
[1024x1024 radius 4 tries 30]: 159.394ms for ~57174 points
[1024x1024 radius 4 tries 60]: 274.662ms for ~58824 points

[1024x1024 radius 2 tries 4]: 172.247ms for ~194463 points
[1024x1024 radius 2 tries 15]: 397.841ms for ~217559 points
[1024x1024 radius 2 tries 30]: 659.043ms for ~228213 points
[1024x1024 radius 2 tries 60]: 1134.083ms for ~234761 points


Refactor script to remove unnecessary entries

Codecov Report

Merging #4 into master will decrease coverage by 0.55%.
The diff coverage is 11.11%.

@@            Coverage Diff            @@
##           master      #4      +/-   ##
=========================================
- Coverage    10.4%   9.84%   -0.56%
=========================================
  Files          11      11
  Lines         125     132       +7
  Branches       12      13       +1
=========================================
  Hits           13      13
- Misses        112     119       +7

Impacted Files
Coverage Δ

src/constants.ts
100% <ø> (ø)
:arrow_up:

src/runner.ts
0% <0%> (ø)
:arrow_up:

src/utils.ts
70% <50%> (+6.36%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 937ced8...456a8ff. Read the comment docs.


refactor: improve matchAll string helper

I think I did a bad job on the first PR about this helper, this aims to improve the method a bit, and I simplify the no match result.

Thanks!


Remove output related to fixed issue (#1244) #2428

I noticed some descriptions in my documentation like @Size: Min - Max (until #1244 gets fixed). Looks like that was fixed back in 2.5, and  I see the correct "minimum" and "maximum" values for all of them, so I opened #2428 and got rid of the generated message.

Thanks for the PR @haelduksf !!


d3.transpose() applied on loud matrices. call stack exception thrown.

First, thanks to all for your great and usefull job.
We noticed an exception from d3.transpose(m) when m is a loud matrix.
bug_transpose_test.zip
Tested on chrome 48.0.2564.97 with d3 versions : 3.5.14 and 3.4.9.
Test code :
	var data1 = [1,2,3,4,5]; 	var data = []; 	for(var i = 0; i<130000; i++) 	{ 		data.push(data1); 	} 	var transposed = d3.transpose(data);

This has already been fixed in d3-array, but I can back-port the fix.


Build system for student feedback on questions

Build system for allowing students to submit feedback for questions (question is too hard, not clear, etc.)
Source: eyurko

Closing because not in near-term plans.


Add ::first-letter CSS pseudo-element

This also covers the :first-letter pseudo-class.
Fixes #990.
Testcase: http://jsfiddle.net/cvrebert/gstmvgcm/

Great work, thanks!


Compute pixel-wise bitrate allocation in latent space

Hi,
Is there a way to compute per-pixel bitrate info in RGB space of the latent variable y or z during evaluation?
Regards

The likelihood map $l = -\log_2 p_{\hat{y}}(\hat{y})$ of dimensions $M_y \times \frac{H}{2^4} \times \frac{W}{2^4}$ can be calculated exactly, so you can plot each the bit cost of each latent element. Since it's 3D, you can flatten it to 2D using a sum along the channel dimension.
In bmshj2018-factorized, each $y$ latent pixel affects reconstructed $\hat{x}$ pixels within a window of 9x9. So, one way to map the $l$ back to the image domain is just to upsample and interpolate.
I guess a more advanced method for estimating rate costs of encoding each pixel would involve training some sort of rate estimation model or maybe something using GradCam.

Thank you for your answer.  I try to do it in the following way, can you maybe please point out any mistake?
model.eval() #model = hyperprior
model.update()

y = model.g_a(x)
y_hat, likelihood_map = model.entropy_bottleneck(latent)
likelihood_map = likelihood_map [0].detach().cpu().numpy()
pixel_bitrates = likelihood_map.sum(dim=1) #channel-wise sum

Now should i just upsample  pixel_bitrates to x using the interpolation techniques like billinear somehow?

import matplotlib.pyplot as plt
import torch
import torch.nn.functional as F
from compressai.ops import compute_padding
from compressai.zoo import bmshj2018_factorized
from PIL import Image
from torchvision import transforms

device = "cuda"
model = bmshj2018_factorized(quality=2, pretrained=True).eval().to(device)
img = Image.open("/data/datasets/kodak/test/kodim01.png").convert("RGB")
x = transforms.ToTensor()(img).unsqueeze(0).to(device)

y = model.g_a(x)
y_hat, y_likelihoods = model.entropy_bottleneck(y)
y_nll = -y_likelihoods.log2()
x_nll = F.interpolate(
    y_nll.sum(dim=1, keepdim=True),
    scale_factor=x.shape[-1] / y_nll.shape[-1],
    mode="bilinear",
    align_corners=False,
)

fig, ax = plt.subplots()
ax.imshow(x_nll[0, 0].detach().cpu().numpy())
fig.savefig("x_nll.png")


For showing Match Winner, show Team Name if populated, otherwise, show player names

Resolved in version 1.056


It does not work in case of warnings

I'm using php from homebrew on OS X and it seem that php tries to load module several times (I found only one load of that module in /usr/local/etc/php/7.1/conf.d/ext-pcntl.ini after I removed that file error is gone and module still exist, but it seems this error comes by default after installation by brew)
PHP Warning:  Module 'pcntl' already loaded in Unknown on line 0
if this case vim just throws error for invalid argument for json_decode.
for debug I tried to add
if a:response =~ "WARNING:"
return
endif
if a:response =~ "Warning:"
return
endif
to the beggining of s:OnCall and it works, but really not sure that it's possible to validate it, especially msgpack (neovim version did not work for me too)

phpcd.vim read from stdout of PHPCD.php process and parse the result as json/msgpack. If you php output some warning message, phpcd.vim will be confused. So the PHPCD.php should only output json/msgpack stream.

Yes, I got this, I just was saying that it may be difficult to debug and maybe you can validate it somehow? At least to tell user that something is wrong with output?

For NeoVim, phpcd use the builtin msgpack-rpc api. NeoVim will read from phpcd's output and unpack according msgpack. If it read invalid msgpack stream (containing the error output), an error will be trigged. There no change to check these error.
For Vim, phpcd use the job channel and json api to emulate rpc call. We can do some check before call the json_decode function. But these approach is vim only.
But how to debug this problem? We need do the following,

Run a normal php file to ensure that no error/warning message has been output.
This step is essential, because we can prevent php output error message during bootstrap.
set the display_errors to stderr by ini_set('display_errors', 'stderr'), so any further will be output to the stderr, and vim/nvim will display these error.


How to contribute

Hello, I was planning to build a CSS Framework for material design and I found Materialize. I would love to contribute to this project by adding new components and fixing issues posted by others. Please let me know how can I do this. Here's my linkedin profile: https://www.linkedin.com/in/achintverma.

Hey @achintverma, great to hear you want to help! There is a file CONTRIBUTING.md which relates to this.


How to add all columns into the Transaction log for querying, not 'select *'

System information:

Operating system (distribution) and version
Fedora 32

DBeaver version
7.2.2

Connection specification:

Database name and version
MySQL 8

Describe your question:
Is there any way to add all columns into the Transaction log for querying, not just 'select * from a_table'.
I see that updating and inserting sqls has columns.
I like to copy SQLs in Transaction log so that it can save me some time to write sqls by hand.

It isn't possible. The transaction log shows a query that was executed.
If you need to see columns, you could use autosuggestion in SQL editor. If you place the cursor after "*" symbol in query and press Ctrl+Space - * will be replaced by columns name.

@uslss
Thank you. This is quite useful. It will improve my workflow.


Standardise get custom resource at start of reconcile

Hello,
In every reconcile loop that I've seen, we start to GET the resource the same way every time:

I cannot imagine a case scenario where we would do something else than this.
Would it be interesting to make a function to do that or to standardise it in other ways ? Since every reconcile need to do it.
Have a good day,

Hey,
Operator-SDK leverages kubebuilder's create api command to generate controllers, so this suggestion can probably be made upstream at kubebuilder and discussed there as well.

Hey @MathieuCesbron,
Just wanted to check in on the status of this issue. Feel free to ping me if you have any questions.

@MathieuCesbron closing this. Feel free to re-open it if you want to discuss this further.


Add absl/log/vlog_is_on.h #includes

Abseil 20240722.0 has made it mandatory to explicitly #include vlog_is_on.h in order to use VLOG_IS_ON ^1. Add those #includes so that the code builds with the latest Abseil.

not sure about this, I would like to keep the code inline with the internal one.
Need to check if we get vlog_is_on from https://github.com/google/or-tools/blob/stable/ortools/base/logging.h
i.e. add it there

Fixed by:

https://github.com/google/or-tools/commit/9509cabec12d930d752add2471fa20617c1948ff
https://github.com/google/or-tools/commit/05bfe988e498063942de2a253692828df57d4b5a

Thanks @Mizux!


Getting error for twitter

After getting ng2-auth-ui-example working for facebook, I tried it for twitter and getting following error. Any help is appreciated! Thanks
loginWithTwitter() {
this.auth.authenticate('twitter')
.subscribe({
error: (err: any) => alert(err.message),
complete: () => this.router.navigateByUrl('main')
})
}
<button type="button" class="btn btn-default" (click)="loginWithTwitter()">Login with Twitter
POST /auth/twitter - - ms - -
[0] (node:28176) UnhandledPromiseRejectionWarning: TypeError: Cannot destructure property client_id of 'undefined' or 'null'.
[0]     at /Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/src/auth.ts:203:50
[0]     at Generator.next ()
[0]     at /Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/src/auth.ts:7:71
[0]     at new Promise ()
[0]     at __awaiter (/Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/src/auth.ts:3:12)
[0]     at twitter (/Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/src/auth.ts:214:12)
[0]     at Layer.handle [as handle_request] (/Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/node_modules/express/lib/router/layer.js:95:5)
[0]     at next (/Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/node_modules/express/lib/router/route.js:137:13)
[0]     at Route.dispatch (/Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/node_modules/express/lib/router/route.js:112:3)
[0]     at Layer.handle [as handle_request] (/Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/node_modules/express/lib/router/layer.js:95:5)
[0]     at /Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/node_modules/express/lib/router/index.js:281:22
[0]     at Function.process_params (/Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/node_modules/express/lib/router/index.js:335:12)
[0]     at next (/Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/node_modules/express/lib/router/index.js:275:10)
[0]     at middleware (/Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/node_modules/express-jwt/lib/index.js:78:16)
[0]     at Layer.handle [as handle_request] (/Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/node_modules/express/lib/router/layer.js:95:5)
[0]     at trim_prefix (/Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/node_modules/express/lib/router/index.js:317:13)
[0]     at /Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/node_modules/express/lib/router/index.js:284:7
[0]     at Function.process_params (/Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/node_modules/express/lib/router/index.js:335:12)
[0]     at next (/Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/node_modules/express/lib/router/index.js:275:10)
[0]     at Function.handle (/Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/node_modules/express/lib/router/index.js:174:3)
[0]     at router (/Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/node_modules/express/lib/router/index.js:47:12)
[0]     at Layer.handle [as handle_request] (/Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/node_modules/express/lib/router/layer.js:95:5)
[0] (node:28176) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 3)
[0] POST /auth/twitter - - ms - -
[0] (node:28176) UnhandledPromiseRejectionWarning: TypeError: Cannot destructure property client_id of 'undefined' or 'null'.
[0]     at /Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/src/auth.ts:203:50
[0]     at Generator.next ()
[0]     at /Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/src/auth.ts:7:71
[0]     at new Promise ()
[0]     at __awaiter (/Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/src/auth.ts:3:12)
[0]     at twitter (/Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/src/auth.ts:214:12)
[0]     at Layer.handle [as handle_request] (/Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/node_modules/express/lib/router/layer.js:95:5)
[0]     at next (/Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/node_modules/express/lib/router/route.js:137:13)
[0]     at Route.dispatch (/Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/node_modules/express/lib/router/route.js:112:3)
[0]     at Layer.handle [as handle_request] (/Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/node_modules/express/lib/router/layer.js:95:5)
[0]     at /Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/node_modules/express/lib/router/index.js:281:22
[0]     at Function.process_params (/Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/node_modules/express/lib/router/index.js:335:12)
[0]     at next (/Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/node_modules/express/lib/router/index.js:275:10)
[0]     at middleware (/Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/node_modules/express-jwt/lib/index.js:78:16)
[0]     at Layer.handle [as handle_request] (/Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/node_modules/express/lib/router/layer.js:95:5)
[0]     at trim_prefix (/Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/node_modules/express/lib/router/index.js:317:13)
[0]     at /Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/node_modules/express/lib/router/index.js:284:7
[0]     at Function.process_params (/Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/node_modules/express/lib/router/index.js:335:12)
[0]     at next (/Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/node_modules/express/lib/router/index.js:275:10)
[0]     at Function.handle (/Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/node_modules/express/lib/router/index.js:174:3)
[0]     at router (/Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/node_modules/express/lib/router/index.js:47:12)
[0]     at Layer.handle [as handle_request] (/Users/mohammadchowdhury/Downloads/SocialChamp/ngx-admin-master/server/node_modules/express/lib/router/layer.js:95:5)
[0] (node:28176) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 4)

Twitter flow is a bit different since it uses oauth 1.0a and not oauth 2. the server side implementation is a bit different, what did you receive as a body on your endpoint?

This is what I receive. For Client id I receive my client id . Looks I am not receiving anything for
authorizationData.  The callback URL for the app is set to http://localhost:3000. Thanks
{ name: 'twitter',
[0]   url: '/auth/twitter',
[0]   authorizationEndpoint: 'https://api.twitter.com/oauth/authenticate',
[0]   oauthType: '1.0',
[0]   popupOptions: { width: 495, height: 645 },
[0]   clientId: '******************',
[0]   redirectUri: 'http://localhost:4200' }


🐛Trying to get RC522 to work Multi

Hi,
I am trying to get the RC522 reader to work together with a USB reader. Accordingly I am using the Reader.py.experimental.Multi and RegisterDevice.py.Multi scripts. Each of the readers works fine on its own (using Reader.experimental) and the USB-Reader also works in the .experimental.Multi setup. But the RC522 doesn't.
The first minor problem is that RegisterDevice.py.Multi doesn't work but that is some weird python dependency problem that I think has nothing to do with this. Anyway, I think I figured out what the correct deviceName.txt should look like (for reasons not clear to me the .Multis use a different format from the non-.Multis) namely:
    MFRC522;
    Sycreader USB Reader;usb-3f980000.usb-1.1.2/input0

But then for some reason the RC522 is ignored. Has someone gotten this to work and has a suggestion?
Best,
Stefan

Quick update (referring to Reader.py.experimental.Multi throughout): I just checked that everything high-level works as expected: the Reader registers the USB reader as well as the RC522 and polls each of their read functions.
I also checked that replacing Reader by Mfrc522Reader in daemon_rfid_reader.py works without any further changes (except removing the mysterious extra reader. in [https://github.com/MiczFlor/RPi-Jukebox-RFID/blob/b3e91ae567cd3d089708f5b9d1a5f634bb41e8db/scripts/daemon_rfid_reader.py#L95]) what is that about?!). So is it possible that there is some synchronous stuff happening in Mfrc522Reader.readCard() that doesn't go well with multiprocessing?

I think one might have to rewrite Mfrc522Reader.readCard() to not use RFID.wait_for_tag(). Then the reader could be stopped.

I get this error when I select my RC522 reader during the installation
phoniebox git clone -q git://github.com/ondtryaso/pi-rc522.git /tmp/pip-install-5yjpx8ai/pi-rc522 failed with error code 128 in None

Ok, manually installing RC522 following exactly these instructions solved the issue for me:
https://github.com/pimylifeup/MFRC522-python

Fixed by #1791, but not merged to master yet.


[Enhancement] Refresh = 0  ==> Send Msg update only on event TRUE/FALSE

Hi,
I do like your node and just updating my flows accordingly.  At moment you are updating every 60sec (or as specified  by REFRESH) with a MSG payload of TRUE/FALSE.
Is it possible to have the state update only when the event occurs (not every 60 secs)?
Why do I need this (use case):

I have a Poolpump scheduled for 7am-8am and 7pm to 8pm
I can activate the Pool Pump by Alexa, however it get switched-off after max 60sec when your node is sending an update

Thanks for considering

Hi,
yes there a two common possibilities:

Simply add an RBE node (with default, block unless value changes, setting) to the output.

Switch to the EVENT mode and schedule single events instead of a time frame, they will only fire at the specified time.

Please let me know if this helps!

Hi,
yes there a two common possibilities:

Simply add an RBE node (with default, block unless value changes, setting) to the output.

Switch to the EVENT mode and schedule single events instead of a time frame, they will only fire at the specified time.

Please let me know if this helps!

yes, I have used the RBE node already in my flow but for some reason I get the occasional update a few ms after the initial event. Therefore I'm suppressing any update for 10sec after each new event

==> These are 2 workarounds for this refresh=0  ;-) ... but I will survive
I tried your Option 2 as well but i have too many On/Off scenarios ;-) to use this option
great node ... keep up the good work

yes, I have used the RBE node already in my flow but for some reason I get the occasional update a few ms after the initial event. Therefore I'm suppressing any update for 10sec after each new event

==> These are 2 workarounds for this refresh=0  ;-) ... but I will survive
I tried your Option 2 as well but i have too many On/Off scenarios ;-) to use this option
great node ... keep up the good work

is a workaround that is correct... I haven't implemented that into the node because it is just so easy to add the RBE node.
is perfekt solution to this problem and one of the reasons why the event mode even exists. There is no limit for On/Off scenarios, you can add as many as you want.

I did some testing today and I am not sure anymore if the RBE might actually behaves weird if no topic is set - would be nice if we could figure that out together. Can you enable the topic output within the ui schedulers setting and see if this changes anything to the better?

is a workaround that is correct... I haven't implemented that into the node because it is just so easy to add the RBE node.
is perfekt solution to this problem and one of the reasons why the event mode even exists. There is no limit for On/Off scenarios, you can add as many as you want.

I did some testing today and I am not sure anymore if the RBE might actually behaves weird if no topic is set - would be nice if we could figure that out together. Can you enable the topic output within the ui schedulers setting and see if this changes anything to the better?

Yes, will do... never experienced anything wrong with RBE so far and I'm using it a lot

Yes, will do... never experienced anything wrong with RBE so far and I'm using it a lot

1.9.0 now has the ability to block output unless a value changes! I'd appreciate your feedback!

works great !!


Azure AD B2C: Azure AD IdP does not work with custom domain

Hi!
I'm using custom policy to add Azure AD as an identity provider for my Azure Ad B2C instance.
To make it work i had to add the following redirect uri for my app registration (otherwise i saw an error when redirecting to b2c from azure ad):

https://login.microsoftonline.com/te/coredwexposureb2c.onmicrosoft.com/oauth2/authresp

It works fine on my localhost, with the following redirect uris configured:

https://login.microsoftonline.com/te/coredwexposureb2c.onmicrosoft.com/oauth2/authresp
http://localhost:55838/signin-oidc

however when i publish it to my web app, i need to add another redirect uri for my application:

https://coredwexposure.onmicrosoft.com

The portal does not allow me to save 3 redirect uris for the app registration - it says "You may not use more than 1 external domain(s)"

https://login.microsoftonline.com/te/coredwexposureb2c.onmicrosoft.com/oauth2/authresp
http://localhost:55838/signin-oidc
https://coredwexposure.onmicrosoft.com/signin-oidc

Please advise how can i fix that.

I'm unclear why the Azure AD app needs to redirect back to your application as well (both the localhost and the coredwexposure). In theory, your application should only talk to B2C and only B2C should talk to that non-B2C Azure AD app, so you should only need the login.microsoftonline.com... URL.

Hi! Thanks for your response.
I tried to remove http://localhost:55838/signin-oidc from the list of reply uris in b2c and my application stopped working - it is giving 404 when trying to redirect from b2c to my app. When i am adding it back it works fine.

My application is registered in B2C.
Azure AD is registered as an IdP.
When i log into my app, i'm redirected to B2C page, then to AzureAD page, then back to B2C (https://login.microsoftonline.com/te/coredwexposureb2c.onmicrosoft.com/oauth2/authresp) where the claims are transformed and then to my app.

Thus it looks like i need both redirect uris, however cannot do that on any host besides localhost due to limit of 2 custom namespaces.

You need two separate app registrations:

One in Azure AD B2C for your app  - this one has your apps' redirect URLs:

http://localhost:55838/signin-oidc
https://coredwexposure.onmicrosoft.com/signin-oidc

One in Azure AD for Azure AD B2C - this one has B2C as the redirect URL:

https://login.microsoftonline.com/te/coredwexposureb2c.onmicrosoft.com/oauth2/authresp

This article has more info.

I believe my scenario is a bit different.
I'm trying to link the common tenant to support the multi-tenant scenario. Thus i cannot use any app registrations inside Azure AD, since i do not know the directories in advance and can support login from any of them.
Here is my policy provider for AzureAD:

MultiTenant
Login using Azure AD

Organizational or Microsoft Account
Organizational or Microsoft Account

JWT

true
https://login.microsoftonline.com/
https://login.windows.net/common/oauth2/v2.0/authorize
a4802af6-d0a1-4551-810e-2dee5d2edcbd

a4802af6-d0a1-4551-810e-2dee5d2edcbd
AuthorizationHeader
openid profile
POST
id_token
false

The guidance still stands though, create both apps in your tenant and make the Azure AD one a "multi-tenant" app. (See these instructions to configure it as such).

One in Azure AD B2C for your app - this one has your apps' redirect URLs:

http://localhost:55838/signin-oidc
https://coredwexposure.onmicrosoft.com/signin-oidc

One in Azure AD for Azure AD B2C - this one has B2C as the redirect URL and is configured as multi-tenant:

https://login.microsoftonline.com/te/coredwexposureb2c.onmicrosoft.com/oauth2/authresp

Having 2 applications: one in Azure ADB2C and one in Azure Ad will give me 2 application ids.
How do i link them together? I will use the id from B2C  in my application's config file.
Where should i put the id from Azure AD?

The application ID that is registered with the Azure AD tenant, i.e. the identifier of the application that represents the Azure AD B2C tenant, must be set in the "client_id" metadata item of the "MultiTenantProfile" technical profile:
<Item Key="client_id"><!-- Enter the application ID --></Item>

This application ID is how the Azure AD B2C tenant identifies itself with the Azure AD tenant.

Should it also be set to IdTokenAudience?
Will i need to change the client secret to be the secret of AzureAD application?

Yes to both.

Thanks for your assistance.
Once i introduced AzureAd app registration i managed to overcome the domain name restriction.
However it showed me the following error then:
AADSTS90130: Application '529438c4-ae3d-4101-a1ae-9812e3bab261' (CoreDWExposure ODATA API) is not supported over the /common or /consumers endpoints. Please use the /organizations or tenant-specific endpoint.
I had to change the authorization endpoint from /common to /organizations for it to work. Thus my ClaimsProvider looks like below:
<ClaimsProvider>
      <Domain>MultiTenant</Domain>
      <DisplayName>Login using Azure AD</DisplayName>
      <TechnicalProfiles>
        <TechnicalProfile Id="MultiTenantProfile">
          <DisplayName>Organizational or Microsoft Account</DisplayName>
          <Description>Organizational or Microsoft Account</Description>
          <Protocol Name="OpenIdConnect"/>
          <OutputTokenFormat>JWT</OutputTokenFormat>
          <Metadata>
            <Item Key="DiscoverMetadataByTokenIssuer">true</Item>
            <Item Key="ValidTokenIssuerPrefixes">https://login.microsoftonline.com/</Item>
            <Item Key="authorization_endpoint">https://login.windows.net/organizations/oauth2/v2.0/authorize</Item>
            <Item Key="client_id">529438c4-ae3d-4101-a1ae-9812e3bab261</Item>
            <!-- AppID from B2C tenant's WebApp Application -->
            <Item Key="IdTokenAudience">529438c4-ae3d-4101-a1ae-9812e3bab261</Item>
            <Item Key="BearerTokenTransmissionMethod">AuthorizationHeader</Item>
            <Item Key="scope">openid profile</Item>
            <Item Key="HttpBinding">POST</Item>
            <Item Key="response_types">id_token</Item>
            <Item Key="UsePolicyInRedirectUri">false</Item>
          </Metadata>
          <CryptographicKeys>
            <Key Id="client_secret" StorageReferenceId="B2C_1A_MultiTenantSigningKey2"/>
            <!-- AppSecret from B2C tenant's WebApp Application, saved to policy keys in Azure -->
          </CryptographicKeys>
          <OutputClaims>
            <OutputClaim ClaimTypeReferenceId="socialIdpUserId" PartnerClaimType="oid"/>
            <OutputClaim ClaimTypeReferenceId="tenantId" PartnerClaimType="tid"/>
            <OutputClaim ClaimTypeReferenceId="givenName" PartnerClaimType="given_name" />
            <OutputClaim ClaimTypeReferenceId="surName" PartnerClaimType="family_name" />
            <OutputClaim ClaimTypeReferenceId="displayName" PartnerClaimType="name" />
            <OutputClaim ClaimTypeReferenceId="authenticationSource" DefaultValue="AzureAD" />
            <OutputClaim ClaimTypeReferenceId="identityProvider" DefaultValue="AzureADMultiTenant" />
            <OutputClaim ClaimTypeReferenceId="email" PartnerClaimType="preferred_username" />
          </OutputClaims>
          <OutputClaimsTransformations>
            <OutputClaimsTransformation ReferenceId="CreateRandomUPNUserName"/>
            <OutputClaimsTransformation ReferenceId="CreateUserPrincipalName"/>
            <OutputClaimsTransformation ReferenceId="CreateAlternativeSecurityId"/>
            <OutputClaimsTransformation ReferenceId="CreateSubjectClaimFromAlternativeSecurityId"/>
          </OutputClaimsTransformations>
          <UseTechnicalProfileForSessionManagement ReferenceId="SM-Noop"/>
        </TechnicalProfile>
      </TechnicalProfiles>
    </ClaimsProvider>

"Application '529438c4-ae3d-4101-a1ae-9812e3bab261' (CoreDWExposure ODATA API)" indicates you have entered the client ID for the "CoreDWExposure ODATA API" app, not that for Azure AD B2C, that is registered with the Azure AD tenant.
The "/common" endpoint is supported.

Ok. I'm a bit lost then.
Could you please advise which application identifier (Azure AD/B2C) and secret for which application (Azure AD/B2C) need to go into these fields:

...

As per @gsacavdm's earlier comment, you must create an app registration in Azure AD for use by Azure AD B2C, as described by https://docs.microsoft.com/en-us/azure/active-directory-b2c/active-directory-b2c-setup-aad-custom#create-an-azure-ad-app.
The reply URL for this app registration is https://login.microsoftonline.com/te/coredwexposureb2c.onmicrosoft.com/oauth2/authresp.
It is the client ID and secret for this app registration that must be entered in the "MultiTenantProfile" technical profile.
This technical profile uses this client ID and secret to generate the authorization and token requests to Azure AD's common endpoint.

Could we use my specific case as an example here:

My B2C App registration:
CoreDW Exposure API
a4802af6-d0a1-4551-810e-2dee5d2edcbd

My Azure AD App registration:
CoreDWExposure ODATA API
529438c4-ae3d-4101-a1ae-9812e3bab261
The key for this application is stored in B2C_1A_MultiTenantSigningKey2 key of B2C.

My ClaimsProvider element from custom policy:
<ClaimsProvider>
      <Domain>MultiTenant</Domain>
      <DisplayName>Login using Azure AD</DisplayName>
      <TechnicalProfiles>
        <TechnicalProfile Id="MultiTenantProfile">
          <DisplayName>Organizational or Microsoft Account</DisplayName>
          <Description>Organizational or Microsoft Account</Description>
          <Protocol Name="OpenIdConnect"/>
          <OutputTokenFormat>JWT</OutputTokenFormat>
          <Metadata>
            <Item Key="DiscoverMetadataByTokenIssuer">true</Item>
            <Item Key="ValidTokenIssuerPrefixes">https://login.microsoftonline.com/</Item>
            <Item Key="authorization_endpoint">https://login.windows.net/common/oauth2/v2.0/authorize</Item>
            <Item Key="client_id">529438c4-ae3d-4101-a1ae-9812e3bab261</Item>
            <Item Key="IdTokenAudience">529438c4-ae3d-4101-a1ae-9812e3bab261</Item>
            <Item Key="BearerTokenTransmissionMethod">AuthorizationHeader</Item>
            <Item Key="scope">openid profile</Item>
            <Item Key="HttpBinding">POST</Item>
            <Item Key="response_types">id_token</Item>
            <Item Key="UsePolicyInRedirectUri">false</Item>
          </Metadata>
          <CryptographicKeys>
            <Key Id="client_secret" StorageReferenceId="B2C_1A_MultiTenantSigningKey2"/>
          </CryptographicKeys>
          <OutputClaims>
            <OutputClaim ClaimTypeReferenceId="socialIdpUserId" PartnerClaimType="oid"/>
            <OutputClaim ClaimTypeReferenceId="tenantId" PartnerClaimType="tid"/>
            <OutputClaim ClaimTypeReferenceId="givenName" PartnerClaimType="given_name" />
            <OutputClaim ClaimTypeReferenceId="surName" PartnerClaimType="family_name" />
            <OutputClaim ClaimTypeReferenceId="displayName" PartnerClaimType="name" />
            <OutputClaim ClaimTypeReferenceId="authenticationSource" DefaultValue="AzureAD" />
            <OutputClaim ClaimTypeReferenceId="identityProvider" DefaultValue="AzureADMultiTenant" />
            <OutputClaim ClaimTypeReferenceId="email" PartnerClaimType="preferred_username" />
          </OutputClaims>
          <OutputClaimsTransformations>
            <OutputClaimsTransformation ReferenceId="CreateRandomUPNUserName"/>
            <OutputClaimsTransformation ReferenceId="CreateUserPrincipalName"/>
            <OutputClaimsTransformation ReferenceId="CreateAlternativeSecurityId"/>
            <OutputClaimsTransformation ReferenceId="CreateSubjectClaimFromAlternativeSecurityId"/>
          </OutputClaimsTransformations>
          <UseTechnicalProfileForSessionManagement ReferenceId="SM-Noop"/>
        </TechnicalProfile>
      </TechnicalProfiles>
    </ClaimsProvider>

Could you please advise which specific changes are necessary?

<Item Key="DiscoverMetadataByTokenIssuer">true</Item>
<Item Key="ValidTokenIssuerPrefixes">https://sts.windows.net/</Item>
<Item Key="authorization_endpoint">https://login.microsoftonline.com/common/oauth2/authorize</Item>
<Item Key="client_id">df5b2515-a8d2-4d91-ab4f-eac6e1e416c2</Item>
<Item Key="BearerTokenTransmissionMethod">AuthorizationHeader</Item>
<Item Key="scope">openid</Item>
<Item Key="UsePolicyInRedirectUri">false</Item>
<Item Key="HttpBinding">POST</Item>
<Item Key="response_types">id_token</Item>
<Item Key="IdTokenAudience">df5b2515-a8d2-4d91-ab4f-eac6e1e416c2</Item>

If you want to use the common endpoint, here is the snippet that you need to modify to get working.
I highly recommend that you read the article https://docs.microsoft.com/en-us/azure/active-directory-b2c/active-directory-b2c-setup-aad-custom from top to bottom and get Azure AD single tenant authentication working, then use the above snippit to enable multi-tenant authentication. Note that you will only be able to receive the following claims from users via AAD
https://docs.microsoft.com/en-us/azure/active-directory/develop/active-directory-token-and-claims
If you want more claims, you'll have to enable another orchestration step to retrieve claims via the Graph API. Here's a general idea on how to do that with Logic Apps. https://monteledwards.com/2017/10/18/a-complete-integration-azure-ad-b2c-azure-ad-graph-api-logic-apps/

Guys,
I really appreciate your help, but let's stick to my particular example. Jumping back and forth does not bring more understanding.
I have done exactly as you said: i have configured the single tenant scenario according to https://docs.microsoft.com/en-us/azure/active-directory-b2c/active-directory-b2c-setup-aad-custom.
In your latest example you are using some applucation id df5b2515-a8d2-4d91-ab4f-eac6e1e416c2
In my case i have 2 app registrations:
My B2C App registration:
CoreDW Exposure API
a4802af6-d0a1-4551-810e-2dee5d2edcbd
My Azure AD App registration:
CoreDWExposure ODATA API
529438c4-ae3d-4101-a1ae-9812e3bab261
Which identifier should be used in my case?
Also which application should be used to create a key in CryptographicKeys?

Understand that people are trying to help, no need to discourage others from offering help, tips or insights, even if they are not doing the work for you.
My B2C app registration is used in your app's code.
My Azure AD app registration is used in your MultiTenantProfile, including the CryptographicKeys.

Thanks for the update. I did not mean to be rude or offensive.
I did use the identifier from Azure AD app registration in the MultiTenantProfile, but got the following error with the common tenant:
AADSTS90130: Application '529438c4-ae3d-4101-a1ae-9812e3bab261' (CoreDWExposure ODATA API) is not supported over the /common or /consumers endpoints. Please use the /organizations or tenant-specific endpoint.

You need to configure your application as a multi-tenant application

Hi,
We also had this problem but finally we were able to call custom idp from B2C.
We used IdentityServer4 as custom IDP.
Step 1:
Create a Client inside Custom IDP . Say Client ID=>B2C_Client
Add OpenId Scope
Add reply url => https://login.microsoftonline.com/te/coredwexposureb2c.onmicrosoft.com/oauth2/authresp
Publish to APP service (example =>www.customidp.azure.net)
Step2:
Have to create Identity server from B2C to configure your custom IDP www.customidp.azure.net
Select Identity server => OPenIdConnect(preview)
Configure your custome idp well-know/openid-configuration url
Configure your app for implicit flow dont add secret key . select id_token as response.
From claim mapping username text give sub. (very important).
Step4:
Create Application in B2C.
Provide reply url (very important step)
http://localhost:55838/signin-oidc
https://coredwexposure.onmicrosoft.com/signin-oidc
Step5:
Create your signin policy and check this custom idp inside your policy.
**Step6:
**
Refer and download quick start from this url to configure your client application.
https://docs.microsoft.com/en-us/azure/active-directory-b2c/active-directory-b2c-quickstarts-web-app
This should work.

@sewalsh I dont have repository.Ii have given detailed steps in my previous comment. Just follow the comments and let me know.


Adding Contributors Section to the Readme.md File

Adding a contributors section to the README.md file provides recognition, fosters community, and enhances project transparency and accountability.
Example:-

@erikspears Kindly assign this Issue to me! Thank You !!

Done!  @mohitd404 :)

@erikspears I have made the changes Kindly review it ! Thank you!

Nice idea!  One question before we merge the PR:  does this solution create a dynamic list (image, really) of contributors that will change as new folks have code merged, etc.?  Or is it a static output that requires some manual intervention to update?
Thanks,
--Erik

@mohitd404 , Nice idea! One question before we merge the PR: does this solution create a dynamic list (image, really) of contributors that will change as new folks have code merged, etc.? Or is it a static output that requires some manual intervention to update?
Thanks, --Erik
It creates a Static image of the contributors it does not update as new contributions are adds to the repo! However, I can also create a Dynamic Contributors section which will update with the new contributions, but it will need to maintain ".all-contributorsrc" file for every new contributor to the maintainer manually it is not possible to make it dynamic.

@mohitd404 ,
OK, thanks for that.  I think a static image doesn't bode well for future maintenance, to be honest 😆
Would you mind PR-ing the more dynamic approach, with the understand that some manual upkeep is still required?  You could either close your current PR and open a fresh one, or update the one you have.  Whichever makes more sense to you is just fine.
Thanks,
--Erik

@mohitd404 ,
OK, thanks for that. I think a static image doesn't bode well for future maintenance, to be honest 😆 Would you mind PR-ing the more dynamic approach, with the understand that some manual upkeep is still required? You could either close your current PR and open a fresh one, or update the one you have. Whichever makes more sense to you is just fine.
Thanks, --Erik

Okay @erikspears I'll implement it in dynamic approach !


Change default dimensions number in OpenAIEmbeddings class

Thank you for contributing to LangChain!

[ ] PR title: "partners: default dimension number in OpenAIEmbeddings class"

Description: Change default number of dims in OpenAIEmbeddings class
Issue: #25879
Dependencies: none
Twitter handle: @ValentinKovalev

[x] Lint and test: Run make format, make lint and make test from the root of the package(s) you've modified. See contribution guidelines for more: https://python.langchain.com/docs/contributing/

not desired as the default! This is for the embedding models that support this parameter, so defaulting to none is good (which defaults to max for those models)
If you need this in your code, you can pass dimensions=x manually :)

Could you please clarify the reasoning behind using the default model without specifying the dimensions parameter? I understand that hardcoding dimensions (like 1536 for the text-embedding-ada-002 model) might not be ideal. However, I'm concerned that leaving embed.dimensions as None could lead to confusion or errors. It might be more intuitive for the default model to automatically set the correct dimensions.

if you have a test case that's failing let me know. Using null/none just gives the max dimensions provided by the model https://platform.openai.com/docs/api-reference/embeddings/create#embeddings-create-dimensions


node.__rf is undefined

Hai,
I am trying to implement layout/Re-Distribute nodes in a graph as mentioned in https://github.com/wbkd/react-flow/issues/5#issuecomment-643805398  but accessing __rf is being undefined.
here is codesandbox to reproduce.
Thanks In Advance.

hi, you are using elements from your local state there. the __rf is only available when using the nodes from useStoreState (https://reactflow.dev/docs/api/internal-state-actions/).

@chrtze  thanks for a quick response is there no other way for me to access __rf on my current codebase ?
if any could you please explain how
Thanks in advance

the two options are using the internal state or the newly introduced toObject function which has the positions.


About window is still a bit too big in mobile

But there should an opportunity to shorten the text (with the help of @ralfstrandell's "Short" mode in getText).

Good now I think.


[test-framework] Integrating test apps into framework

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: nmittler
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: costinm
If they are not already assigned, you can assign the PR to them by writing /assign @costinm in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

pkg/OWNERS [nmittler]
tests/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: nmittler, ozevren
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: costinm
If they are not already assigned, you can assign the PR to them by writing /assign @costinm in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

pkg/OWNERS [nmittler]
tests/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment