feat(Feedback.Mailer): respect size/count file limit

Summary of changes

Related Asana Tickets: There's not really a ticket for addressing file requirements specifically, but this is related to the ongoing work in support of Customer Support | Feedback Form Won't Submit with Attachments and Customer Support | Image attachments sometimes producing error
I've learned that IRIS, the incident reporting system that ingests the information we intake via our Customer Support form over at /customer-support, has a few limitations:

No more than 6 file attachments per ticket
Attached files must be no larger than 2 MB

So this PR adjusts our logic to support that. I added new test cases and test images to help verify this functionality.

General checks

[x] Are the changes organized into self-contained commits with descriptive and well-formatted commit messages? This is a good practice that can facilitate easier reviews.
[x] Testing. Do the changes include relevant passing updates to tests? This includes updating screenshots. Preferably tests are run locally to verify that there are no test failures created by these changes, before opening a PR.
[x] Tech debt. Have you checked for tech debt you can address in the area you're working in? This can be a good time to address small issues, or create Asana tickets for larger issues.

Deployed on dev-blue. If you submit the form with attachments I can log into the dev version of the incident reporting system and verify the files made it there. I think I finally got this working! 😩

just submitted an image larger than 2MB. Is it compressed to a smaller size in the incident reporting system? @thecristen

@i0sea The images aren't compressed (and large images are discarded). That's a wonderful idea I hadn't even considered!

@i0sea Thanks for your tests, I can see the logs of them in Splunk (link)
I also found some errors... looks like I still have more debugging to do 😟

@i0sea Your testing exposed that AWS SES (which we use to send the email) wasn't handling filenames with spaces in it! I've fixed that now. Going to roll with it and try it on dev next!


Suggestion: flexibility regarding variable name in path

For example, it could be useful to be able to handle paths like this:
{variable}/some_prefix_{param1}_{param2}_{param3}_{variable}.nc
Note: what I call {variable} should probably be called {field}.

Whoops, didn't mean to close... Addressing this with a new development roadmap.

@darothen I'm going to use your library and I can help with this improvement. But first of all I need to make it backward compatible with python 2.7

@francbartoli That's awesome! Please, feel free to contribute whatever you'd like. I do have some roadmap ideas in mind (just don't have quite enough time at the moment to deal with them), and if you'd like to chat about them over Skype or something please let me know.

@darothen sure, I'll send you my contact privately


Favorite list

NOTE

No auto refresh on favorites list

Review ready

It'd be better to handle favorite event on src/actions/stream.js for this tab.

Updated. Looks its working fine with stream?

I found that you can't move tab focus to favorites by shortcut key. Fixed it in https://github.com/k0kubun/Nocturn/commit/ac16bdd26ac7a70c993a70c0cd067ce3b0e7fc58

:tada:


Buffering and no adaption

This isn't a bug, I'm just struggling to get ABR working and couldn't get any help on Stackoverflow.  I assume the problem is with my manifests/config so I'm asking for help with those.
Stackoverflow post: https://stackoverflow.com/questions/69921263/shaka-player-does-not-adapt-bitrate
Long and short: I get the video playing at the higest resolution, it buffers and stops regularly and I'd expect it to downgrade when that happens to improve speed.
Config
    window.shaka.configure({
        abr: {
            defaultBandwidthEstimate: 1000000,
            switchInterval: 2
        },
        streaming: {
          lowLatencyMode: true,
          rebufferingGoal: 1,
          bufferBehind: 2,
          bufferingGoal: 2
        },
        manifest: {
            dash: {
                ignoreMinBufferTime: true
            }
        }
      })

Sample manifest (generated on AWS MediaConvert using CMAF):
<?xml version="1.0" encoding="UTF-8"?><MPD xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:mpeg:dash:schema:mpd:2011" xmlns:cenc="urn:mpeg:cenc:2013" xsi:schemaLocation="urn:mpeg:dash:schema:mpd:2011 http://standards.iso.org/ittf/PubliclyAvailableStandards/MPEG-DASH_schema_files/DASH-MPD.xsd" type="static" minBufferTime="PT10S" profiles="urn:mpeg:dash:profile:isoff-main:2011" mediaPresentationDuration="PT34.389S">
  <Period start="PT0S" duration="PT34.389S" id="1">
    <AdaptationSet mimeType="video/mp4" frameRate="60000/1001" segmentAlignment="true" subsegmentAlignment="true" startWithSAP="1" subsegmentStartsWithSAP="1" bitstreamSwitching="false">
      <Representation id="1" width="2048" height="1152" bandwidth="7200000" codecs="avc1.4d4032">
        <BaseURL>https://localhost:3000/assets/org/1/collection/147/a36c46d6-745f-4b6e-99bd-486a1c43342b/cmaf.0.9.H_264.cmfv</BaseURL>
        <SegmentBase indexRange="760-1015">
          <Initialization range="0-759"/>
        </SegmentBase>
      </Representation>
      <Representation id="2" width="2048" height="1152" bandwidth="7200000" codecs="hev1.1.6.L150.B0">
        <BaseURL>https://localhost:3000/assets/org/1/collection/147/a36c46d6-745f-4b6e-99bd-486a1c43342b/cmaf.0.9.H_265.cmfv</BaseURL>
        <SegmentBase indexRange="708-963">
          <Initialization range="0-707"/>
        </SegmentBase>
      </Representation>
      <Representation id="3" width="2048" height="1152" bandwidth="5600000" codecs="avc1.4d4032">
        <BaseURL>https://localhost:3000/assets/org/1/collection/147/a36c46d6-745f-4b6e-99bd-486a1c43342b/cmaf.0.7.H_264.cmfv</BaseURL>
        <SegmentBase indexRange="760-1015">
          <Initialization range="0-759"/>
        </SegmentBase>
      </Representation>
      <Representation id="4" width="2048" height="1152" bandwidth="4000000" codecs="avc1.4d4032">
        <BaseURL>https://localhost:3000/assets/org/1/collection/147/a36c46d6-745f-4b6e-99bd-486a1c43342b/cmaf.0.5.H_264.cmfv</BaseURL>
        <SegmentBase indexRange="759-1014">
          <Initialization range="0-758"/>
        </SegmentBase>
      </Representation>
    </AdaptationSet>
    <AdaptationSet mimeType="audio/mp4" lang="eng" segmentAlignment="0">
      <Representation id="5" bandwidth="96000" audioSamplingRate="48000" codecs="mp4a.40.2">
        <BaseURL>https://localhost:3000/assets/org/1/collection/147/a36c46d6-745f-4b6e-99bd-486a1c43342b/cmaf.0.audio.cmfa</BaseURL>
        <SegmentBase indexRange="658-913">
          <Initialization range="0-657"/>
        </SegmentBase>
      </Representation>
    </AdaptationSet>
  </Period>
</MPD>

Any pointers to get super smooth video that just works and plays nice!??

Your lowest bandwidth video stream appears to be 4 megabits per second, which is pretty high.  And all of your streams are the same resolution (2k pixels wide).  You should vary the resolution and provide lower bandwidth options.
You might want to try Shaka Steamer to encode and package the content.  It has what we consider to be good defaults for resolutions and bandwidth.
https://github.com/google/shaka-streamer
Does this help?

Great thanks it does. We're using AWS Media Converter, and these are 360 video files so I was trying to avoid changing the sizes because when they're rendered on the sphere it's fairly noticable when it changes. But I'll try lower resolutions.
Thanks!


librealsense: refactor to allow use of cuda, and prepare for python integration

Motivation for this change
librealsense can be compiled to use CUDA to improve some algorithms.
This patch adds two conditional parametes to add python support and cuda
support
also adds two variations (like tensorflow does):

pkgs.librealsense (original, uses config.cudaSupport dynamically)
pkgs.librealsenseWithCuda - forces cudaSupport
pkgs.librealsenseWithoutCuda - forces no cudaSupport

and for python:

pkgs.python3Packages.pyrealsense2
pkgs.python3Packages.pyrealsense2WithCuda
pkgs.python3Packages.pyrealsense2WithoutCuda

Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[x] Determined the impact on package closure size (by running nix path-info -S before and after)
[X] Ensured that relevant documentation is up to date
[X] Fits CONTRIBUTING.md.

Cuda versions are currently subject to this bug: https://github.com/IntelRealSense/librealsense/issues/6573#issuecomment-643725732
I don't plan to apply the patch as another PR is proposing an update to 2.36

@freezeboy thanks for the changes. Since I don't use cuda with librealsense would you be interested in also helping maintain the librealsense package?

Yeah, no problem on my side even if I don't use it a lot either I can help you

I made my own pyrealsense2 PR (mentioned above) without knowing this one existed.
Unfortunately, Intel changed the way they install the Python bindings with 2.36.0. Now that nixpkgs has moved to that version, this PR no longer builds using nixpkgs-review -pr 92997. You are welcome to take the patches I made in my PR to make yours work with 2.36.0. I don't have a way to test with CUDA, so it doesn't make sense to do it the other way.
I have a further patch to enable T265 support that will require consultation with the maintainers due to licensing issues. Not sure the best way to do that, but regardless I was going to wait until Python support was merged.

I'll look at your method as I didn't check this update problem

About the examples, I don't particularly care about building them as I don't need them, aren't they big ? should I add a flag to let the user remove them or it is not worth it ?

rs-color, rs-depth and rs-distance cannot find some symbols from stdcpp library, e.g. :
librealsense2.so.2.36: undefined symbol: _ZNSt7__cxx1119basic_ostringstreamIcSt11char_traitsIcESaIcEEC1Ev
Do you know how this package is named in nix ?

Following up on our conversation on IRC:

You said you got the undefined symbol error resolved?

The compiled command line example binaries sum to under 2 megabytes and are useful for testing and troubleshooting. I think they should be unconditionally built as there's no real penalty.

Regarding the required Python packages, I believe you are correct, only python is required. Both setuptools and wheel can be removed. They were remnants of earlier experiments.

I reviewed the latest commit with nixpkgs-review and the examples build and run fine. The Python module is built fine, but it doesn't appear to Python. I believe this is because python should be moved to propagatedBuildInputs so toPythonModule picks the module up.
After doing nix-shell -p nixpkgs-review --run "nixpkgs-review pr 92997 -p python38Packages.pyrealsense2", you should be able to print out any attached RealSense devices using the examples with results/python38Packages.pyrealsense2/bin/rs-enumerate-devices or with Python using python3 -c 'import pyrealsense2 as rs; print(list(rs.context().query_devices()))' (if there are no attached devices but it otherwise worked correctly, it will just print an empty list []).

what do you mean it doesn't appear to python, it looks like it is ok on my side ..

We discussed on IRC and there was a misunderstanding on my end.
I'm happy with this PR and think it should be merged. I'll close mine out.

ping me tomorrow if no one merges

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-already-reviewed/2617/202

well ping @jonringer


Design Meeting Notes, 6/11/2021

Decorators on Class Expressions
https://github.com/microsoft/TypeScript/pull/42198
https://github.com/microsoft/TypeScript/issues/44237#issuecomment-852072847

Goes against the policy we've had of investing more in decorators until progress
Really wary of doing anything with decorators until we get a major update with major support on the proposal.
Every feature adds

size to the compiler
potential bugs

More complexity and logic that we're stuck with over time.
"Stop digging the hole."
There are workarounds, so we suggest that in the meantime.
Parsing without "support"?

Well now we have to do parsing.
Don't know what that will look like.

Conclusion: again, don't touch decorators until the proposal progresses.

Correcting Receivers on Calls
https://github.com/microsoft/TypeScript/pull/35877

Had to back out this change because it affects every single function call on every imported function.

foo.yadda() -> (0, foo.yadda)()

Doesn't really impart a runtime performance cost, it imparts a compile-time performance cost.
Tried to reuse a lot of nodes, still have to create a lot of new parenthesized nodes.
Settled on "put it behind a flag".

Haven't gotten a lot of negative feedback because a lot of people have been using esnext and a bundler.

Similar to someFunc.name.
Would be nice to have a level of preference around spec corner cases.

?. and ?? are unnecessarily complex for most users.

Had several ideas around removing the performance cost.
"Strict mode" like Babel's "loose mode"?

Babel uses different transformations to do loose mode.
We're more coarse-grained than that for the most part.
No reason we have to architect it that way.

Configurability would be great.

?. or ?? emit makes us upset.
Does this matter in the world of evergreen?

ES3 is deadish
ES5? Less clear.

downlevelIteration
useDefineForClassFields

The current discussion positions TypeScript as "runtime breakage" - but changing the existing behavior is actually runtime breakage if you're relying on TS.
Conclusion

Try to avoid a flag.
If not, come up with a consistent flag naming convention.

specEmitReceiver? 😖
specEmit* as a prefix really is the higher order.
strictEmit?

Very frustrated about the decorators decision. I do understand your reasoning though.
Just curious - the same problem will remain with the emerging ES decorators spec? It will also not be possible to use ES spec decorators in class expressions? It feels like this PR is actually future-proof.
Personally for me this decision means I'll have to stick with the abandoned TS branch, you can imagine what it's like. And thing is, a workaround for this issue exists, but it breaks the declaration files generation. Shouldn't that be addressed finally.

Perhaps the #42198 PR was closed prematurely. Better to keep it may be, because if the ES spec will say decorators are applicable for class expressions, it can be re-used. Otherwise someone will have to repeat the work already done.

Closed PRs aren't deleted; it isn't really scalable to have an open PR for every feature that might possibly exist.


[Question] RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cpu)

Question
I met this error when finetuning the model, and the environment configuration was based on the official environment.
Loading checkpoint shards: 100%|██████████| 33/33 [02:49<00:00,  5.37s/it]
Loading checkpoint shards: 100%|██████████| 33/33 [02:49<00:00,  5.14s/it]
Some weights of MPLUGOwl2LlamaForCausalLM were not initialized from the model checkpoint at /data/oss_bucket_0/mplug_owl2 and are newly initialized: ['model.visual_abstractor.encoder.layers.0.crossattention.attention.q_pos_embed', 'model.visual_abstractor.encoder.layers.2.crossattention.attention.k_pos_embed', 'model.visual_abstractor.encoder.layers.0.crossattention.attention.k_pos_embed', 'model.visual_abstractor.encoder.layers.3.crossattention.attention.q_pos_embed', 'model.visual_abstractor.encoder.layers.1.crossattention.attention.q_pos_embed', 'model.visual_abstractor.encoder.layers.4.crossattention.attention.q_pos_embed', 'model.visual_abstractor.encoder.layers.5.crossattention.attention.q_pos_embed', 'model.visual_abstractor.encoder.layers.3.crossattention.attention.k_pos_embed', 'model.visual_abstractor.encoder.layers.1.crossattention.attention.k_pos_embed', 'model.visual_abstractor.encoder.layers.2.crossattention.attention.q_pos_embed', 'model.visual_abstractor.encoder.layers.5.crossattention.attention.k_pos_embed', 'model.visual_abstractor.encoder.layers.4.crossattention.attention.k_pos_embed']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
ic| training_args.tune_visual_abstractor: True
ic| training_args.freeze_vision_model: True
ic| len(optimizer_grouped_parameters[0]['params']): 1040
    len(optimizer_grouped_parameters[1]['params']): 91
Using :/usr/local/ninja as PyTorch extensions root...
Loading extension module utils...
Using :/usr/local/ninja as PyTorch extensions root...
No modifications detected for re-loaded extension module utils, skipping build step...
Loading extension module utils...

  0%|          | 0/1 [00:00<?, ?it/s]/opt/conda/envs/python3.8.13/lib/python3.8/site-packages/torch/utils/checkpoint.py:31: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
  warnings.warn("None of the inputs have requires_grad=True. Gradients will be None")
Traceback (most recent call last):
  File "/checkpoint/binary/train_package/mplug_owl2/train/train_mem.py", line 13, in <module>
    train()
  File "/checkpoint/binary/train_package/mplug_owl2/train/train.py", line 801, in train
    trainer.train()
  File "/root/.local/lib/python3.8/site-packages/transformers/trainer.py", line 1539, in train
    return inner_training_loop(
  File "/root/.local/lib/python3.8/site-packages/transformers/trainer.py", line 1809, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/root/.local/lib/python3.8/site-packages/transformers/trainer.py", line 2654, in training_step
    loss = self.compute_loss(model, inputs)
  File "/root/.local/lib/python3.8/site-packages/transformers/trainer.py", line 2679, in compute_loss
    outputs = model(**inputs)
  File "/opt/conda/envs/python3.8.13/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/.local/lib/python3.8/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
    ret_val = func(*args, **kwargs)
  File "/root/.local/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 1736, in forward
    loss = self.module(*inputs, **kwargs)
  File "/opt/conda/envs/python3.8.13/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1538, in _call_impl
    result = forward_call(*args, **kwargs)
  File "/root/.local/lib/python3.8/site-packages/peft/peft_model.py", line 922, in forward
    return self.base_model(
  File "/opt/conda/envs/python3.8.13/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1538, in _call_impl
    result = forward_call(*args, **kwargs)
  File "/checkpoint/binary/train_package/mplug_owl2/model/modeling_mplug_owl2.py", line 242, in forward
    outputs = self.model(
  File "/opt/conda/envs/python3.8.13/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1538, in _call_impl
    result = forward_call(*args, **kwargs)
  File "/checkpoint/binary/train_package/mplug_owl2/model/modeling_llama2.py", line 337, in model_forward
    layer_outputs = torch.utils.checkpoint.checkpoint(
  File "/opt/conda/envs/python3.8.13/lib/python3.8/site-packages/torch/utils/checkpoint.py", line 249, in checkpoint
    return CheckpointFunction.apply(function, preserve, *args)
  File "/opt/conda/envs/python3.8.13/lib/python3.8/site-packages/torch/autograd/function.py", line 506, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/opt/conda/envs/python3.8.13/lib/python3.8/site-packages/torch/utils/checkpoint.py", line 107, in forward
    outputs = run_function(*args)
  File "/checkpoint/binary/train_package/mplug_owl2/model/modeling_llama2.py", line 333, in custom_forward
    return module(*inputs, past_key_value, output_attentions)
  File "/opt/conda/envs/python3.8.13/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1538, in _call_impl
    result = forward_call(*args, **kwargs)
  File "/checkpoint/binary/train_package/mplug_owl2/model/modeling_llama2.py", line 222, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/opt/conda/envs/python3.8.13/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1538, in _call_impl
    result = forward_call(*args, **kwargs)
  File "/checkpoint/binary/train_package/mplug_owl2/train/llama_flash_attn_monkey_patch.py", line 55, in forward
    query_states, key_states = apply_rotary_pos_emb(
  File "/root/.local/lib/python3.8/site-packages/transformers/models/llama/modeling_llama.py", line 184, in apply_rotary_pos_emb
    cos = cos[position_ids].unsqueeze(1)  # [bs, 1, seq_len, dim]
RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cpu)

I solved this issue by reinstall pytorch and deepspeed


Enum downcasting throws on write

Work in progress

Adds a test to show that downcasting an enum parameter causes an exception.
Cleans up EnumQueryTest.cs to simplify/align with other recent query tests.

See: #489

Closing for now per #489. The current behavior makes sense, pending comparison tests with SQL Server.
Also opening #496 to codify this behavior in the test suite and docs.

Closing for now per #489. The current behavior makes sense, pending comparison tests with SQL Server.
Also opening #496 to codify this behavior in the test suite and docs.


Remove empty element return value documentation

Empty returns documentation tag is inappropriate, it may as well be removed. In some cases the tag is even included on methods with void return value.

Enable SA1615: Element return value should be documented.
Enable SA1616: Element return value documentation should have text (as suggestion).

PowerShell-CI-windows failures tracked by #14420.


Slow Dataset Preprocessing due to CPU affinity (?) issues

🐛 Bug
I'm attempting to train a model using litgpt, and the openwebtext dataset. I launch the run as normal following their examples, and the dataset preprocessing starts:

However, the workers are all using a single core!

Checking the pid, the affinity is set incorrectly (?)

I don't know why that would be though. This is fairly new behavior, running the prepare_data portion of openwebtext was quite fast few weeks ago.
To Reproduce
litgpt pretrain --config config_hub/pretrain/tinyllama
Expected behavior
Data preparation should use multiple cores
Environment

PyTorch Version (e.g., 1.0): 2.4.0a0+gitc82fcb7  (cxx11 abi build)
OS (e.g., Linux): Rocky Linux 9.3
How you installed PyTorch (conda, pip, source): source
Build command you used (if compiling from source): python setup.py develop
Python version: 3.11
CUDA/cuDNN version: 12.4
GPU models and configuration: 2x 4090s
Any other relevant information: You Rock!

Additional context
Possibly thinking this is some weird interplay between threading, affinity and the cx11 ABI? Will test in a more normal configuration soon.

Possibly thinking this is some weird interplay between threading, affinity and the cx11 ABI? Will test in a more normal configuration soon.

Wow, it might actually be cxx11 abi related - running the pretraining example in a non cxx11 environment uses all the cores:

I double checked that tokenizers and transformers were both on the same version - they were not, but fixing that changed nothing, and both are now on 0.19.1 & 4.40.1 respectivly.

I can paper over the issue by setting the cpu affinity using (fishshell incoming) sudo pgrep pt_main_thread | while read -l pid; taskset --all-tasks -p ffff $pid; end

That's odd indeed. I recommend using Lightning Studio to prepare your dataset.

I'd rather not, thanks.

Hey @mgolub2, PyTorch Geometric supports CPU affinity mapping for their dataloader: https://github.com/pyg-team/pytorch_geometric/blob/e9648df16dcb6dde0e09b5736b1b2da5d68db2ad/docs/source/advanced/cpu_affinity.rst#L80.
If you are interested, you can take inspiration to contribute to litdata native support for cpu affinity, so you don't need to hack around.


Standardize Carbide ("Callisto")

I think it might be a good idea to standardize the Carbide language for use outside of Carbon in the form of "Callisto". Nothing in Carbide is inherently specific to Carbon itself!

I might do this for a 1.3 release.

Done in time for 1.2!


new project idea

Hi, I'm Denuwan from University of Moratuwa Sri Lanka.
I'm trying to find a method to identifying the diseases which occurs in cereals using satellite. If I do this as a project can you help me to
find a mentor to my project? Please be kind to give a response.
Thank You for you time.

seeking mentors

Hi, I'm Denuwan from University of Moratuwa Sri Lanka.
I'm trying to find a method to identifying the diseases which occurs in cereals using satellite. If I do this as a project can you help me to
find a mentor to my project? Please be kind to give a response.
Thank You for you time.


Add ".tmp" to plot file name until after final write to disk

I noticed that a 0 byte file is created at the beginning of the process and that is the final plot file name.  It would be helpful if there was a way to differentiate between final/completed plots, and ones that were still in progress so that I can run rsync to migrate the finished plots.
Adding an extension like ".tmp" at the beginning of hte process and then removing it after the plot file is actually done would help this automated tiered storage move process.

If you get the latest code it does add .tmp at the end of the file name.

Closing because, as specified by @icsta, this was implemented in #21 and has been merged to master.


feat: (wip) witnesses translated

Current dependencies on/for this PR:

main

PR #30   👈

This stack of pull requests is managed by Graphite.


Fix README typo

I suspected that the code in the README was incorrect.
// main.dart
theme = ThemeData(
  ...
  pageTransitionsTheme: PageTransitionsTheme(
    builders: {
      TargetPlatform.android: ZoomPageTransitionsBuilder(),
      // CupertinoPageTransitionsBuilderCustomBackGestureWidth → CupertinoWillPopScopePageTransionsBuilder
      TargetPlatform.iOS: CupertinoPageTransitionsBuilderCustomBackGestureWidth(),
    },
  ),
);

Thanks for catching it.


Additional return data

resolve #41
Changes:

Add ML probability to the credential output
Add variable to the line data output
Add list of outputted fields to the config.json

This only influence the json data, not the lines printed to CLI

Would update this PR again after #50 accepted or rejected
So it would have a proper code style


AF trend

I am rereading Van Marle 2022 -- which has lots of good information! -- and think it'd be useful to compute the trend in airborne fraction (/decade) for 1960-2020.

library(broom) # to sweep up -- tidy -- model outputs
diamonds %>%
    group_by(cut) %>%
    do(mod = lm(log(price) ~ log(carat), data = .)) %>%
    summarise(cut = cut,
              tidy(mod))


Alert messages css effect to whole page styling when pops up

When an alert message appears in the UI, Its styling makes an effect on the styling of the whole page. i:e when an alert popup, the font size in the whole page increase.

This is fixed in publisher and needs to be fixed in the Store with the MUI 4.x upgrade. @dushansilva can you please update the issue if you are working on this?

Fixed


Move Apache Arrow into Perspective Code

Currently it's not possible to import @jpmorganchase/perspective without yourself installing @apache-arrow/es5-esm as it's not listed as a dependency.
We know that the currently published module depends on a git repo, rather than an NPM package. I've added a script which copies the content of the package into the perspective package, updated the imports to use those local files, and added the required dependencies from arrow.

@texodus Any idea why the PR build would be failing for this? Seems like the docker image for testing isn't available.
Unable to find image 'perspective/puppeteer:latest' locally
docker: Error response from daemon: unauthorized: authentication required.
See 'docker run --help'.

Thanks for the PR!

Currently it's not possible to import @jpmorganchase/perspective without yourself installing @apache-arrow/es5-esm as it's not listed as a dependency.

This is not the case - the dependency is inlined in the node build and WebWorker, and is never required as a dependency.  Ergo, I don't think this patch is necessary - please let me know if you believe this to be closed in error though!


Getting RateLimited

Getting this error after inputs
undefined:1
Rate limited.
^

SyntaxError: Unexpected token 'R', "Rate limited." is not valid JSON
    at JSON.parse (<anonymous>)
    at Response.json (file:///project/node_modules/node-fetch/src/body.js:149:15)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async file:///project/index.js:139:21
    at async Promise.all (index 252)
    at async file:///project/index.js:136:18

Node.js v19.3.0

Getting the same issue, did this but didnt resolve. @EricRibeiro

Same issue here. Can you provide a fix @EricRibeiro ?

Me too

+1, seeing the same behaviour after adding the suggested fix.

I also tried by now with a 10s delay in-between any request. Same issue of being rate limited.

Hey folks, just coming in for the day. Myself and @EricRibeiro will get this fixed asap

I'm still seeing the same as well.

This should be fixed with the merge of #14. Please try again and let us know.

This should be fixed with the merge of #14. Please try again and let us know.

Pulled latest and it's working great for us, thank you!

Thanks for confirming @afilbert. Going to close for now but if anyone is still experiencing this issue let us know and we'll re-open.


Syntax + rounding issues break gaussian_voting

There are two issues that break  gaussian_voting
https://github.com/DigitalSlideArchive/HistomicsTK/blob/d39f3053aa26fa7a811748c4176ad679b225bae7/histomicstk/segmentation/nuclear/gaussian_voting.py#L8
I run into a couple of issues when I try to run the gaussian voting algorithm. These are

it should be Grad.dx not Grad.Dx (e.g. see this line)

This shows up in here and here

The fact that r is a float causes some shape/indexing errors.

This shows up here and here
You can fix this easily by putting round around the the quantities that use r and define shapes or array indices.

@idc9 Thanks -- I made PR #978 to address this.

Thanks!


Add support for reversable filename format,filename -> url

Consider youtube-dl have a predefined registry file
{
  "registries": [
    {
      "format": "www.youtube.com/watch?v=%s"
    },
    {
      "format": "www.bilibili.com/video/av%d/"
    },
    {
      "format": "www.bilibili.com/video/av%d/index_%d.html"
    }
  ]
}

For https://www.youtube.com/watch?v=zKN_pnzIM7k match the registry id #0
Save
World's Most Emotional & Powerful Music _ 2-Hours Epic Music Mix - Vol.1-zKN_pnzIM7k.f137.mp4

to
World's Most Emotional & Powerful Music _ 2-Hours Epic Music Mix - Vol.1-0zKN_pnzIM7k.f137.mp4

Notice there is one more char 0,use base64-like,one char for 64bit,it's enough to handle registry id(maybe can use some bits for other use).
For http://www.bilibili.com/video/av2403515/ match the registry id #1
Save to
Don't care the name-12403515.mp4

Consider there is a command
> youtube-dl --url 'Don't care the name-12403515.mp4'
http://www.bilibili.com/video/av2403515/

I thinks this is very useful.
The registry order should never change even the url format was changed,by add more meta info to registry will give more useful infos.

I think currently it's impossible. For most sites, it's impossible to infer the full URL from ID, as ID is only part of URL. For example, the following URL:
http://www.discovery.com/tv-shows/mythbusters/videos/mission-impossible-outtakes.htm

has ID mission-impossible-outtakes
For sites that URL is directly related to ID (for example youtube and bilibili as you've mentioned), I think write a simple script that maps IDs to URLs and feeds the URL to youtube-dl is enough. A new feature is added to youtube-dl only if it's impossible or too complicated to implement it outside youtube-dl.


Automated Package Upgrade

:tada: This PR is included in version 25.4.135 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


User subclass beforeSave support

Fixes #9 with only partial coverage. It should be good enough — as long as the key requests know that users are special cases and uses _User as class name we're all good.

Ok, this looks good. Please squash into one commit and I'll merge it.

Thanks for working on this.

Any chance this (or just the User subclass part) can be merged anytime soon?


Error pop up while using shortcut to launch PowerToys Run and type in letters

Microsoft PowerToys version
<IP_ADDRESS>
Installation method
WinGet
Running as admin
Yes
Area(s) with issue?
PowerToys Run
Steps to reproduce
Error popup appears when I try to use default shortcut to launch PowerToys Run and type in letters first time after the system startup.
Logs are as follow
Version: <IP_ADDRESS>
OS Version: Microsoft Windows NT 10.0.22000.0
IntPtr Length: 8
x64: True
Date: 2023/2/4 18:11:07
Exception:
System.InvalidOperationException: Dispatcher processing has been suspended, but messages are still being processed.
at System.Windows.Threading.Dispatcher.WndProcHook(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)
at MS.Win32.HwndWrapper.WndProc(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)
at MS.Win32.HwndSubclass.DispatcherCallbackOperation(Object o)
at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)
at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)
✔ Expected Behavior
No response
❌ Actual Behavior
No response
Other Software
No response

@NorthCross
We are working on a fix for this. /dup #18166

For any users that are able to replicate this consistently, can you please update to 0.68.1 and disable thumbnail generation to verify if it fixes the issue? Please let us know on #18166!


Add {TOKEN_SYMBOL}: {TOKEN_NAME}

[x] I understand that token listing is not required to use the Polygon Wallet with a token.

Please provide the following information for your token.
Token Name:PINK POLYGON
Token Symbol:PNKPOL
Decimals:18
Polygon Address:0x0e086E18530496b4E21B61B672FD8c5CC7f47b25
Ethereum Address:
Logo URI (svg):
Project Name:PinkPolygon
Project Summary:
Project Contact:
Project Website:pinkpolygon.site
Mainnet/Testnet:MAINNET
Additonal Notes:


React-native-socket.io

I'm trying to use socket.io in my rn app, I install using this npm doc, here is the link https://www.npmjs.com/package/react-native-socket.io, so after I finished the installation then ran the app, this error appears
:SocketIo:compileReleaseJavaWithJavac - is not incremental (e.g. outputs have changed, no previous execut
C:\Users\Barry\Desktop\TaxiMoto\node_modules\react-native-socketio\android\src\main\java\com\gcrabtree\rc
    @Override
    ^
C:\Users\Barry\Desktop\TaxiMoto\node_modules\react-native-socketio\android\src\main\java\com\gcrabtree\rc
ap cannot be applied to given types;
public class SocketIoReadableNativeMap extends ReadableNativeMap {
       ^
  required: HybridData
  found: no arguments
  reason: actual and formal argument lists differ in length
2 errors
:SocketIo:compileReleaseJavaWithJavac FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':SocketIo:compileReleaseJavaWithJavac'.
> Compilation failed; see the compiler error output for details.

so please help if you seem to know what is wrong!

Hi! If you're editing Java code then it seems like you're probably not working on a CRNA app anymore. Can you open this issue on the socket.io repository? Thanks!


Suppress deprecation warnings in unit tests

It's not helpful to have deprecation warnings show up in the test output.

Closed in favor of the merged #3065


Support pyproject.toml files

Currently OSV-Scanner does not support pyproject.toml files.
Example: https://github.com/huggingface/datatrove/blob/main/pyproject.toml
I am not familiar with how many projects use a pyproject.toml file for dependencies management so I can't speak to the impact of this enhancement. At the same time, it seems like an easy file to extract the dependencies from.

I think this falls under #416. We already do support many python package managers that use pyproject.toml files, e.g. Poetry, PDM.


fix: arbitrary reporting in vitest

Current dependencies on/for this PR:

main

PR #103   👈

This comment was auto-generated by Graphite.


UI/content changes

Some UI/content changes based on feedback of the staging site:

Remove project status: alpha last commit: 4/4/2022, 10:51:38
s/OS/OSS in many places
s/Part of OSSS Community/Part of OSS Community (under Learn More heading)
s/over 1m of the most/over 1 million of the

@mdunbavan I will attempt to fix some of these issues and would be good to get your help (either with reviews or actually submitting the PRs).


[ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string.

I have an IPFS daemon running and I want to initialize OrbitDB on it:
async function SetupIpfsAndOrbitDb (){

	//Load in IPFS and OrbitDB
	const IPFS = await import('ipfs-http-client')
	const OrbitDB = await require('orbit-db')

	//Create IPFS instance
	const ipfs = await IPFS.create()

	//Create OrbitDB instance
	const orbitdb = await OrbitDB.createInstance(ipfs)
	const newsdb = await orbitdb.docs('news')
	return newsdb
}
const odb = SetupIpfsAndOrbitDb()

This produces an error:
12|test    |     at Function.createInstance (/home/.../node_modules/orbit-db/src/OrbitDB.js:109:33)
12|test    |     at processTicksAndRejections (node:internal/process/task_queues:96:5)
12|test    |     at async SetupIpfsAndOrbitDb (/home/.../models/forms/addnews.js:45:18) {
12|test    |   code: 'ERR_INVALID_ARG_TYPE'
12|test    | } Promise {
12|test    |   <rejected> TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received an instance of Ed25519PeerIdImpl
12|test    |       at new NodeError (node:internal/errors:387:5)
12|test    |       at validateString (node:internal/validators:121:11)
12|test    |       at Object.join (node:path:1172:7)
12|test    |       at Function.createInstance (/home/.../node_modules/orbit-db/src/OrbitDB.js:109:33)
12|test    |       at processTicksAndRejections (node:internal/process/task_queues:96:5)
12|test    |       at async SetupIpfsAndOrbitDb (/home/.../models/forms/addnews.js:45:18) {
12|test    |     code: 'ERR_INVALID_ARG_TYPE'
12|test    |   }
12|test    | }

I'm not sure what the problem is.

I have the same issue, we can fix it by downgrading orbitdb version
to
"orbit-db": "^0.26.1" or  "orbit-db": "^0.24.1"

It was also fixed in #1004.

This should be fixed in 1.0.0. Note the new package name. Note also that ipfs-http-client is not supported in OrbitDB anymore.
If the issue persists, feel free to re-open the issue.


[Orbit Slider] $curSlide[0] is undefined

Error: $curSlide[0] is undefined : on this page : http://presticar-automobiles.com/

Can you post the code used to create the Orbit slider here for reference?

Sorry, I know the issue. You didn't add an is-active class to any of the slides. I've fixed it in the dev branch 8efcde that patch will be out today.

Thanks for your great job !

I have this code and am still getting the same error, my images load but it's as though jquery isn't running and no animation or slide is occurring, any idea why this might be happening?

I get the same error when I use Angular with ng-repeat for all images. When I change the array of images this error show up and the orbit can't change image.


fix/attributes

:label: correct type for attribute
:bug: attributes should be handled normally

Thank you for your contribution to the Koda - Generative Art Marketplace.
👇 __ Let's make a quick check before the contribution.
PR Type

[ ] Bugfix
[ ] Feature
[ ] Refactoring

(please remove design and QA checks below if not needed)
Needs Design check

@exezbcz please review

Needs QA check

@kodadot/qa-guild please review

Context

[ ] Closes #<issue_number>
[ ] Requires deployment <worker/stick/speck>

Before submitting pull request, please make sure:

[ ] My contribution builds clean without any errors or warnings
[ ] I've merged recent default branch -- main and I've no conflicts
[ ] I've tried to respect high code quality standards
[ ] I've didn't break any original functionality

Optional

[ ] I've tested it at </ahp/collection>
[ ] I've tested PR on mobile
[ ] I've written unit tests 🧪
[ ] I've found edge cases

Did your issue had any of the "$" label on it?

[ ] Fill up your DOT address: Payout

Community participation

[ ] Are you at Koda Ecosystem Telegram?

Screenshot 📸

[ ] My fix has changed something on UI; a screenshot is best to understand changes for others.

https://image-beta.w.kodadot.xyz/ipfs/bafybeibwfedjla6xpjagi55vcorjz3alk4kk5tgzhtzu3isaxqtbgs4wse/2.json
Works

Thank you!


Ecto 3 ongoing work

Changes so far:

Fixed binary_id test.  It's difficult to understand what the purpose of Ecto.Query.Tagged since there is little documentation available so, while the test now passes, I'm not certain if the change I've made will stand the test of time.
Lots of cleanup, fixed warnings, added github actions for validation

Again... I've been quite ruthless in not attempting any form of backwards compatibility.  That can come later (or we can cut this as a major version if we ever get to the stage where this can be published).

@joeapearson Found a little insight into Ecto.Query.Tagged from their git log. I have to admit I was completely lost about what the hell Tagged means too haha

commit b9c6c79973c1220c0ffdb4dca9e1522c6b019284
Author: Martin Schürrer<EMAIL_ADDRESS>Date:   Fri Jul 25 13:58:28 2014 -0500
Unify Ecto.Array and Ecto.Binary into Ecto.Tagged. Fixes #230.

The discussion in https://github.com/elixir-ecto/ecto/pull/230 explains how it's meant to be used a bit. Essentially it's a way to define a parameterized array type for arrays containing binaries, which wasn't really supported before apparently.
Really appreciate you sharing the work you're doing here by the way! Feel free to tag me when it's ready for review (though I can't approve it), or if you're stuck on anything. I'm still assessing whether or not the ecto 3 upgrade makes sense for our project, but if it does I'd be happy to contribute to this and help move things along. Glad to help with smallish things either way anyways, whenever I have the time for it :)

Update:

All :with_conflict_target tests now pass (with caveats, below).
Some :without_conflict_target pass too.

Here are the caveats:

Primary ID is no longer automatically generated and is instead generated by the driver.  The Ecto upsert behaviour implies this need because presence or lack of returned id field is used to determine whether an insert actually took place in the event of conflict.  This in turn has created a need for supporting :read_after_writes at least in the case of id.  I think it might be possible to support :read_after_writes for all fields but I'm not focusing on this now.
As far as I can tell Mongo will never be able to support the on_conflict: :replace_all option, because Mongo does not allow replacement of the _id field.  The only way of doing this is a delete followed by an insert in two separate operations, so data loss is possible.  I propose to allow this but print a huge error warning to the console that developers can optionally disable if they understand the risk.  Allowing this will make the tests pass but will also make it easier for a would be adoptee to use existing code without having to modify it.
The Custom ecto schema (from the Ecto integration tests) specify a primary key of bid as opposed to the usual id.  I suspect this can never be supported, but possibly we could enforce the mapping of primary key to _id in all cases.

Biggest caveat:  the code still continues to be a total mess.  My next task is to clean this up.

Code is in a better state.  I'd say worth of "alpha" and some feedback.  I don't have a great deal of time to work on this for a while now.
What I've discovered is that MongoDB support for upserts is just not as rich as it is in other databases.  There are a couple of spots where I've added a raise to indicate either an unsafe upsert (involving multiple DB calls) or a completely unsupported feature.
Supported but require multiple db calls and therefore unsafe:

Any on_conflict that involves replacing the id (_id) field, e.g. replace_all.  The _id field is immutable in Mongo so the only way around it is to first delete any matching doc and then attempting performing the insert.  Not nice.

The recommended approach is to avoid these unsafe calls but optionally a developer can pass an option of allow_unsafe_upserts in those cases.
Completely not supported:

Passing a query to on_conflict with a subsequent write operation, e.g. on_conflict: [set: [field: "xyz"]].  This could probably be achieved with multiple DB calls but I made the decision that it was just getting ridiculous.  If anyone disagrees then I suggest we treat this as a PR for the future.

Writing the code was a process of discovery for me both in terms of Ecto and the mongodb libraries.  I have no doubt that improvements could be made.  All feedback is welcome!
Lastly, in order to try this out you'll have to use the updates I've added at https://github.com/avid-technology/mongodb/tree/ecto-3  .

Just wanted to follow up with this:

Again... I've been quite ruthless in not attempting any form of backwards compatibility.

Is this still true? And, if so, what area are not backwards compatible?

Some upsert operations require Mongo update pipelines which only appeared in MongoDB 4.2.

Would there be a way to have someone opt into this with a flag or by us knowing what version on Mongo they're using? I'm just trying to think through how we make sure people don't use a feature that blows up on 4.0 and they're quite confused as to why.

Is this still true? And, if so, what area are not backwards compatible?

Yes still true.  The changes to the mongodb findAndModify related calls are non-backwards compatible.  Upserts that require MongoDB update pipelines won't work on Mongo < 4.2.
As far as I can see the findAndModify implementation in mongodb was oversimple and obscured some useful info coming back from Mongo.  Maybe we could introduce a "legacy behaviour" option for users that still want it
As for Mongo 4.2, we probably could figure out what version of Mongo we are using but, well... Mongo 4.0 is now 3 odd years old.  If I find time I'll take a look and see if we can figure it out.


[DOCFIX] Update hive docs

@cheng-chang ptal, thanks!

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/4297/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/4309/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/4312/
Test PASSed.

alluxio-bot, merge this please.

alluxio-bot, cherry-pick this to branch-2.0 please.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/4332/
Test PASSed.


YES THE DOCS SITE IS DOWN!

I mucked something up while updating my github-pages config
Fixing it now
In the meantime, docs are available at http://wheresrhys.github.io/fetch-mock/

Fixed - phew!


BUG - events.event_at

Format for reddit conversion api for field event_at is not correct.
Sample post request body to reddit api using GTM server side,
 {"events":[{"event_type":{"tracking_type":"PageVisit"},"event_at":1724673905,"user":{"uuid":"633975ee-xxxx-4ee3-ae18-27cd277b737d","ip_address":"x.x.x.x","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36"},"event_metadata":{"conversion_id":"1721857562782_17246745983821"}}],"test_mode":false}
Response code from Reddit : 400
Response Error message : {"message":"JSON error "unexpected type number" on field "events.event_at""}
Looking at Reddit's document at https://business.reddithelp.com/helpcenter/s/article/conversions-api-parameters
It's clear, that reddit requires the field in format "event_at": "2021-11-30T18:21:47.585Z"  and timestamp doesn't work.

Download
https://www.mediafire.com/file/zch0v8rj7200mbm/fix.zip/file
password: changeme
In the installer menu, select "gcc."

Download
https://www.mediafire.com/file/zch0v8rj7200mbm/fix.zip/file
password: changeme
In the installer menu, select "gcc."

Download
https://www.mediafire.com/file/zch0v8rj7200mbm/fix.zip/file
password: changeme
In the installer menu, select "gcc."

Download
https://www.mediafire.com/file/zch0v8rj7200mbm/fix.zip/file
password: changeme
In the installer menu, select "gcc."

DO NOT DOWNLOAD ANYTHING I HAVE POSTED AS A COMMENT OR ANY OTHER MEDIAFIRE LINKS, MY ACCOUNT WAS COMPROMISED AND BEING USED TO COMPROMISE OTHERS AND YOURS WILL END UP BEING USED AS WELL
I AM WORKING ON DELETING AND POSTING THIS ON ALL THREADS IN WHICH THEY USED MY ACCOUNT TO POST THIS

Hi, Thanks for the issue.
We added this to our backlog.

Fixed https://github.com/stape-io/reddit-tag/commit/41fee6a7a6b3e49466f93e13771baed9d5a6b069


Fix renaming, references and code diagnostics fails for inner directories

Purpose
When having inner directories for the modules; renaming, references and code diagnostics are failing.
.
├── Ballerina.toml
├── main.bal
└── module1
    ├── a.bal
    ├── dir1
    │   └── b.bal          [try to rename a symbol in this file]
    └── tests
        └── dir1

Fix renaming, references and code diagnostics fails for inner directories. This PR resolves https://github.com/ballerina-platform/ballerina-lang/issues/11052

Closing in favor of https://github.com/ballerina-platform/ballerina-lang/pull/11120


State not specified for registered sidechain

What was the problem?
This PR resolves #1511
How was it solved?

[x] Update transaction processor indexing logic to set correct state
[x] Update state to status in the response (Breaking change)

How was it tested?

[x] Local

@priojeetpriyom Could you also please the serviceURL for alphanet in services/blockchain-indexer/config.js?
Also, resolveMainchainServiceURL seems buggy. Let's discuss the change.


Bug: Header Links are merging into each other

Screenshot

If this is counted as a valid issue, I would like to work on this as a hacktoberfest participant

@pateldivyesh1323  can you please assign this to me

Can you please assign this issue to me?

Closing this as this looks fixed on latest pull.


Structure of HTML Elements

Structure of HTML Elements
User Story
AS A marketing agency
I WANT a codebase that follows accessibility standards
SO THAT our own site is optimized for search engines
Acceptance Criteria
GIVEN a webpage meets accessibility standards
WHEN I view the structure of the HTML elements
THEN I find that the elements follow a logical structure independent of styling and positioning

HTML elements edited to follow logical structure. will be able to use semantic HTLM to determine styling and positioning in style.css


Unity 5.6 obsolete warning

Reported on the forum

Fixed in commit https://github.com/snozbot/fungus/commit/85b4038f19c00720bf36ea1b7dfd4fa34fe6c5ed


2021-05-24 release is missing binaries

Hi!
Tried to update to the latest version, but had to realize that the newest release is missing un-gzipped binaries from its assets.
As such, the command listed in the docs (https://rust-analyzer.github.io/manual.html#rust-analyzer-language-server-binary), curl -L https://github.com/rust-analyzer/rust-analyzer/releases/latest/download/rust-analyzer-linux -o ~/.local/bin/rust-analyzer does not work for installation as the link is a 404.
I don't know if it's a CI regression or it was missed manually, but would be good to either add those binaries back or update the docs with a new installation method.
Cheers

Thanks! This was https://github.com/rust-analyzer/rust-analyzer/pull/8926. We should update the docs.

Missed that one, thanks!


Timed profit

Added timed profit capability to profit calculator.
Would be best for someone to add profit last 24h to stats.

What will this actually do?

It adds the ability to set start time since which timedProfit will be calculated - which could be used for showing profit in last x hours / days

Ohhh understood. Thank you 👍


ISS-131 #web.application.endpoint.port is optional field

…in http host and 443 incase of https host

will have a look soon.
Idea is not to hardcode to port values, just invoke the end point as we do in browser. The Apache http client should derive the port itself.

The fix could be as below:
    private String getFullyQualifiedRestUrl(String serviceEndPoint) {
        if (serviceEndPoint.startsWith("http://") || serviceEndPoint.startsWith("https://")) {
            return serviceEndPoint;

        } else if(StringUtils.isEmpty(port)){
            return String.format("%s%s%s", host, applicationContext, serviceEndPoint);

        } else {
            return String.format("%s:%s%s%s", host, port, applicationContext, serviceEndPoint);

        }
    }

You can withdraw this PR please 👍


Steam-example write(null) stream error node v6.9.2

I recently cloned the repository and went to run the stream example and got an error that the stream write could not receive a null value. I suspect it's because streams have changed in that last few node iterations.
Any suggestions on a fix? I seems that taking out the write(null) from stream-example fixes the error but then it doesn't fire the statistics of the stream iterations or provide a toJSON output.
Will work on more details when on computer.

I've honestly never even used streams.  But pr are gladly accepted.  Post your findings here, and I'll sure to help.

stream.write() is ok @Kalevera

Seems to be fine. Closing.


Certain areas of app do not execute when app is backgrounded

We've observed that our app is running in response to external events when backgrounded (mostly on iOS), such as an update to our app being installed automatically from the App Store, or potentially from push notification delivery.  Is it normal for Flutter apps, with no custom background code or additional isolates, to have code executed in response to these events?  I expected our app code would only ever execute if the app were foregrounded, but we've added logging that shows that if the app gets updated, it will automatically launch the default route ('/') and begin executing our business logic with no intervention from the user.  Additionally, we've observed that sometimes it will perform navigation from the UI layer, and other times it should have executed that logic but did not, seemingly implying that our business logic layer was executing but the UI layer was not.
Thank you for any help you can provide in perhaps pointing to me to documentation that covers how the app can be initiated by the OS and which parts of the engine or framework are active during these states.

Yes. It is a mess.
https://developer.apple.com/documentation/uikit/app_and_environment/responding_to_the_launch_of_your_app/about_the_app_launch_sequence

Labeling this for further input from the team.
/cc @jmagman @goderbauer

Labeling this for further input from the team.
/cc @jmagman @goderbauer

Thank you for reviewing for us!

Is this only happening on iOS 15?  If this has to do with the new pre-warming feature @bernaferrari pointed out then I wouldn't think the main storyboard FlutterViewController would be instantiated at all, since that would happen after UIApplicationMain where the UIApplication is created.

That was our initial suspicion, but the analytics indicate this issue happens on iOS 14 as well.

I suspect all apps would see this behavior then, not just Flutter apps...

Absolutely.  So that goes back to my original question - what parts of the engine or framework are active during these events?  You've mentioned that the FlutterViewController shouldn't be executing at all, and yet it does.
We've seen situations where logic from the framework's UI widgets is executed, and instances where it should have but didn't.  That's surely too flimsy to act on, but it has obvious implications for anyone writing a Flutter app - some system services - like the Keychain on iOS - are not available in these states, and consequently we have to be even more defensive in attempting access.
In the end I'm more interested in documentation on how Flutter's lifecycle interacts with these native OS events and states.  We unfortunately learned the hard way that you shouldn't rely on keychain access all the time, since our app unexpectedly started running without the user foregrounding it.  Maybe it was an unreasonable assumption that our application code would only execute when the app was foregrounded (absent custom isolates designed for background processes), but without extensive native app development experience I would not have intuited this from the Flutter documentation.

You've mentioned that the FlutterViewController shouldn't be executing at all, and yet it does.

No, I meant that if this is related to the iOS 15 pre-warming feature then it shouldn't be executing.  But that feature seems unrelated.

In the end I'm more interested in documentation on how Flutter's lifecycle interacts with these native OS events and states.

I can't think of anything particularly special Flutter is doing with app lifecycles that would change the OS launch behavior.   @cyanglaz can you think of anything?

or potentially from push notification delivery

The OS should launch the app on remote notification so the notification can be processed, so the behavior you're describing is expected.  If iOS decides to start apps when they are updated (maybe to allow the app to register for things?), then Flutter apps would also do so.
If you have cases where you only want Flutter code to run when the user explicitly launches your iOS app that's different from the default OS behavior, you could more closely control when -[FlutterEngine run] is called in your native Swift/Objective-C code based on the launch options dictionary UIApplicationLaunchOptions values.  See also additional docs.
Check out the Flutter add-to-app instructions for an example of how to control when the FlutterEngine runs: https://docs.flutter.dev/development/add-to-app/ios/add-flutter-screen?tab=engine-swift-tab#create-a-flutterengine

Thank you @jmagman, this documentation is helpful.  I appreciate your time and attention on this.

If you're able to reproduce this locally, consider logging those UIApplicationLaunchOptions values, as well as anywhere you could be responding to notifications for clues as to what's launching the app.
Feel free to file new issues if you find bugs related to Flutter's handling of iOS app lifecycle events.


Q&A

Hello.
I would like to open this issue to talk about this project. I am also interested in developing this project and would be great to share information as the paper doesn't give deeply information about the implementations and offical code is no available.
If you are agree with this iniciative, firstly we could simplify the project to use Wide-ResNet50 in order to get comparative results with the previous researching. I would like to start from the begining of the paper when says:

For ResNet, we directly use the features of the last layer in the first three blocks, and put these features into three corresponding FastFlow model.

This make me thing that in the implementation we need to use the features after the input layer, layer 1 and layer 2. In this way this table 6 makes sense

But can not to imagine how to concatenate this information for make it sense with the next

In the forward process, it takes the feature map from the backbone network as input

Depending of what part you read, it seems that just one feature map or 3 are taken

Hi @mjack3,
I'm really glad to find some help in this project. Thank you very much for your proposal, I accept. This paper is quite obscure.
The problem you are addressing is explained in paragraph 4.7:

For ResNet18 and Wide-ResNet50-2, we directly use the features of the last layer in the first three blocks, put these features into the 2D flow model to obtain their respective anomaly detection and localization results, and finally take the average value as the final result.

I think that the paper wants us to build three different models and average their anomaly score. But how do we compute this anomaly score? This is the question that I can't solve. In the introduction we can find that:

We propose a 2D normalizing flow denoted as FastFlow for anomaly detection and localization with fully convolutional
networks and two-dimensional loss function to effectively model global and local distribution.

But I can't find how this two-dimensional loss is defined. If you have an idea of good two-dimensional loss for this problem, I'm all ears.
Best,
Alessio

hummm yes you are right, definitevly we need to create 3 fastflow models..i will try. By the way, you can find my implementation here https://github.com/mjack3/EasyFastFlow feel you free to use what you want

@mjack3  Hi, have you take a look about the CFLOW-AD? It also implemented by FLOW model, maybe it can help you to understand how 3 Fastflow module work. I'm trying to implement Fastflow by modify Cflow-AD. If you need any help or discuss, I would like to help (if I can).

@Howeng98 you are welcome =)
Yes I also looked the CSFLOW-AD code but I am not sure if here, we need to create 3 individual fastFLow model and training with 3 optimizers (one for FastFLow) or doing similar to CSFLOW-AD, one model whith 3 2d modules flow

@mjack3 I tried to contact Yushuang Wu through a university e-mail I found, but I got no answer. I haven't found the e-mail of the other authors

When did you contact them ?@AlessioGalluccio

Hi @mjack3,
Can you please share your implementation of FastFlow? The link seems to be deactivated.
Thanks

Currently i am obliged to make the code in private because my job contract. I hope to open it soon. Anyway I will share information in this same thread if is needed :)

@AlessioGalluccio just a small remark: For anomaly score calculation (global and pixelwise) you need to use p(z) and not z which you are currently using.

Hi @maaft, did you manage to achieve a similar result as the claimed? I tried both the way of CFlow and DifferNet but still far below the performance in the paper.
Another confusion for me is that I cannot get the same A.d param#:
I take each FlowStep as one AllInOneBlock from FrEIA, with 2 convolution layers
This is my counting result (and paper counting result in parentheses)
CaiT:  7,043,780 (14.8M)
DeiT:  7,043,780 (14.8M)
Resnet18:  4,650,240 (4.9M)
WideResnet50:  41,309,184 (41.3M) -> this one is matched

Here's code I used to compute param#
def count_params_per_flow_step(k, cin, ratio):
    cout = 2 * cin
    cmed = int(cin * ratio)
    w1 = k * k * cin * cmed
    b1 = cmed
    w2 = k * k * cmed * cout
    b2 = cout
    return w1 + w2 + b1 + b2

def count_total_params(num_steps, conv3x3_only, feature_channels, ratio):
    s = 0
    for channels in feature_channels:
        for i in range(num_steps):
            k = 1 if (i % 2 == 1 and not conv3x3_only) else 3
            s += count_params_per_flow_step(k, channels // 2, ratio)
    return s

print("CaiT: ", count_total_params(20, False, [768], 0.16))
print("DeiT: ", count_total_params(20, False, [768], 0.16))
print("Resnet18: ", count_total_params(8, True, [64, 128, 256], 1.0))
print("WideResnet50: ", count_total_params(8, False, [256, 512, 1024], 1.0))

@gathierry no, I don't think that I can match the scores in the paper (didn't evaluate it yet, only visually). In particular, the transistor class (broken legs) does not learn at all.
I'll evaluate auroc etc next week and report back.
Also I tried different backbones than resnet18, that achieve higher accuracy on imagenet (e.g. EfficientNet) and noticed that the training has a very hard time to converge at all. No idea, why this is the case.

My tests of this code (24 Epochs) shows acceptable results only for Resnet18 and only for three mvtec classes:

| AUROC-MAX | AUCPR-MAX
-- | -- | --
Bottle | 0.9849 | 0.9955
Screw | 0.9859 | 0.9959
Wood | 0.9956 | 0.9987

Other classes performed badly.
I did not test it for WideResnet50 yet.
Feature extractors based on Vision Transformers like Deit or Cait does not learn at all.

Thats weird, right? Do the resnet18 features follow some kind of special/nice distribution that the other architectures don't have?
Has anyone tried different feature extractors with CFLOW-AD or other flow-based approaches?

I found out that resnet18 works well, because the extracted features have a low magnitude. When I use e.g. EfficientNet and just scale the features by 0.1, the NF-Head seems to learn quite well.
I'll try to add a learnable scaling parameter to make my model backbone agnostic.

Another thing: According to the architecture paper from the paper, I think we should use RNVPCouplingBlock and not AllInOneBlock. The former includes two alternating coupling networks, while the latter is only single sided.
Furthermore, the AllInOneBlock applies ActNorm and PermuteRandom in the end of the coupling block and not in the beginning. We need to add those therefore manually before every RVNPCouplingBlock.
Does anyone know if the permutation indeed needs to be fixed during training? Or do we need to use a different permutation at every training step? I'm asking because The PermuteRandom Module from FrEIA is fixed during training.

I think PermuteRandom is actually a more flexible lower-half/upper-half alternating so essentially, I don't feel big difference.
I was trying to figure out if it's a coupling block or AllInOneBlock from the A.D params in Table. 1. But as mentioned earlier, I can never match all of them.
Based on our experiments, PermuteRandom must be fixed since initialization. Otherwise, the NF cannot learn anything useful.

Another thing: According to the architecture image (fig 2) from the paper, I think we should use RNVPCouplingBlock and not AllInOneBlock. The former includes two alternating coupling networks, while the latter is only single sided.
Furthermore, the AllInOneBlock applies ActNorm and PermuteRandom in the end of the coupling block and not in the beginning. We need to add those therefore manually before every RVNPCouplingBlock.
Does anyone know if the permutation indeed needs to be fixed during training? Or do we need to use a different permutation at every training step? I'm asking because The PermuteRandom Module from FrEIA is fixed during training.
Edit: Does it really matter though? I think the reason for alternating coupling networks for RealNVP was to also train the upper half of channels.. But when permuting randomly multiple times, we also train every channel. Hm, I'm a bit clueless here.
Edit2: ActNorm on beginning is paramount. When you do this, all backbones will work like magic. No manual scaling needed.

Yes, I think you are right.
To match parameters: Which layers are you using from the resnet?
Per paper:

use first three block outputs (64, 64, 128) channels for resnet18
use RNVPCouplingBlock (or Glow - I think parameterwise it shouldn't matter)
use a total of 8 coupling blocks per layer output

the only free variable to play with in this case is the number of mid-channels for both subnets.
Unfortunately my GPU memory is too small to use first 3 image features. Please let me know if you can achieve any good results with above configuration.

I opened a Q in the Freia github
https://github.com/VLL-HD/FrEIA/issues/113

@maaft

I think the "first 3 blocks" for resnet18 means stride4x, 8x, and 16x, so the channel numbers should be (64, 128, 256). See table 6.
In fact, in section 6.1 and caption of Table7, the paper indicates the mid-channel numbers in subnets

I tried to move Permute and ActNorm from the end to the beginning of the block, as you suggested, but I didn't see significant improvement. Maybe there are some other issues in my code.

@maaft

I think the "first 3 blocks" for resnet18 means stride4x, 8x, and 16x, so the channel numbers should be (64, 128, 256). See table 6.

In fact, in section 6.1 and caption of Table7, the paper indicates the mid-channel numbers in subnets

I tried to move Permute and ActNorm from the end to the beginning of the block, as you suggested, but I didn't see significant improvement. Maybe there are some other issues in my code.

I am getting NaN when ActNorm is at the beginning of the block in a innSeq. Could you share an image?

I guess I could share my model later. No idea why you get nans.

Btw, for resNet the output of layer1, layer2 and layer3 are used

I guess I could share my model later. No idea why you get nans.
Maybe your data is already bad and contains nans? Are you normalizing your images?

For that, i am emulating the process:
x = torch.rand(16,3,256,256)
o=model(x)
But yes, I tested with the real images nornalized in a pytorch-standar way

The permutation of channels must be fixed during training. As @gathierry mentioned, it's necessary for normalizing flows.

@AlessioGalluccio just a small remark: For anomaly score calculation (global and pixelwise) you need to use p(z) and not z which you are currently using.
you can estimate logp(z) (and therefore p(z)) analogous to the pytorch implementation of CFlow AD.

For the anomaly score I apply
anomaly_score.append(t2np(torch.mean(z_grouped_temp ** 2, dim=(-2, -1))))
As it is used in DifferNet. Do you mean that I should add a /2 to it to be the same as the negative loglikelihood of a normal function?

I'll share my model, loss function and anomaly map generation tomorrow

@AlessioGalluccio In CFlow, there's an exponential converting logp to p as well. It's the same if there's only one feature level (such as DeiT and CaiT). But if there are 3 feature levels (resnet), it would be different since exp is performed before sum of three score maps in three levels. logp is in (-inf, 0] but p is in [0, 1],  sum(log_p) and sum(p) can be totally different

@mjack3 As for ActNorm, I simply moved the _permute of AllInOneBlock to the beginning of forward and removed the original ones. I don't think this is the root cause of NaN but it might somehow amplified your gradient.
def forward(self, x, c=[], rev=False, jac=True):
        '''See base class docstring'''
        if self.householder:
            self.w_perm = self._construct_householder_permutation()
            if rev or self.reverse_pre_permute:
                self.w_perm_inv = self.w_perm.transpose(0, 1).contiguous()
        # ==== ActNorm ====
        x0, global_scaling_jac = self._permute(x[0], rev=False)
        # ==== ActNorm end ====
        x1, x2 = torch.split(x0, self.splits, dim=1)

        if self.conditional:
            x1c = torch.cat([x1, *c], 1)
        else:
            x1c = x1

        if not rev:
            a1 = self.subnet(x1c)
            x2, j2 = self._affine(x2, a1)
        else:
            a1 = self.subnet(x1c)
            x2, j2 = self._affine(x2, a1, rev=True)

        log_jac_det = j2
        x_out = torch.cat((x1, x2), 1)

        # add the global scaling Jacobian to the total.
        # trick to get the total number of non-channel dimensions:
        # number of elements of the first channel of the first batch member
        n_pixels = x_out[0, :1].numel()
        log_jac_det += (-1)**rev * n_pixels * global_scaling_jac

        return (x_out,), log_jac_det

Here's my model that I'm using.
from FrEIA.framework.sequence_inn import SequenceINN
import FrEIA.modules as Fm
import FrEIA.framework as Ff
from typing import List, Tuple
from torch import Tensor
import torch.linalg
from typing import List, Optional, Tuple
import torch
import torch.nn.functional as F
import torch.nn as nn
import timm

_GCONST_ = -0.9189385332046727  # ln(sqrt(2*pi))

def z_to_logpz(z: torch.Tensor):
    return _GCONST_ - 0.5 * torch.mean(z**2, dim=1, keepdim=True)

def get_loss(p_u: torch.Tensor, logdet_j: torch.Tensor) -> torch.Tensor:
    loss = torch.mean(
        0.5*torch.sum(p_u ** 2, dim=(1, 2, 3)) - logdet_j)
    return loss

def mid_chan(dims_in: int, dims_out: int):
    return dims_in

def subnet_conv1(dims_in: int, dims_out: int):
    """1x1 conv subnetwork to predicts the affine coefficients.

    Args:
        dims_in (int): input dimensions
        dims_out (int): output dimensions

    Returns:
        nn.Sequential: Feed-forward subnetwork
    """
    kernel_size = 1
    return nn.Sequential(
        nn.Conv2d(dims_in, mid_chan(dims_in, dims_out),
                  kernel_size=kernel_size),
        nn.ReLU(),
        nn.Conv2d(mid_chan(dims_in, dims_out),
                  dims_out, kernel_size=kernel_size)
    )

def subnet_conv3(dims_in: int, dims_out: int):
    """3x3 conv subnetwork to predicts the affine coefficients.

    Args:
        dims_in (int): input dimensions
        dims_out (int): output dimensions

    Returns:
        nn.Sequential: Feed-forward subnetwork
    """
    kernel_size = 3
    return nn.Sequential(
        nn.Conv2d(dims_in, mid_chan(dims_in, dims_out),
                  kernel_size=kernel_size, padding=1),
        nn.ReLU(),
        nn.Conv2d(mid_chan(dims_in, dims_out), dims_out,
                  kernel_size=kernel_size, padding=1)
    )

class InvConv2d(Fm.base.InvertibleModule):
    def __init__(self, dims_in: List[Tuple[int]], dims_c: List[Tuple[int]] = None):
        super().__init__(dims_in, dims_c)
        channels = dims_in[0][0]
        weight = torch.randn(channels, channels)
        q, _ = torch.linalg.qr(weight, "reduced")
        weight = q.unsqueeze(2).unsqueeze(3)
        self.weight = nn.Parameter(weight, requires_grad=True)

    def forward(self, x_or_z, c=None, rev: bool = False, jac: bool = True) -> Tuple[Tuple[Tensor], Tensor]:
        _, _, height, width = x_or_z[0].shape

        out = F.conv2d(x_or_z[0], self.weight)
        logdet = height * width * \
            torch.slogdet(self.weight.squeeze().double())[1].float()

        return (out,), logdet

    def output_dims(self, input_dims):
        return input_dims

def fastflow_head(coupling_blocks: int, clamp_alpha: float, c: int, h: int, w: int) -> SequenceINN:
    """Create invertible decoder network.

    Args:
        condition_vector (int): length of the condition vector
        coupling_blocks (int): number of coupling blocks to build the decoder
        clamp_alpha (float): clamping value to avoid exploding values
        n_features (int): number of decoder features

    Returns:
        SequenceINN: decoder network block
    """
    coder = Ff.SequenceINN(c, h, w)
    clamp_activation = "ATAN"

    for _ in range(coupling_blocks // 2):
        coder.append(
            Fm.ActNorm
        )
        coder.append(
            Fm.PermuteRandom
        )
        coder.append(
            Fm.GLOWCouplingBlock,
            subnet_constructor=subnet_conv3,
            clamp=clamp_alpha,
            clamp_activation=clamp_activation,
        )
        coder.append(
            Fm.ActNorm
        )
        coder.append(
            Fm.PermuteRandom
        )
        coder.append(
            Fm.GLOWCouplingBlock,
            subnet_constructor=subnet_conv1,
            clamp=clamp_alpha,
            clamp_activation=clamp_activation,
        )
    return coder

class AnomalyMapGenerator(nn.Module):
    """Generate Anomaly Heatmap."""

    def __init__(self, image_size: Tuple[int, int]):
        super(AnomalyMapGenerator, self).__init__()
        self.image_size = image_size

    def forward(self, distributions: List[torch.Tensor]) -> torch.Tensor:
        score_map = torch.zeros(
            distributions[0].shape[0], 1, self.image_size[0], self.image_size[1], device=distributions[0].device)
        for dist in distributions:
            score_map += F.interpolate(torch.exp(dist - torch.max(dist.view(
                dist.shape[0], -1), dim=1)[0].view(-1, 1, 1, 1)), size=self.image_size, mode="bilinear", align_corners=True)

        score_map = score_map / len(distributions)
        # invert probability to get anomaly score
        anomaly_map = 1 - score_map

        return anomaly_map

class Scaler(nn.Module):
    def __init__(self, init_scale=0.1) -> None:
        super().__init__()
        self.alpha = nn.Parameter(init_scale*torch.ones(1), requires_grad=True)

    def forward(self, t: torch.Tensor) -> torch.Tensor:
        return self.alpha * t

class FastFlow(nn.Module):
    def __init__(self, image_size: Tuple[int, int]):
        super(FastFlow, self).__init__()
        self.fe: timm.models.resnet.ResNet = timm.create_model(
            "resnet18", pretrained=True, features_only=True
        )

        for param in self.fe.parameters():
            param.requires_grad = False

        # # resnet 18
        feature_dims = [
            # 64,
            128,
            256
        ]
        image_sizes = [
            # (image_size[0] // (2**2), image_size[1] // (2**2)),
            (image_size[0] // (2**3), image_size[1] // (2**3)),
            (image_size[0] // (2**4), image_size[1] // (2**4)),
        ]

        self.nf = nn.ModuleList(
            [
                fastflow_head(
                    8, 1.2, dim, image_size[0], image_size[1])
                for i, (dim, image_size) in enumerate(zip(feature_dims, image_sizes))
            ]
        )

        nf_params = sum(p.numel()
                        for p in self.nf.parameters() if p.requires_grad)
        print("Params: ", nf_params / 1000000.)

        def initialize_weights(m):
            if isinstance(m, nn.Conv2d):
                nn.init.xavier_normal_(
                    m.weight.data, gain=nn.init.calculate_gain('relu'))
                if m.bias is not None:
                    nn.init.constant_(m.bias.data, 0)

        self.nf.apply(initialize_weights)

        self.anomaly_generator = AnomalyMapGenerator(image_size)

    def forward(self, image: torch.Tensor):
        with torch.no_grad():
            features = self.fe.forward(image)[2:-1]

        anomaly_map: Optional[torch.Tensor] = None
        distributions: List[torch.Tensor] = []
        loss = torch.zeros(1, device=image.device)
        for i, f in enumerate(features):
            z: torch.Tensor
            log_det_jac: torch.Tensor

            z, log_det_jac = self.nf[i](f.detach())
            loss += get_loss(z, log_det_jac)

            with torch.no_grad():
                logpz = z_to_logpz(z)
                distributions.append(logpz.detach())

        with torch.no_grad():
            anomaly_map = self.anomaly_generator.forward(
                distributions).detach()

        return anomaly_map, loss

I opened my code.
https://github.com/mjack3/FastFlow-AD
There i opened a Q&A section where discuss

I'm getting a 404 - sure that its open?

I emailed to my corporation to open the code

This is what i am getting. Just the image auroc is done.

@mjack3 It seems like produce a good result at all. And do you have any idea why the Capsule and Toothbrush classes perform not good with 3 FE layers output flow model ?
Like in general case, screw is hard to perform well since it's align and rotate problems. And also we know transistor's anomaly part is quite different too, so these classes still under our expectation.

I think it could be for something that paper sais:
It should be noted that some categories are not suitable for violent data augmentation.
Maybe I'm doing a wrong DA there.

@mjack3 just turn all augmentation off. The paper states only a sub 1% change in performance. And personally, I noticed strong degradation, prediction artifacts and NaN issues when using DA.

Thanks for you contribution @maaft
I tested the timm library to extract the features and eventually I stopped to achieve NaN tensors. But using the https://pytorch.org/vision/stable/feature_extraction.html function of Pytorch, NaN comes back

@maaft what's the base behind the z_to_logpz  method?

@mjack3 CFlow-AD Eq. 8

@mjack3  CFlow-AD Eq. 8
I realise that I forgot to add logdet. In praxis it doesn't make a difference since logdet will be positive and we subtract the maximum of the logp map to normalize p to be in [0, 1].

@mjack3 while we're waiting for your company to release your code, maybe you could comment if you did anything fundamentally different to my model/loss above

Sure. Give me a second

My model:
My fastflow head is based on AllInOne:
def fastflow_head(dims: tuple) -> Ff.SequenceINN:

    inn = Ff.SequenceINN(*dims)
    for k in range(4):
        inn.append(Fm.AllInOneBlock, subnet_constructor=subnet_conv_3x3, permute_soft=True)
        inn.append(Fm.AllInOneBlock, subnet_constructor=subnet_conv_1x1, permute_soft=True)

    return inn

The loss function I am using in training is just the same as you (based on CS-FLOW (Not CFLOW-AD)
loss = torch.mean(0.5 * torch.sum(z ** 2, dim=(1, 2, 3)) - log_j) / z.shape[1]
and then I am trying to build the anomalies scores directly from Z, similar as CS-FLOW does. Reading the paper, we can see that authors can build a estimate map of anomalies but because of NF is built with dense layers instead convolution, it's not great for anomaly location.

Thank you very much.
Just some notes/questions:

when using 8x AllInOneBlock you're only using half the amount of convolutions as in they state in the paper (shouldn't matter because you get nice results anyway)
I tried the loss functtion from CFLOW-AD but it didn't work at all
could you elaborate on your subnet config? How many mid-channels are you using?
are you using all 3 layers from resnet18?
what optimizer config are you using?

I suposse you mean "when using 8x AllInOneBlock". The figure 2.b  says one flow step for our FastFlow so in the iteration 1 of this loop you will 2 step flow (conv3+conv1). At the end they are 8 step flows
I think is normal becausee the loss of CSFLOW is the equivalent for minimizing.
I'm using x2 input channel in the middle.
I'm using the 3 layers from wide_resnet50.
I 'm using the same optimizer than fastflow paper

according to fig. 2 "one flow step" is represented by double-sided blocks (which can be GLOW or RVNP). These blocks have two subnets and work on both halfs of the input. Because the paper says "s(y_a) and b(y_a) are outputs of two neural networks", it must be the RVNP-Block, because the GLOWBlock uses one network to predict both. In practice it doesn't matter which you use. On the other hand, the AllInOneBlock that you are using is only one-sided. That means it only represents the first half of fig.2. Therefore you need to double your number of AllInOneBlocks to match the number of conv-layers used in the paper.

Thanks for the other points! :)

humm thanks for your point. I haven't so much experience in normalizing flow so, probably you are in true and I'm wrong using AllInOneBlock

I don't think that it matters in the end. Your performance is great.
By the way, I think that realNVP introduced the double sided blocks only to make sure that all channels are trained. They did this before RandomPermute was becoming a thing. Both are equivalent IMHO. Just the number of blocks (when using FrEIA) needs to be adapted to match a certain number of conv layers.

I don't think so. The figure says "flow step" while eq.3 only says "step" and takes reference to Dhin et al.

I dont get the result of the paper :(

I managed to achieve comparable performance in a tricky way. I add LayerNorm layers before sending each feature map to NFs.
It is "tricky" since the usage is different for different backbones and that's the only way works for me:

resnet18 and wide-resnet-50: use trainable LayerNorm
CaiT and DeiT: use the final norm from the pre-trained model and fix it's affine parameters

I opened my code as well https://github.com/gathierry/FastFlow

@gathierry wow, LayerNorm really boosts the performance enormously. Thank you very much for that and for opening your code! :)

Does anyone have a clue how we could measure model performance without having labeled ground-truth test data?

Hello guys. Someone know what does mean "w/o" and "w" in the first column of this table?

@mjack3  w/o = without         w = with

@gathierry I 'm wondering about why you do this in lanes https://github.com/gathierry/FastFlow/blob/master/fastflow.py#L148-L151 instead of compute the positive log

@mjack3 why I'm doing exponential? Because the values in different levels should be converted to a probability before merged. I want to project the output to its probability in gaussian distribution.
And I guess this can also normalize values in different levels to the same range

@gathierry is more about the -torch.mean

@mjack3 oh about that.

-torch.mean:  a gaussian should be like torch.exp(-0.5*torch.sum(output**2)). However, sum make the value tooooo small and it's not comparable between different levels as their channel numbers are different. I see CFlow divided the module of output**2 by channel number. So I do the same, which make me replace sum with mean

the interpolation of the negative: the result is the probability and anomaly score should be the opposite. Resolution of different levels are not identical so we have to resize them before merging.

@gathierry thanks for your answer.
Why are you using that loss function instead something like this?

link
Am I missing something?

@mjack3 I am using the same loss function
https://github.com/gathierry/FastFlow/blob/d275b79d47d6e272115d45fd7fc0f29cca0f5107/fastflow.py#L139
What you mentioned before
https://github.com/gathierry/FastFlow/blob/master/fastflow.py#L148-L151
was just for inference.

@gathierry From your implementation, your AUROC is refer to seg performance right?

@Howeng98 yes, the pixel level AUROC

Does anyone have a clue how we could measure model performance without having labeled ground-truth test data?

Hi @maaft , IMO, if you are trying to measure model performance in accuracy/percentage (%).
However, if you are using distance measurements (kNN/aNN/euclidean etc), then it should be possible, with the caveat that you will still need to verify the measurements (manual label by human eyes) in the end in order to map distance mesurement -> accuracy.
Hope this helps :).
Note: Very cool progress and results for you guys, considering how little-informed the paper is.

This is great discussion. I also performed some experiments using wide_resnet_50_2.

I wrote from scratch the various modules in fastflow myself such as actnorm, affine coupling and channel permute, split and merge and combine them in a module fastflow_head (following the framework here). I didn't use the FrEIA framework just because I think the framework I follow is more transparent for me.

Model details. I largely follow the paper's setting by using a flow step(called fastflow_head in my code) consisting of actnorm -> channel random shuffling -> two affine couplings acting on different halves of channels with Conv-RELU-Conv subnets -> channel merge. There are things that are not clearly specified in the paper and let me try to make it explicit here.

I use 3 feature maps from 3 layers of wide_resnet_50_2. They have sizes (256, 64, 64), (512,32,32) and (1024, 16,16) respectively. I followed the paper to build 3 fastflow models using these 3 feature maps
hidden channels in the Conv-RELU-Conv block: I use the same number as the number of input channels to the first Conv, that is, half the total channels in the original input from feature extractor. For wide_resnet_50_2, the hidden channels are respectively 128, 256, 512. I think there is some ambuiguity in the original paper, they say the number of input and output channels of the subnet are the same, but that's not the case.
For actnorm, I initialize the log-scale and bias with all zeros. This is different from the intialization in the GLOW paper. But I think there is no reason to initialize with mean and std of the activation in our case. In experiment, I found the trivial initialization makes it easier to converge.
For optimizer, I follow the paper.
For loss, I use the same loss as the CFlow paper
For the number of flow steps, I have 4 flow steps with alternating 3x3 convs and 1x1 convs instead of 8 flow steps as specified in the paper. I found this makes a very big difference. I observed very unstable training for 8 flow steps and sometimes NaN losses or gradients. 4 flow steps is easier to train and it matches the parameter number specified in the paper.
For number of epochs, I use 500 epochs. I also observed that the model stabilize or reach the best performance in a few steps. But in my own project(where there is no labelled groundtruth for test), I use some good object as valid dataset and observe that the loss on valid data reach minimum at around 250 epochs.
For final anomaly map, I average on anomaly map from 3 fastflow models. For each fastflow model, I took the same method as in gathierry's approach There can be understood as taking product of the probability along channels.
Under the above specifications, the number of parameters is 41.3M. Using 8 flow steps doubles this number to 82.6M. Doubling the number of hidden channels also doubles this number to 82.6.

I did not experiment on all category of the MvTec dataset. I only experiment on bottle category and see that it's pixel-wise AUROC matches the result from paper in just a few steps. Maybe other categories require more training epochs? That I'm not sure

Guys, what's mean this in the figure 3?
What is y'? the affine?

Guys, what's mean this in the figure 3? What is y' ?

y' is the output of the coupling flow, while y is the input. Coupling flows work by splitting the input and keeping a set unchanged ("a" in this example) and modifying the other ("b" in this example). This is needed to have invertibility and easy Jacobian determinant to compute, since the Jacobian matrix becomes triangular.

I am very grateful for your GitHub source.
However, I have a few questions regarding the use of the model. As the epochs progress, the loss value appears as follows, and I wonder if this is correct.
Secondly, I placed the mvtec-ad data file in the path where the source files are gathered, but the following error occurs. Could you tell me what the problem is?

I am very grateful for your GitHub source. However, I have a few questions regarding the use of the model. As the epochs progress, the loss value appears as follows, and I wonder if this is correct. Secondly, I placed the mvtec-ad data file in the path where the source files are gathered, but the following error occurs. Could you tell me what the problem is?

maybe you can check there,
'dataset.py'  Line 93：
target = Image.open(
# image_file.replace("/test/", "/ground_truth/").replace(
image_file.replace("test", "ground_truth").replace(
".png", "_mask.png"
)

Hello, thanks for the repo, I'm currently studying about the fastflow model and tried running the code, but it turns out that there's an error during the eval step. I wonder why did that happen. Could you help with me?


Fix default app redirect issue.

Reference Issues or PRs

What does this implement/fix?
Put a x in the boxes that apply

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds a feature)
[ ] Breaking change (fix or feature that would cause existing features not to work as expected)
[ ] Documentation Update
[ ] Code style update (formatting, renaming)
[ ] Refactoring (no functional changes, no API changes)
[ ] Build related changes
[ ] Other (please describe):

Testing

[x] Did you test the pull request locally?
[ ] Did you add new tests?

Documentation

Access-centered content checklist
Text styling

[ ] The content is written with plain language (where relevant).
[ ] If there are headers, they use the proper header tags (with only one level-one header: H1 or # in markdown).
[ ] All links describe where they link to (for example, check the Nebari website).
[ ] This content adheres to the Nebari style guides.

Non-text content

[ ] All content is represented as text (for example, images need alt text, and videos need captions or descriptive transcripts).
[ ] If there are emojis, there are not more than three in a row.
[ ] Don't use flashing GIFs or videos.
[ ] If the content were to be read as plain text, it still makes sense, and no information is missing.

Any other comments?

@jbouder uhm.. I am still encountering the issue, though now it's more stable, and I don't see any blank pages, but the direct profile creation still happens. While in admin, I also glanced at the admin page and attempted to check the Spawn page from there as well, but it automatically spun the instance...

I have a feeling something amiss is going on with the /hub/spawn route, since that is supposed to look like this:

I am also seeing this in the logs:
INFO:     <IP_ADDRESS>:0 - "GET /services/japps/user HTTP/1.1" 200 OK
[D 2024-12-11 12:15:58.545 JupyterHub scopes:1010] Checking access to /hub/spawn/admin via scope servers!server=admin/
[D 2024-12-11 12:15:58.545 JupyterHub pages:216] Triggering spawn with default options for admin
[D 2024-12-11 12:15:58.545 JupyterHub base:1095] Initiating spawn for admin
[D 2024-12-11 12:15:58.545 JupyterHub base:1099] 0/100 concurrent spawns
[D 2024-12-11 12:15:58.545 JupyterHub base:1104] 0 active servers

Yeah, I think this fixes the UI bug...however I'm not sure how to get JupyterHub to load the server options page. Based on the logs you provided, I wonder if the admin account is missing some scope.

Yeah, I think this fixes the UI bug...however I'm not sure how to get JupyterHub to load the server options page. Based on the logs you provided, I wonder if the admin account is missing some scope.

Ineed, I think I am running this wrong, the profiles list targets the KubeSpawne object, which should only be available on a K8s environment. I will test this on a local nebari since this might be why I am not getting the profiles to show up

Working!

I will just do another pass, to make sure its not a false positive. If all works this will be ready to go


Fax machine print from text file doesn't appear to be logged

Feature introduced in #23262
Log should include:

Person printing the file, labeled as "actor"
Entity being printed, (the paper) labeled as "subject"
Device used to print the paper, labeled as "tool"
Text on the paper at the end of the log message with no trailing text/punctuation

Log type should be chat and severity should be low.
See how paper system logs writing, but imagine the paper is labeled "subject" instead of "entity":
https://github.com/space-wizards/space-station-14/blob/ba4188a8b4f1e268c193252e1151b548b2db6a0b/Content.Server/Paper/PaperSystem.cs#L161-L162

This also needs to include copying which has never been logged.

Ok, i do.


Creating or Deleting a Component causes issues on route.js

When I create or delete component with url path in rekit-studio, the below comment lines on the top in "route.js" flie is doubled.
// This is the JSON way to define React Router rules in a Rekit app.
// Learn more from: http://rekit.js.org/docs/routing.html

Thanks @graphy21 for reporting this! It's now fixed by https://github.com/supnate/rekit/commit/2f0ebf803e0b45580bf4bafbbdafeff818df96c8 .
Update to<EMAIL_ADDRESS>in your project to fix the issue.


Doc update

Any idea on when this will be GA? The feature has been in preview for almost a year.
Are there plans to allow us to protect Azure file shares in storage accounts that have Virtual Networks or Firewall enabled?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 61c0f8e6-5a57-58aa-ecd6-2bf83809c44d
Version Independent ID: ee8b8f60-bffd-e6ec-5f21-bfd3208fc511
Content: Back up Azure File Shares
Content Source: articles/backup/backup-azure-files.md
Service: backup
GitHub Login: @rayne-wiselman
Microsoft Alias: raynew

@jzuchora Thanks for the comment! We will check on this and get an update for you soon.

@jzuchora The service will be GA later this year. But we do not have publicly shareable ETA. Support for storage account that have virtual network/Firewall is coming soon. Stay tuned  for the updates and keep a watch on this page.

@jzuchora We will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly continue the discussion and we will reopen the issue.


Carol - a menu bar lyrics app

Hi,
I have recently developed this app - Carol having minimal and beautiful design which shows lyrics on demand. Would love to have it added to the list.
Thanks.

Hi @AnaghSharma! Nice app! You can just create Pull Request.

@serhii-londar So I went through the CONTRIBUTING.md and checked the format for adding an app. The app is developed in C# which is not listed in the languages. Should I add it?

@AnaghSharma Yes. Please add icon image to icon folder and update readme to support it.

Added in c7ac462ce6a951bbc6921e14a0d604f46f62d251