Changes BytesMut not to rely on Vec + Arc

This changes the internal BytesMut representation to allocate memory on
its own, and not to use Arc + Vec under the hood.
Doing this allows to have a single representation which is shareable (by
being able to modify an embedded refcount) with zero-cost right from
the start. No representation changes have to be performed over the
lifetime of BytesMut. Therefore BytesMut::freeze() is now also free.
The change also opens the door for adding support for buffer pools or
custom allocators, since an allocator is directly called. Adding this
support would mostly mean having to pass an allocator to BytesMut
which is used called instead of a fixed function. This would however be
a different change.
Fixes #401

cc/ @seanmonstar who did most of this work most recently.

I'm now rather happy with this, so I removed the WIP.
I added some more documentation around the memory allocation strategy - based on my understanding of previous versions - and questions on what we should do there.
I'm somewhat inclined to drop the behavior of taking capacity that an old backing storage had into account for reservations - it just seems like a footgun leading to high memory usage in case people split BytesMut and work on them later in individually.

Not sure what is going on with Loom. I think it doesn't like my use of alloc::alloc::alloc (direct use of the global allocator)- but don't know how to fix it.

cargo bench comparison:
name                      old ns/iter      new ns/iter      diff ns/iter   diff %  speedup
 alloc_big                 52               53                          1    1.92%   x 0.98
 alloc_mid                 53               51                         -2   -3.77%   x 1.04
 alloc_small               53,018           51,740                 -1,278   -2.41%   x 1.02
 alloc_write_split_to_mid  104              67                        -37  -35.58%   x 1.55
 bytes_mut_extend          568 (225 MB/s)   538 (237 MB/s)            -30   -5.28%   x 1.06
 clone_frozen              9,711            9,785                      74    0.76%   x 0.99
 deref_shared              242              241                        -1   -0.41%   x 1.00
 deref_two                 237              235                        -2   -0.84%   x 1.01
 deref_unique              239              240                         1    0.42%   x 1.00
 deref_unique_unroll       257              258                         1    0.39%   x 1.00
 drain_write_drain         298              228                       -70  -23.49%   x 1.31
 fmt_write                 14 (2642 MB/s)   14 (2642 MB/s)              0    0.00%   x 1.00
 put_slice_bytes_mut       15 (8533 MB/s)   14 (9142 MB/s)             -1   -6.67%   x 1.07
 put_slice_vec             16 (8000 MB/s)   15 (8533 MB/s)             -1   -6.25%   x 1.07
 put_slice_vec_extend      10 (12800 MB/s)  10 (12800 MB/s)             0    0.00%   x 1.00
 put_u8_bytes_mut          508 (251 MB/s)   454 (281 MB/s)            -54  -10.63%   x 1.12
 put_u8_vec                538 (237 MB/s)   513 (249 MB/s)            -25   -4.65%   x 1.05
 put_u8_vec_push           128 (1000 MB/s)  127 (1007 MB/s)            -1   -0.78%   x 1.01

It looks like you are inlining the capacity with the data? Given that buffers are often allocated to be page length, would this not result in odd allocation patterns? The size would often be size_of::<Shared>() + PAGE.

Looking through this, I believe this should be able to land w/o a breaking change. I don't think we need to block 0.3 on this.

It looks like you are inlining the capacity with the data? Given that buffers are often allocated to be page length, would this not result in odd allocation patterns? The size would often be size_of::<Shared>() + PAGE.

Are you aware about any applications which do that? I don't think you can currently rely on that, since your system memory allocator might also not do the same - e.g. it might also need to store it's metadata inside the allocation header and therefore take size away from it. But it's true that buffer sizes which are multiples of 4k are often programmers favorites.
What we could do is maybe rounding up allocations to page size, so that the capacity isn't lost? But that might require adding a with_exact_capacity method for use-cases which don't want that.

regarding

It looks like you are inlining the capacity with the data? Given that buffers are often allocated to be page length, would this not result in odd allocation patterns? The size would often be size_of::() + PAGE.

I confirmed that e.g. mimalloc will optimize fixed sized chunks, and Shared would might mean getting pushed to the next chunk size.
Based on this I see 2 ways forward for this:

Don't care about it in the normal case, and add more optimized APIs for people that care about the differences.E.g. BytesMut::with_exact_capacity(size) would return exactly what the user requires (even it if it is not optimal), BytesMut::with_optimized_capacity(size) could either round up (to next multiple of 2) or down (subtract - size_of::<Shared>).
Add another pointer to Bytes itself which tracks the shared state separately from the data. The benefit of this is that this could also eliminate the special casing for Vec in Bytes, since the field can be used to store the capacity. The downsides are however:

2 allocations for BytesMut instead of.
2 times the pointer chasing on resizes/frees/etc.

I guess for the use-case of "have a small serialization buffer and continuously add stuff" approach 2) will end up slower than 1) (and this CR). However it's more deterministic for managing large chunks of data.

regarding

It looks like you are inlining the capacity with the data? Given that buffers are often allocated to be page length, would this not result in odd allocation patterns? The size would often be size_of::() + PAGE.

I confirmed that e.g. mimalloc will optimize fixed sized chunks, and Shared would might mean getting pushed to the next chunk size.
Based on this I see 2 ways forward for this:

Don't care about it in the normal case, and add more optimized APIs for people that care about the differences.E.g. BytesMut::with_exact_capacity(size) would return exactly what the user requires (even it if it is not optimal), BytesMut::with_optimized_capacity(size) could either round up (to next multiple of 2) or down (subtract - size_of::<Shared>).
Add another pointer to Bytes itself which tracks the shared state separately from the data. The benefit of this is that this could also eliminate the special casing for Vec in Bytes, since the field can be used to store the capacity. The downsides are however:

2 allocations for BytesMut instead of.
2 times the pointer chasing on resizes/frees/etc.

I guess for the use-case of "have a small serialization buffer and continuously add stuff" approach 2) will end up slower than 1) (and this CR). However it's more deterministic for managing large chunks of data.

Do we know why we can't entirely hide the the Arc / Vec implementation behind the vtable? IIRC it is due to wanting to avoid an allocation when converting Vec and needing to track the pointer to the head of the memory as well as the original capacity? Is there another reason that I am forgetting?

Do we know why we can't entirely hide the the Arc / Vec implementation behind the vtable? IIRC it is due to wanting to avoid an allocation when converting Vec and needing to track the pointer to the head of the memory as well as the original capacity? Is there another reason that I am forgetting?

Would this also allow for free Bytes::try_mut()? #368


Issue with conditional-gan for tensorflow version

Hey guys, I'm using the conditional-gan code for tensorflow version to generate garment drawings, the training images is RGB png file, labels are vectors like this [1,2,3,4,...,2]，I just changed the feeding training data from mnist to my data, but the generated images are shapeless, I don't know why, someone helps me? thanks!!
The attached is the modified code and images generated.
code+generated images.zip

BTW, the training loss for the generator is always very large(>20), while for the discriminator is very small(< 0.001), is there something wrong with the training system?


Prepare v0.16.0

Pull Request Checklist:

[ ] This PR addresses an already opened issue (for bug fixes / features)

This PR fixes #xyz

[ ] (If applicable) Documentation has been added / updated (for bug fixes / features).
[ ] (If applicable) Tests have been added.
[x] CHANGELOG.rst has been updated (with summary of main changes).

[ ] Link to issue (:issue:number) and pull request (:pull:number) has been added.

What kind of change does this PR introduce?

Prepares the next minor release (v0.16.0)

Does this PR introduce a breaking change?
No.

coverage: 81.343%. remained the same
when pulling 00949106e4b3b44adb41d4b63cc43e4f251399fb on prepare-v0160
into b39f3b279cd14fbd4a05b7065e55715cef9abb98 on master.


Strangely, LanguageBind_Image preprocessor_config.json is missing while running demo

Traceback (most recent call last):
File "/data/miniconda3/envs/bind/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/data/miniconda3/envs/bind/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/data/Projects/FactCheck/MultiModel/LVLMs/Video-LLaVA/llava/serve/tmp.py", line 57, in
main()
File "/data/Projects/FactCheck/MultiModel/LVLMs/Video-LLaVA/llava/serve/tmp.py", line 20, in main
tokenizer, model, processor, context_len = load_pretrained_model(model_path, None, model_name, load_8bit, load_4bit, device=device)
File "/data/Projects/FactCheck/MultiModel/LVLMs/Video-LLaVA/llava/model/builder.py", line 154, in load_pretrained_model
image_tower.load_model()
File "/data/Projects/FactCheck/MultiModel/LVLMs/Video-LLaVA/llava/model/multimodal_encoder/clip_encoder.py", line 23, in load_model
self.image_processor = CLIPImageProcessor.from_pretrained(self.vision_tower_name)
File "/data/miniconda3/envs/bind/lib/python3.10/site-packages/transformers/image_processing_utils.py", line 165, in from_pretrained
image_processor_dict, kwargs = cls.get_image_processor_dict(pretrained_model_name_or_path, **kwargs)
File "/data/miniconda3/envs/bind/lib/python3.10/site-packages/transformers/image_processing_utils.py", line 269, in get_image_processor_dict
resolved_image_processor_file = cached_file(
File "/data/miniconda3/envs/bind/lib/python3.10/site-packages/transformers/utils/hub.py", line 388, in cached_file
raise EnvironmentError(
OSError: LanguageBind/LanguageBind_Image does not appear to have a file named preprocessor_config.json. Checkout 'https://huggingface.co/LanguageBind/LanguageBind_Image/None' for available files.

yes... i failed on this

@LinB203 ,can you resolve it ?

Sorry for the late reply. I have uploaded preprocessor_config.json. Feel free to let me know if this works.

Thank you for the update, but now there is a new problem. It seems that the hidden dimension of mm_video_tower used by video-llava-7B does not match the hidden dimension of video-llava-7B. Are you sure that LanguageBind_Video_merge on hugging face is the correct version?
...
- This IS NOT expected if you are initializing LlavaLlamaForCausalLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

['Video', 'Image']
You are using a model of type LanguageBindImage to instantiate a model of type clip_vision_model. This is not supported for all configurations of models and can yield errors.
Traceback (most recent call last):
  File "/data/miniconda3/envs/bind/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/data/miniconda3/envs/bind/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/data/Projects/FactCheck/MultiModel/LVLMs/Video-LLaVA/llava/serve/cli.py", line 144, in <module>
    main(args)
  File "/data/Projects/FactCheck/MultiModel/LVLMs/Video-LLaVA/llava/serve/cli.py", line 32, in main
    tokenizer, model, processor, context_len = load_pretrained_model(args.model_path, args.model_base, model_name,
  File "/data/Projects/FactCheck/MultiModel/LVLMs/Video-LLaVA/llava/model/builder.py", line 154, in load_pretrained_model
    image_tower.load_model()
  File "/data/Projects/FactCheck/MultiModel/LVLMs/Video-LLaVA/llava/model/multimodal_encoder/clip_encoder.py", line 24, in load_model
    self.vision_tower = CLIPVisionModel.from_pretrained(self.vision_tower_name)
  File "/data/miniconda3/envs/bind/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2881, in from_pretrained
    ) = cls._load_pretrained_model(
  File "/data/miniconda3/envs/bind/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3278, in _load_pretrained_model
    raise RuntimeError(f"Error(s) in loading state_dict for {model.__class__.__name__}:\n\t{error_msg}")
RuntimeError: Error(s) in loading state_dict for CLIPVisionModel:
        size mismatch for vision_model.embeddings.class_embedding: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current model is torch.Size([768]).
        size mismatch for vision_model.embeddings.position_ids: copying a param with shape torch.Size([1, 257]) from checkpoint, the shape in current model is torch.Size([1, 50]).
        size mismatch for vision_model.embeddings.patch_embedding.weight: copying a param with shape torch.Size([1024, 3, 14, 14]) from checkpoint, the shape in current model is torch.Size([768, 3, 32, 32]).
        size mismatch for vision_model.embeddings.position_embedding.weight: copying a param with shape torch.Size([257, 1024]) from checkpoint, the shape in current model is torch.Size([50, 768]).
        size mismatch for vision_model.pre_layrnorm.weight: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current model is torch.Size([768]).
        size mismatch for vision_model.pre_layrnorm.bias: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current model is torch.Size([768]).
        size mismatch for vision_model.encoder.layers.0.self_attn.k_proj.weight: copying a param with shape torch.Size([1024, 1024]) from checkpoint, the shape in current model is torch.Size([768, 768]).
        size mismatch for vision_model.encoder.layers.0.self_attn.k_proj.bias: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current model is torch.Size([768]).
        size mismatch for vision_model.encoder.layers.0.self_attn.v_proj.weight: copying a param with shape torch.Size([1024, 1024]) from checkpoint, the shape in current model is torch.Size([768, 768]).
        size mismatch for vision_model.encoder.layers.0.self_attn.v_proj.bias: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current model is torch.Size([768]).
        size mismatch for vision_model.encoder.layers.0.self_attn.q_proj.weight: copying a param with shape torch.Size([1024, 1024]) from checkpoint, the shape in current model is torch.Size([768, 768]).
        size mismatch for vision_model.encoder.layers.0.self_attn.q_proj.bias: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current model is torch.Size([768]).
        size mismatch for vision_model.encoder.layers.0.self_attn.out_proj.weight: copying a param with shape torch.Size([1024, 1024]) from checkpoint, the shape in current model is torch.Size([768, 768]).
        size mismatch for vision_model.encoder.layers.0.self_attn.out_proj.bias: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current model is torch.Size([768]).
        size mismatch for vision_model.encoder.layers.0.layer_norm1.weight: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current model is torch.Size([768]).
...

Could you share your code?
The merge version is not recommended in our api.

I just ran a test demo:
python -m llava.serve.cli --model-path "LanguageBind/Video-LLaVA-7B" --image-file "./assets/main.jpg" --load-4bit
Then change "mm_image_tower" and "mm_video_tower" in LanguageBind\Video-LLaVA-7B\config.json to the local address

I just ran a test demo:
python -m llava.serve.cli --model-path "LanguageBind/Video-LLaVA-7B" --image-file "./assets/main.jpg" --load-4bit
Then change "mm_image_tower" and "mm_video_tower" in LanguageBind\Video-LLaVA-7B\config.json to the local address

If you want to load based on local paths, make sure your local path is correct, here's a sample I loaded with my local path. It works fine.

Then remove some of the restrictions.

Thank you very much. It works.


Rework surface creation for less complex surfaces

Currently, we use the OpenCASCADE curve and surface interpolation as a blackbox. This currently leads to surfaces with too many knots and strange knot vectors.
We should rework this to get less comples surfaces in the end. There are different approaches:

Resample each airfoil by a constant number of points and the interpolate those.
Use fixed parameters in the curve interpolation routines.
Use knot removal after surface creation to reduce complexity.

This is a duplicate of #668, I will close this in favor of the newer issue.


~Rule will delete chainedRule

Came across this memory leak when reloading nginx with hundreds of rule chains

Hi @dkamen,
Thank you for the patch.

Merged.


Add multiple fitness variables

Instead of each genome having a fitness and being sorted by it, you should be able to add as many fitness variables as you want, sorted by priority.
For example, the first fitness could be "survival time", while the higher prioritised fitness could be "number of victories". This means that we first sort by the number of victories. Then we sort by survival time. This means that a genome with more victories and a low survival time is better than a genome with a high survival time and no victories.

If we allow the creation of unlimited fitness variables with priority, then we could add the topology cost as a low priority (priority 0) instead of subtracting it from the actual fitness.

What can be used as an alternative would be to set the average computing time as priority 0. In principle, both would fulfil the same goal. However, from a human perspective, it is much easier to set a desired reaction time as a termination condition than to estimate how small the network should be. Of course, this has the disadvantage that the response time depends on the computer. So, if a response time of 30ms is desired, it could be 50ms or more on another computer.
The user will probably simply be able to specify which of the two methods is to be used via the config.
The user can then theoretically switch this feature off completely, at the risk of having an endlessly growing network without any increase in performance. Except for a visual representation or to observe how yane grows, it should offer no added value.
The only problem I see with this approach is if, for example, the highest priority is survival time yane can never really shrink his net. This is because the lower-priority fitness variables only become relevant when the higher-priority variables of the genomes are identical. This means that fitness variables with decimal places can also become problematic.
This means that, in addition to prioritization, the user should also have the option of using the classic method with just one fitness variable.
However, you could also consider setting a threshold for the similarity of the fitness values. In other words, genome A with a fitness of 1400 and genome B with 1350 are practically identical in this sense. However, a difference of 3.5% is relatively large. You could therefore set the threshold to 0.5%, for example. This means that if Genome B were 1393, the difference would be exactly 0.5%. Of course, this brings the risk that progress is hindered, especially with slowly growing fitness, but it also offers the possibility of using lower prioritized values. You just have to think about how exactly you want to group them. However, the idea of the threshold will probably be discarded and the user should ideally round the fitness values in case of doubt. Otherwise, simply use one fitness variable and follow the classic principle: fitness = survival time0.4 + reaction time0.1 + number of victories*0.5


FIX-#6396: set __factory to None in case of any problems during initialization

What do these changes do?

[x] first commit message and PR title follow format outlined here

NOTE:  If you edit the PR title to match this format, you need to add another commit (even if it's empty) or amend your last commit for the CI job that checks the PR title to pick up the new PR title.

[x] passes flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py
[x] passes black --check modin/ asv_bench/benchmarks scripts/doc_checker.py
[x] signed commit with git commit -s
[x] Resolves #6396
[x] tests added and passing
[x] module layout described at docs/development/architecture.rst is up-to-date

Could you please elaborate on what exact use case are you improving here?

get_factory initializes the __factory if it has not been initialized before.
https://github.com/modin-project/modin/blob/457c926aa41708a89c882e889896b748578cddf8/modin/core/execution/dispatching/factories/dispatcher.py#L110-L118
_update_factory assigns a factory to cls.__factory in any case even if factory.prepare() raise an Exception. As a result, all subsequent calls will fall to a partially initialized factory, which can generate non-obvious errors.

Thanks @vnlitvinov @YarShev!


AntiVirus won't let FU.WhyAmIBlocked run

Here is my PowerShell output
PowerShell 7.2.6 Copyright (c) Microsoft Corporation.
 https://aka.ms/powershell
Type 'help' to get help.
PS C:\Users\richs> Install-Module FU.WhyAmIBlocked -Scope CurrentUser PS C:\Users\richs> Import-Module FU.WhyAmIBlocked
Write-Error: Failed to import function C:\Users\richs\OneDrive\Documents\PowerShell\Modules\FU.WhyAmIBlocked\<IP_ADDRESS>\Public\ConvertFrom-BinToXML.ps1: At C:\Users\richs\OneDrive\Documents\PowerShell\Modules\FU.WhyAmIBlocked\<IP_ADDRESS>\Public\ConvertFrom-BinToXML.ps1:1 char:1 + <#
+ ~~ This script contains malicious content and has been blocked by your antivirus software.
Write-Error: Failed to import function C:\Users\richs\OneDrive\Documents\PowerShell\Modules\FU.WhyAmIBlocked\<IP_ADDRESS>\Public\Expand-SDB.ps1: At C:\Users\richs\OneDrive\Documents\PowerShell\Modules\FU.WhyAmIBlocked\<IP_ADDRESS>\Public\Expand-SDB.ps1:1 char:1 +
This script contains malicious content and has been blocked by your antivirus software. PS C:\Users\richs>`
Here is the report from my AntiVirus software:
`PowerShell tried to load a malicious resource detected as Heur.BZC.ZFV.Linx.51.36DA30A5 and was blocked. Your device is safe.
Not sure how to proceed. Please advise

I haven’t heard of others seeing this issue. Check the code for yourself and you can see what it’s doing and determine whether it’s a false positive or not.


"Corund" is listed in the drop-down menu

Description:
"Corund" is listed in the drop-down menu alongside its corresponding county and commune, according to the law
Precondition:
The website is up an running.

Step 1
Type "Corund" in the search bar.

Expected results
The location is listed in the drop-down menu as "Comuna Corund, HARGHITA".


FIX: generate docs resulting in an empty index.html

w+ was truncating index.html content, then reading+replacing was working over empty strings
r+ is the proper read+write mode, as it doesn't truncate the file at open-time

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 46.57%. Comparing base (c9fcd60) to head (500d927).
Report is 226 commits behind head on main.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #448      +/-   ##
==========================================
+ Coverage   44.65%   46.57%   +1.91%
==========================================
  Files          34       42       +8
  Lines        2770     3605     +835
==========================================
+ Hits         1237     1679     +442
- Misses       1533     1926     +393

Flag
Coverage Δ

unittests
46.57% <100.00%> (+1.91%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Unify group responsibility for administration tasks

I'd like to propose to unify the group responsible for administration tasks such as:

Organization ownership
Read and Write access to our mailbox<EMAIL_ADDRESS>Read and Write access to all technical account(s) & profiles

I'd like to have this clear and unified for both of our organizations "baloise" and "baloise-incubator".
I'd like to propose the "CoP Open-Source" team member's to be 1:1 this group.

Links to https://github.com/baloise/open-source/issues/307

@kullmanp incubator and main org aligned.


API Host Security check

How do you protect the API_HOST from being accessed by unauthorized client code.
So for example, the self-provisioning-demo.js script is accessible from the client/browser meaning anyone can setup their own web page pointing to the API_HOST by looking at this line:
var API_HOST = 'http://localhost:8080';

The same way you would make a log for employess/contributors;
Seems It would only make sense to have a logged file of "visitors"
At least untill their foot is all the way in the door, not trying to sound
'Vague', but it's like letting someone in your house that you hardly know,
Even though you may feel comfortable with them , you still keep a close eye on them,
Possibly like a felon just being release from prison in a sense, if you will.
Block the important parts of the site off to them, seems fairly logical...
To go into detail as far as coding in which I'm sure that's what you had
Been referring to, 'I have no clue'..
-Resins-

Schools that have implemented the self-provisioning approach have put the account creation site behind a school portal. This school portal authenticates students through an existent username/password. In these cases existent usernames were hard to read/remember (e.g. 0129381@domain.com). Therefore the motivation to create new usernames for Google Apps.
If a school has no username/password they could use other methods, such as student ID/birthdate, but then that becomes unsafe. A safer approach is to have the account creation site in a non-public computer (something like a kiosk) and ask students to be physically present in that computer to create their new account. A person can then play the role of authenticating students.

Thanks for the feedback guys. Of late, I've also been getting errors around disabled CORS (Cross Origin Resource Sharing) by this simply meaning the API server won't accept requests coming from external domains. For a public API, that's bad but for our case, I think we should actually expound on this as a security feature ie to only accept requests coming from our own domain.


Translation (Transformer & GNMT) Vocab Changes

SWG Notes:
There is some confusion around the transformer vocab generation. This will be updated with more information from interested submitters/WGs.
No Concrete proposal -- waiting for mailing list or Special Topics to propose solution (from Nvidia?).

SWG Notes:
NMT - No changes for v0.6 (re: vocab)
Transformer - change to load the vocab from file coming to the reference. Help make sure everyone is using the same vocab.

Fixed by mlperf/training PR 244: transformer vocab


Nested Math in Non-Default Text Fix

Resolves:

https://github.com/Khan/KaTeX/issues/1038
https://github.com/Khan/KaTeX/issues/1112

@rrandallcainc friendly ping to see where things are at with this PR.

@kevinbarabash I expect to get back to this early this week. Thanks for checking in.

@kevinbarabash I apologize for the delay on this one. This should be ready for another look.

@kevinbarabash Updated.
Definitely agree that the makeOrd implementation could be cleaned up. I think it's current state was a side effect of the assumption that math/text fonts behaved similar. That being said, it looks like a lot of solid cleanup has been done around fonts recently (i.e. pulling out the oldFonts from the parser, separating the text functions, etc.). I don't think the cleanup for makeOrd should be too bad.

@rrandallcainc glad to see this PR finally merged.  Thanks for seeing it through.


Create a section of the site with links to tutorials

There's already a few tutorials/articles out there on how to setup ESLint. It would be nice to have a section of the site with those links. Some of them explain how to configure ESLint much better then our documentation does. Although, maybe the goal should be to have our documentation be better then those articles?:-)

:+1:

I can add that page in, and collect a few links. But I just wonder if we should write our own quick guide sort of thing, and then link to those other articles for more information. What do you think?

Probably a good idea, though it would be weird to link out from our own guide and say "for more info, see stuff we didn't write." I'd probably keep the two separate.

We added getting started page, I think that's good enough. At this point, there are too many tutorials out there to link to them all.


Spelling Mistake

Hi,
There is spelling mistake in hoverfly.go at L:20
https://github.com/SpectoLabs/hoverfly/blob/develop/hoverfly.go#L20
const SynthesizeMode = "sinthesize"

Thanks

thanks! :+1:


Erreur lors de l'envoi d'une contribution

Sur le thème https://www.mapcontrib.xyz/t/808620-Parking_velos, ajouter ou changer un tag produit une erreur "Contribution échouée - Une erreur est survenue lors de l'envoi de votre contribution." . La console montre le traceback suivant:
TypeError: n.geometry is undefined
Stack trace:
value@https://www.mapcontrib.xyz/js/theme.bundle.js:29:21189
_buildLayerPopupContent@https://www.mapcontrib.xyz/js/theme.bundle.js:22:24340
updatePoiPopup@https://www.mapcontrib.xyz/js/theme.bundle.js:22:23980
map:updatePoiPopup@https://www.mapcontrib.xyz/js/theme.bundle.js:22:10084
getHandler/<@https://www.mapcontrib.xyz/js/commons.bundle.js:6:5891
execute@https://www.mapcontrib.xyz/js/commons.bundle.js:6:6821
sendContributionToOSM/<@https://www.mapcontrib.xyz/js/theme.bundle.js:17:6586
a@https://www.mapcontrib.xyz/js/commons.bundle.js:7:27637
N/<@https://www.mapcontrib.xyz/js/commons.bundle.js:7:27759
c@https://www.mapcontrib.xyz/js/commons.bundle.js:7:31960

hi @naomap , could you check in OSM if it's only a false message (and modifications are ok) like this issue : https://github.com/mapcontrib/mapcontrib/issues/300
I thought it's only for me ?
If you try in a private session of firefox ?

You're right, the data has actually been changed.

I'm not alone :)

@naomap Can you try in a private session?

Also, can you (@naomap and @vinber) give me some node ids to help me reproduce the issue?
Thanks!

From @vinber:

contribution works when he's connected as an administrator
when he's not, it does not work and display the failure notification
happened on https://www.mapcontrib.xyz/t/e02d2b-Parkings_a_velo_a_Villenave_dOrnon with the node<PHONE_NUMBER>

After some quick checks:

the node does have the geometry geojson item when I request it from Overpass
the theme does not send it in the cache but in the non OSM data features


Incorrect flow while creating a hot wallet

A user was creating a hot wallet but on successful wallet creation modal when user clicks on CTA incorrect import wallet screen appears
To verify this you need to import a tribe wallet first then after that try creating a new hot Wallet
Please check
https://github.com/bithyve/bitcoin-keeper/assets/107177634/a330b650-79bc-4b45-95a0-726ca157c64a

@PraneethGunas
Steps to Reproduce the Issue:

Do not open the Keeper app.
Navigate to the Tribe app.
Access the Tribe settings.
Select the Wallet Backup option.
Press "Backup with Keeper".
You will be redirected to the Keeper app and navigated to the Import Wallet screen.
Import the wallet or cancel the import process.
Create a hot wallet in the Keeper app.
After the hot wallet is created, press "View Wallet".
You will be redirected back to the Import Tribe Wallet screen.

Closing as outdated


[MySQL] Create instance from backup / restore backup into instance

When creating service instance we can specify what are the backup settings. Is it possible, though, to use those backups, to restore (using OSBA) the backup into the new service instance?
I'd like to create kubernetes based CI/CD pipeline which will build a new application environment automatically, using an existing database as a starting point.
If possible I'd like to avoid mysqldump-ing the data and manual data restore as much as possible since the database in question has few GB of data and manual import via mysql ... < dump.sql takes way too long for my liking.

Both restoring backup into new service instance and creating new service instance from an existing database are not supported.
The scenario makes sense to me. Actually SQL Server in OSBA implemented using existing database. That was a big refactor. And as a result, the usage of the module got more complicated and maybe also a little confusing. I really not sure whether it was a good change. So sorry there is no plan to support the scenario for MySQL yet :(.
I'd like to keep this issue open. Appreciate more feedback.


Request - Add Ability to Skip to Where Ripper Was Stopped

When I stop a rip I noticed that it goes through all of the files I already ripped the next time I start to rip that album (an agonizingly slow process to watch 😱), and I was wondering if there was a way to make it so that it downloads any new posts, and when it reaches the old posts (already downloaded ones) it skips over the downloaded ones, instead of verifying all of them, and just picks off from where it left off and continues downloading the older stuff, but before all that and before it allows the user to click "Rip", it consults a log file to see if it's been ripped before and if it was stopped mid-rip, and if it's been ripped before, and stopped, it asks the user "Previous Rip of Album Stopped Mid-Rip: Would you like to rip any new posts, and continue where last rip left off?" Unfortunately I have absolutely no Java experience, so I couldn't pull this off even if I tried. So I was wondering if there was someone who could.

Considering the URL logging is possible, that'll be a good idea.

and I was wondering if there was a way to make it so that it downloads any new posts, and when it reaches the old posts (already downloaded ones) it skips over the downloaded ones

It should be possible to achieve this by checking "Remeber Url history" in the configuration. This will cause ripme to log all urls it downloads and it will auto skip them in future rips.

[...], and I was wondering if there was a way to make it so that it downloads any new posts, and when it reaches the old posts (already downloaded ones) it skips over the downloaded ones, instead of verifying all of them, and just picks off from where it left off and continues downloading the older stuff, [...]

Sounds much diff from the URL remembering. It's possible, but requires a rewrite of

what data is stored (I gave the 1st version of the syntax in some other ticket here)
RipMe
to avoid doing all the work that's now done before the URL remembering tells RipMe that's already ripped


Added Brian Kernighan Algorithm

Hello Respected Sir,
I have added a well written and structured code for "Brian Kernighan's Algorithm". If you feel like it is worth it, then please merge it with the Hactoberfest tag. If any changes are required then please inform me.
Regards,
Debanjan Poddar
@Shubhamrawat5

Kindly merge this with the hacktoberfest tag, if it is okay.
Regards,
Debanjan Poddar
@brenovambaster
@Shubhamrawat5

@brenovambaster  how are you adding yourself as reviewers?

Sir,
I don't know who is @brenovambaster .
Please merge it, if the code is good enough.
@Shubhamrawat5

Sir, @Shubhamrawat5, @debanjan-2002
I don't have the power to merge, I just review the code and give the opinion. Who can merge is just the repository owner.

Yeah. Got it now

@Shubhamrawat5

@brenovambaster how are you adding yourself as reviewers?

Anyone can review third-party code, but cannot merge it. Unless the repository owner gives you this authorization. In this case, I just reviewed it.


Redundant sweep of local balance after remote force close

Background
Maybe I don't understand something here but my lnd sends my balance from one p2wkh address to another after the channel was force closed remotely. There are no anchors, no HTLCs in the commitment, just two outputs: one belongs to the remote party and is a p2wsh address, the other is mine and is a p2wkh address. For some reason I can't understand lnd makes a redundant tx for my output and wastes fee and blockspace. I already fully control this address, why send from it again?
Your environment

version of lnd v0.15.0-beta-164-g115b04187
which operating system (uname -a on *Nix) Debian 11.4, kernel 5.15.56-v8+ aarch64
version of btcd, bitcoind, or other backend bitcoind v23.99.0-225e5b57b2ee
any other relevant environment details

it moves the funds into the main wallet keys space, this will make it easier if you had to do a rescan from seed to detect the funds

Maybe it should be configured? I wasn't very happy when it automatically paid 5 sat/vB without much urgency. I understand that fees are hard to estimate when the tx can be sent at an unknown time and there are certain risks of fraud. But in this case there are none except if my node crashes right after this remote force close which is very unlikely. I'd like to pay the bare minimum if it's needed at all, maybe it's possible to use a wallet address in commitment in the first place? I know it can be specified during channel opening so why not do it by default? What are the downsides?

@guggero is this issue addressed by https://github.com/lightningnetwork/lnd/pull/6868?

I still don't understand why we can't use wallet addresses in the peer commitments for our side (i.e. when the peer force closes we get our part directly into the wallet). I looked it up but couldn't find anything conclusive except some "privacy issues". Can only guess but maybe it's about reusing the same address for on-chain tx and force close? I would gladly use some addresses twice (if they can't be marked internally as reserved for a future force close) and potentially save thousands of sats. Still, I'm only guessing about the reasons for this sweep as it's still not explained.

all the outputs on the anchor channels are script outputs so it couldn't use normal addresses due to the design of the anchor channel protocol
the funds are still part of the same wallet, just a different keyspace in the wallet
in theory moving the funds out of the script output is optional and could be covered with a custom rescan in the case of recovery but that is not implemented in lnd

No, I specifically highlighted in OP that the other party's output is a regular P2WPKH, not P2WSH. That's the whole point of this post, why transfer from one P2WPKH to another P2WPKH when both are managed exclusively by my node and have no special conditions to spend.

sorry I meant for anchors that is the case, although it's a similar scenario, i mean going forward all channels will be anchors and the older format will be phased out but this dynamic will still remain the same across the formats

For anchors yes but that wasn't what happened with the channels I linked. I realize the benefits of anchors but still they're not universally supported and I usually don't check beforehand if the other node supports them. So this additional spend looks redundant and I hope it can be avoided by smarter address selection so that it doesn't need an additional sweep.

the enabling feature for that might be the addition of a recovery scan that can sweep these single key outputs from the HD seed, currently that is a feature available in an external tool but not yet present in lnd
at that point like you said maybe you would then want to turn on something to reuse keys in the commitment transaction to avoid the potential scenario of a too-large gap limit

@rkfg the history of this is:

With legacy channels (pre lnd 0.6) you needed the per_commit_point from the remote peer to sweep the to_remote balance (your balance if the remote party force-closed). So it was crucial to sweep as the funds would've been lost if the peer went offline for good.
With STATIC_REMOTE_KEY channels (which is the default if you don't have anchor channels activated) the above is not necessary. But we never updated the sweeper logic (changes to the channel state machine are always high risk!) to skip the extra sweep. Also, the keys of the to_remote output aren't in the BIP84 key scope so they wouldn't be discovered by the wallet recovery if you recovered from seed. So it's still necessary to sweep them to safety (until we fix https://github.com/lightningnetwork/lnd/issues/4778).
With anchor output channels we now have a script in the to_remote, so it becomes crucial to sweep them again, as the script isn't included in the seed.

@saubyk no, this is related to https://github.com/lightningnetwork/lnd/issues/4778.

@guggero thank you, now it's clear.

Closing as a duplicate of https://github.com/lightningnetwork/lnd/issues/4778


Formatting of training data

Hi Yasamin, awesome work! I'm trying to train the model on my own dataset, and I wonder if you could clarify the formats of the labeled data. Specifically, for normal, what should the channel 1,2,3 of the txt files each store? And how did you convert them into visualization?

Hi. Thanks for your comments. The format can be found here. Each 1,2,3 is the first to third channel of the normal vector.
You can check how the visualization is done in the inference code.

Thanks for the reply!
I'm still a bit unsure about the format of the normal vectors. Are they camera-space normals in OpenGL format (channel 1-right, 2-up, 3-back) ?

After some fiddling with your sample data, I'm guessing that the normals ought to be in camera-space and stored in OpenCV format (channel 1-right, 2-down, 3-front). Is that correct? 😀😀

It is in camera space. I have computed the normal in the 3D world coordinate and multiplied it by the camera rotation matrix to get it in camera space. So yeah it should be what you mentioned.

I see. Many thanks for your replies!

It is in camera space. I have computed the normal in the 3D world coordinate and multiplied it by the camera rotation matrix to get it in camera space. So yeah it should be what you mentioned.

Hi author, could you pls tell me how you generate GT normal map from mesh? use pyrender?

Hi. I use the nearest neighbors to compute the normals myself but you can also use the off-the-shelf pipelines.


fix coveralls

Doesnt seem to have worked since 11 APR 2018 - 7:05 probably something to do with using old version in cake.net

wont fix, no idea


Binance Integration Test: No transactions

time="2019-03-23T13:03:13+01:00" level=info msg="Testing endpoint" @platform=binance
time="2019-03-23T13:03:14+01:00" level=warning msg="No transactions" @platform=binance
time="2019-03-23T13:03:14+01:00" level=info msg="Endpoint works" @platform=binance
time="2019-03-23T13:03:14+01:00" level=info msg="Endpoint tested" @platform=binance fields.time=592.104107ms

Binance DEX Index was reset, a new test address is needed.

https://binance-chain.github.io/api-reference/dex-api/paths.html#apiv1transactions
from the docs:
Query Window: Default query window is latest 24 hours; The maximum start - end query window is 3 months.

Another idea then: To keep the test generic, we define address getters per platform in the integration test.
Binance will just get a list of recent transactions and pick the first one.

The issue was in Binance's API. Fixed by switching to the explorer API.


target for gosec scan

Description
Added gosec scan as target to detect anu major vulnerability that can be detected and patched as and when required. This will also be incorporated in the CI/CD flow.
Output : A short snippet of expected output is shared below however a detailed output with issues detected will be logged in result.json file :
  Gosec  : 2.12.0
  Files  : 58
  Lines  : 4840
  Nosec  : 0
  Issues : 71

Fixes # (issue)
Primary Components Effected
Please mark one option with an "x".

[ ] Shifter (Core CLI and Server Application)
[ ] Shifter UI (The Web Interface)
[x] Cross Component Configuration (.github workflows, templates, docs, .MD files)

Type of change
Please mark one option with an "x".

[ ] #RELEASE New Collection of Features (containing breaking and non-breaking changes)
[ ] #FT New feature (non-breaking change which adds functionality, imrproves performance or code quality)
[ ] #BUG Bug Fix (non-breaking change which fixes an issue)
[x] #PATCH Patch (non-breaking change which modifies only non-code object [Docs, Readme])

Developer / Author Notes
Please feel free to provide any additional notes, content or context relating to this PR and subsequent contribution.

Can you add gosec to the runners also and to the pipeline?

Hi @garybowers ,
I have just added the changes, following is the link which had a succesful run when merged with master(forked repo)
Link : https://github.com/parasmamgain/shifter/actions/runs/2834017282


Links not working correctly

Copied from Ryver.
Mohsin KhalidMon 5:46pm
Hi All;
https://nhsconnect.github.io/CareConnectAPI/design_access.html
This site is not working properly some of the links are not opening ,for example the Care Connect Reference Implementation Guide link does not work

Also experience a number of dead links. A free scan of the site (using online tool https://www.deadlinkchecker.com) shows some 64 deadlinks. There may be more as the free service does not allow a full scan.
CC-API-deadlinks.csv.txt


Use '*' as the default strict_max_version for WebExtension

Fixes #1339
Needs https://github.com/mozilla/amo-validator/pull/391
/!\ DO NOT MERGE before bumping the amo-validator and fixing the requirements

Tests failing and there are conflicts but besides that it looks good. r+


How to i open them in split and tab?

Hi
I am currently using FuzzyBuffers. How do i open the files in :vs/:sp or :tab?
Thanks

In the lastest version you can split buffer vertically/horizontally in FuzzyBuffers by pressing ctrl + v or ctrl + s. Later I will add support for :tab after I am more familiar with that feature.

ctrl + t opens selected file in a new tab page in the latest version.

Thank you so much for the super prompt fix


Entities not updating without manual intervention via the app

I was excited to see this integration, but I am confused. In order to be able to use this for usable automations the entities would have to update without manually updating them in the MyToyota app.
In my case, the door open statuses show a last updated time of about 3 hours previously. I can of course manually update them in the app but this seems somewhat pointless, and it also omits any history of events. So if

Door is closed
Door is opened
Door is closed
Update is forced in the app

In HA the door shows as constantly closed, because the app is never updated when the door is open.
is this how it is supposed to work? If that is the case I'm not clear as to the use case for the integration (for example it cannot issue an alert if the vehicle is accessed). Am I missing something?

Same here. Not updating status of the entities in HA. Both my Bz4X and my Prius gen5.  But i like the integration in HA.


Notification without observe option

The RFC7641 -  Observing Resources in the Constrained Application Protocol (CoAP) § 3.2.  Notifications says :

In the event that the resource changes in a way that would cause a
normal GET request at that time to return a non-2.xx response (for
example, when the resource is deleted), the server sends a
notification with an appropriate response code (such as 4.04 Not
Found) and removes the client's entry from the list of observers of
the resource.  Non-2.xx responses do not include an Observe Option.

I think that currently java-coap doesn't allow this.
Because when I try to send a 4.04 Not Found as notification, it is never sent.
I guess this is because of NotificationValidator, see :
https://github.com/open-coap/java-coap/blob/a0c8294df5a35d5f6a001f3fbc4ab85d95a89534/coap-core/src/main/java/com/mbed/coap/server/NotificationValidator.java#L24-L35
Not directly linked but that exception was silently ignore in my code (I mean I didn't even see a log), I don't know if there is something to improve in java-coap or in my code for this last point.

I think there is more or less same issue at CoAP client side, see https://github.com/open-coap/java-coap/blob/a0c8294df5a35d5f6a001f3fbc4ab85d95a89534/coap-core/src/main/java/com/mbed/coap/server/messaging/CoapDispatcher.java#L65-L68

any news about that ? :slightly_smiling_face:

Sorry, nothing yet, I did not have time to look into it.


September Doc Cleanup based on Github issues

Context
Handles Github issues and a few internal issues in sections that have been recently updated in the Gradle User Manual.
This is a documentation change ONLY.
Fixes:

Continuous build description - https://github.com/gradle/gradle/issues/26252
Document GRADLE_LIBS_REPO_OVERRIDE - https://github.com/gradle/gradle/issues/24217
Deprecated alternative to setOutputFile - https://github.com/gradle/gradle/issues/25586
Fix build order preference - https://github.com/gradle/gradle/issues/26088
How to use system file watch with symlinks - https://github.com/gradle/gradle/issues/26201
Update Centralizing repositories declaration section of Dependency Management for @erichaagdev
Update admonition block CSS for @alllex
Fix javadocs CSS for search background color for @lkasso

Reviewers:
Under development.

@bot-gradle test and merge

Your PR is queued. See the queue page for details.

I've triggered a build for you. Click here to see all failures if there's any.


Font smoothing

Please add font smoothing for Brackets in OSX. The fonts look really bad. See attached.

OSX - brackets 1.2. Is there a better build available?

O yes :+1:

It is enabled.  Just maybe not the type you are looking for.
https://github.com/adobe/brackets/blob/master/src/styles/brackets.less#L58
I am working on a small plugin to help you configure this and a couple of other font related stuff.  We tried to add this as a setting in brackets a while ago... We had a UI and everything...  But this setting is only has any sort visual effect on MacOs, so decided against to reduce confusion for folks in other platforms.


updating slack notification for failure

@benbp @heckj @zyoung51

:+1:

+1

So we had a room that reported all builds on hipchat. Where is got all of the build status. I dont know if everyone ever really looked at it. As you can also go in travis to see the current status.
Im ok with changing success: changed. This means whenever it gets merged to master then we will get a travis update.

I'm worried that if we do success: changed, we'll get duplicate notifications for failure + success changed to failure?

+1


Publish App Mining Audit and Changes Criteria

Gina will draft something here.

With an extended audit period, app founders should have clear means of communication with the PBC team and reviewers to make necessary changes within the timeframe. Outside of the audit period, if an error is found, the error should be updated but no difference in payment will be made unless the error was egregious at the discretion of Blockstack PBC. This encourages app founders to thoroughly audit results during the audit period and can lead to transparent and consistent results on a monthly basis.

Sounds like this is done just need to put in changelog.


BUG: Unbond no amount in memo

Amount is not updated in the memo when the amount is added, only updates when the node address is updated.
Should update when the amount is entered.

Fixed


GAMS system directory not specified

Hi,
I'm a new user on a MacBook.
When I try to run code scenario.solve() in one of the tutorials (Westeros), I receive the following error:
com.gams.api.GAMSException: GAMS system directory is both not specified and not found from your environment!
As per the installation instructions, to include the path to the GAMS program in the PATH environment variable, I have added the following line to the end of .bash_profile:
export PATH=$PATH:/Applications/GAMS29.1
I am not sure if I have specified the path to the correctly, as I still receive the above error.
Thanks,
Bassam

You can add GAMS sysdir to the system path as follows:
export GAMS_PATH=/Applications/GAMS29.1/Resources/sysdir
export PATH=$PATH:${GAMS_PATH}


inso lint cannot run against local file

Describe the bug
When linting a file, it is useful to ensure exactly what file is being checked.
To Reproduce
Steps to reproduce the behavior:

checkout a repo
run inso lint spec ./spec.yaml
see FATAL error

Expected behavior
Expect the file to be read, loaded and linted properly.
Screenshots
n/a cli operation
Desktop (please complete the following information):

OS: [e.g. macOS 10.15.7]
Installation Method: npm for inso, .pkg downloaded from insomnia.rest for ID
App Versions: inso 2.2.25 and ID 2020.4.1

$> ls
README.md           uuid-generator.yaml
$/uuid-generator-service> inso --verbose lint spec ./uuid-generator.yaml
› Data store configured from app data directory at /Users/amiller/Library/Application Support/Insomnia Designer
› Load api specification with identifier ./uuid-generator.yaml from data store
› Found 0.
FATAL  Specification not found.

Currently inso generate config supports loading from a file on the filesystem:
https://github.com/Kong/insomnia/blob/65a333cac317e7082e9a026e43588c404f026259/packages/insomnia-inso/src/commands/generate-config.js#L54-L59
inso lint spec does not support this yet, but adding in a different source is as simple as taking a snippet of the above code, loading the contents of the file, and consuming the contents in the lint-spec implementation:
https://github.com/Kong/insomnia/blob/65a333cac317e7082e9a026e43588c404f026259/packages/insomnia-inso/src/commands/lint-specification.js#L16-L24
the desired process (for generate config and lint spec) is to:

try load from db
then try load from file
then fail with a fatal error if neither can be found

Thanks for the info! I can take this one.

Thanks @joemirizio, awesome!


update?

should be good though you should review

If you want, we can talk about what you need from the bot in my support discord: https://discord.gg/MMJFtCtYPP
Maybe we can figure stuff out together :)

Closed because of inactivity. Can be reopened when needed.


Put apple-source-releases in Hydra

I just realized that a few of my apple open source derivations are broken (developer_cmds, network_cmds, etc.) I wanted to find the first point they failed but can't because they aren't in the Hydra job set. Maybe we can change this for everything in Darwin.*?
Cc @copumpkin

I noticed the same for iterm2. But it's not clear to me why that one is ignored, it's a top level attribute and has a license.

@lnl7 I'm wondering if something else is going on here. Maybe Hydra isn't working right with derivations that are darwin only? I've had a similar issue with another top level derivation like that.

You're right, I think darwin only jobs are just not showing up. I thought it was related to meta.license.

I see what's going on now. pkgsFor "x86_64-linux" is used to discover the platforms here https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/release-lib.nix#L96, but a darwin only drv won't evaluate.
I think https://github.com/NixOS/nixpkgs/pull/22277 would fix this.

The function you refer to creates a set with attribute, platform pairs. Your example iterm2 has meta.platforms specified. so I don't see how that piece of code is relevant. The functions applePackage and appleDerivation don't seem to be setting any meta at all so I think that's best fixed instead of using the solution from #25930.


Add VCache headers to /query and /invoke

Closes #2661

Agreed, definitely want to move caching towards an orthogonal application over time.

@wemrysi Ready for another look


Send nb_classes to check_and_transform_label_format in inference attacks

Description
Send nb_classes to check_and_transform_label_format in inference attacks.
Fixes #1712
Type of change
Please check all relevant options.

[ ] Improvement (non-breaking)
[ X] Bug fix (non-breaking)
[ ] New feature (non-breaking)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

Testing
New test that sends labels as column vector.
Test Configuration:

OS: MacOS 12.3
Python version 3.7

Checklist

[X ] My code follows the style guidelines of this project
[ X] I have performed a self-review of my own code
[ ] I have commented my code
[ ] I have made corresponding changes to the documentation
[X ] My changes generate no new warnings
[ X] I have added tests that prove my fix is effective or that my feature works
[X ] New and existing unit tests pass locally with my changes

Codecov Report

Merging #1713 (e50efb3) into dev_1.11.0 (8df19af) will decrease coverage by 65.54%.
The diff coverage is 7.69%.

@@               Coverage Diff               @@
##           dev_1.11.0    #1713       +/-   ##
===============================================
- Coverage       88.09%   22.54%   -65.55%
===============================================
  Files             259      259
  Lines           21347    21342        -5
  Branches         3789     3512      -277
===============================================
- Hits            18805     4812    -13993
- Misses           1598    16474    +14876
+ Partials          944       56      -888

Impacted Files
Coverage Δ

.../attacks/inference/attribute_inference/baseline.py
26.15% <0.00%> (-45.52%)
:arrow_down:

...rence/membership_inference/black_box_rule_based.py
42.85% <0.00%> (-37.15%)
:arrow_down:

...attacks/inference/attribute_inference/black_box.py
18.10% <11.11%> (-62.98%)
:arrow_down:

...ference/attribute_inference/true_label_baseline.py
21.68% <11.11%> (-46.27%)
:arrow_down:

art/evaluations/evaluation.py
0.00% <0.00%> (-100.00%)
:arrow_down:

art/estimators/object_detection/utils.py
0.00% <0.00%> (-100.00%)
:arrow_down:

art/evaluations/security_curve/__init__.py
0.00% <0.00%> (-100.00%)
:arrow_down:

art/attacks/poisoning/perturbations/__init__.py
0.00% <0.00%> (-100.00%)
:arrow_down:

...essing/expectation_over_transformation/__init__.py
0.00% <0.00%> (-100.00%)
:arrow_down:

...rmation/natural_corruptions/contrast/tensorflow.py
0.00% <0.00%> (-100.00%)
:arrow_down:

... and 198 more


Provide a Helm chart for SecretsManager deployment

Include (inside a deployment/ folder, according to the recommendations in golang-standards/project-layout) a Helm Chart that installs SecretsManager in K8s.

how does this go on? Almost one year has passed

I made my own here.

https://github.com/broadinstitute/datarepo-helm/tree/master/charts/install-secrets-manager

You will still need to provide your own image. Which can be over written here.
  repository: gcr.io/broad-jade-dev/secrets-manager
  version: v1.0.2

I also created a secret creation chart here.

https://github.com/broadinstitute/datarepo-helm/tree/master/charts/create-secret-manager-secret.

These are all first passes and may not include all args and env vars.

I created a Helm chart for Secrets-Manager and we have been using it in production for several months without issues.
https://github.com/jkirkham-ratehub/secrets-manager/tree/add_helm_chart
I can create a PR to merge this into this repo if you want.  Let me know.

@jkirkham-ratehub

I created a Helm chart for Secrets-Manager and we have been using it in production for several months without issues. https://github.com/jkirkham-ratehub/secrets-manager/tree/add_helm_chart
I can create a PR to merge this into this repo if you want. Let me know.

I tried to use same steps what you used here to prepare latest chart version from tag v2.1.0. But i am facing issue after deployment,
Error: Failed to render chart: exit status 1: Error: unable to build kubernetes objects from release manifest: error validating "": error validating data: ValidationError(SecretDefinition): unknown field "type" in io.tuenti.secrets-manager.v1alpha1.SecretDefinition
i am preparing chart using  kustomize build config/default -o /temp/
Here is generated crd file
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: v0.4.1
creationTimestamp: null
name: secretdefinitions.secrets-manager.tuenti.io
spec:
group: secrets-manager.tuenti.io
names:
kind: SecretDefinition
listKind: SecretDefinitionList
plural: secretdefinitions
singular: secretdefinition
scope: Namespaced
versions:

name: v1alpha1
schema:
openAPIV3Schema:
description: SecretDefinition is the Schema for the secretdefinitions API
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: SecretDefinitionSpec defines the desired state of SecretDefinition
properties:
keysMap:
additionalProperties:
description: DataSource represents the actual source of truth path
for a secret
properties:
encoding:
description: Encoding type for the secret. Only base64 supported.
Optional
type: string
key:
description: Key where the actual secret is stored
type: string
path:
description: Path to the actual secret
type: string
required:
- key
- path
type: object
type: object
name:
description: 'INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
Important: Run "make" to regenerate code after modifying this file'
type: string
type:
type: string
required:
- keysMap
- name
type: object
status:
description: SecretDefinitionStatus defines the observed state of SecretDefinition
type: object
type: object
served: true
storage: true
status:
acceptedNames:
kind: ""
plural: ""
conditions: []
storedVersions: []

Could you provide me any direction on it?

@jkirkham-ratehub @eduardogr

I created a Helm chart for Secrets-Manager and we have been using it in production for several months without issues. https://github.com/jkirkham-ratehub/secrets-manager/tree/add_helm_chart
I can create a PR to merge this into this repo if you want. Let me know.

I tried to use same steps what you used here to prepare latest chart version from tag v2.1.0. But i am facing issue after deployment, Error: Failed to render chart: exit status 1: Error: unable to build kubernetes objects from release manifest: error validating "": error validating data: ValidationError(SecretDefinition): unknown field "type" in io.tuenti.secrets-manager.v1alpha1.SecretDefinition
i am preparing chart using kustomize build config/default -o /temp/
Here is generated crd file
apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: controller-gen.kubebuilder.io/version: v0.4.1 creationTimestamp: null name: secretdefinitions.secrets-manager.tuenti.io spec: group: secrets-manager.tuenti.io names: kind: SecretDefinition listKind: SecretDefinitionList plural: secretdefinitions singular: secretdefinition scope: Namespaced versions:

name: v1alpha1
schema:
openAPIV3Schema:
description: SecretDefinition is the Schema for the secretdefinitions API
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: SecretDefinitionSpec defines the desired state of SecretDefinition
properties:
keysMap:
additionalProperties:
description: DataSource represents the actual source of truth path
for a secret
properties:
encoding:
description: Encoding type for the secret. Only base64 supported.
Optional
type: string
key:
description: Key where the actual secret is stored
type: string
path:
description: Path to the actual secret
type: string
required:

key
path
type: object
type: object
name:
description: 'INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
Important: Run "make" to regenerate code after modifying this file'
type: string
type:
type: string
required:
keysMap
name
type: object
status:
description: SecretDefinitionStatus defines the observed state of SecretDefinition
type: object
type: object
served: true
storage: true
status:
acceptedNames:
kind: ""
plural: ""
conditions: []
storedVersions: []

Could you provide me any direction on it?

Was able to resolve issue after removing type from SecretsDefinition file

@a-thorat is it related to the version of your kubernetes cluster?

@a-thorat is it related to the version of your kubernetes cluster?

Thanks for your response. No it was not related to kubernetes cluster version. Found issue in one of SecretDefiniation file where Secret type was mentioned explicitly as Opaque. After removing it everything seems to be working.


API handling for login event

Can you point me into the right direction on how to get a boolean value if a Player has logged in or not (not via event call, but by passing in a Player object and getting a boolean back from your plugin)? In my queue plugin I first need to check if a player is logged in or not. If they are, I add that player to my ArrayList. (This way the authentication plugin is run on a separate minimalistic server along with the queue built in. For example: If the player disconnects from the queue server and connects again, they are automatically logged back in via your plugin, and then added back into the que unless they do /logout or their login session expires, then I am using your event listener to remove that player from the queue). Thanks!

Sorry for my late response
you would do something like:
final PlayerSession session = LoginSecurity.getSessionManager().getPlayerSession(player); if(session.isAuthorized())

Thanks!


Transaction Malfunction

What version of Prisma Dart is running?
^5.1.1
What version of Prisma CLI is running?
5.21.1
What type of app are you using?
Flutter
What database are you using?
SQLite
What steps can reproduce the bug?
Schema
schema-excerpt:
generator client {
  provider   = "dart run orm"
  output     = "../lib/_generated_prisma_client"
  engineType = "flutter"
}

datasource db {
  provider = "sqlite"
  url      = "file:./db.sqlite"
}

model User {
  id   Int @id @default(autoincrement())
  name String @unique

  currentlySelected Boolean
}

prisma.dart and main.dart are initialized as described in https://prisma.pub/getting-started/flutter.html
Query Examples
I tried to run the transactions as described in the official docs (https://prisma.pub/queries/transactions.html).
After restarting my application I can see, that no changes were actually written to the database.
Attempt 1:
Future<List<D>> deleteManyAndReturn<T, D>(
  List<T> itemsToDelete,
  Future<D?> Function(PrismaClient tx, T item) doDelete,
) async {
  final deleted = <D>[];
  await prisma.$transaction(
    (tx) async {
      for (final item in itemsToDelete) {
        final del = await doDelete(tx, item);
        if (del != null) {
          deleted.add(del);
        }
      }
    },
  );
  return deleted;
}

await deleteManyAndReturn(
  ids,
  (tx, id) => tx.user.delete(
        where: UserWhereUniqueInput(id: id),
      ));

Attempt 2:
import 'package:work_time_table_mobile/_generated_prisma_client/client.dart';
import 'package:work_time_table_mobile/prisma.dart';

Future<List<D>> deleteManyAndReturn<T, D>(
  List<T> itemsToDelete,
  Future<D?> Function(PrismaClient tx, T item) doDelete,
) async {
  final deleted = <D>[];
  final tx = await prisma.$transaction.start();
  try {
    for (final item in itemsToDelete) {
      final del = await doDelete(tx, item);
      if (del != null) {
        deleted.add(del);
      }
    }
    await tx.$transaction.commit();
  } catch (err) {
    await tx.$transaction.rollback();
  }
  return deleted;
}

await deleteManyAndReturn(
  ids,
  (tx, id) => tx.user.delete(
        where: UserWhereUniqueInput(id: id),
      ));

What is the expected behavior?
The transactions should run all overhanded deletions and return the deleted elements.
This is successful if running the queries without any transactions.
E.g.:
Future<List<D>> deleteManyAndReturn<T, D>(
  List<T> itemsToDelete,
  Future<D?> Function(PrismaClient tx, T item) doDelete,
) async {
  final deleted = <D>[];
  for (final item in itemsToDelete) {
    final del = await doDelete(prisma, item);
    if (del != null) {
      deleted.add(del);
    }
  }
  return deleted;
}

await deleteManyAndReturn(
  ids,
  (tx, id) => tx.user.delete(
        where: UserWhereUniqueInput(id: id),
      ));

What do you see instead?
Once transactions are used, no change is performed in the database. - When restarting my applicaiton, all formally deleted entries are re-appearing in the UI.
Additional information
I also tried it for several update statements. This leads to the same behavior. (Running queries without transactions works - once transactions are introduced to ensure consistency between interdependent statements, no update is persisted.)
For example, I tried it with the following statements for setting one single user as "currentlySelected":
await prisma.$transaction((prisma) async {
  await prisma.user.updateMany(
    data: const PrismaUnion.$1(UserUpdateManyMutationInput(
      currentlySelected: PrismaUnion.$1(false),
    )),
  );
  await prisma.user.update(
    data: const PrismaUnion.$1(UserUpdateInput(
      currentlySelected: PrismaUnion.$1(true),
    )),
    where: UserWhereUniqueInput(id: id),
  );
});

My app is currently running with:

sdk: ^3.5.2
orm_flutter: ^0.2.0
orm: ^5.1.1

I testet the functionality on Android 12.

@Joschiller Thank you for your detailed feedback. I have reproduced the error. Currently, it is located in the engine problem. I will notify you in the comments if there are any fixes or other progress. Thank you again for your contribution. ❤

Thank you very much for the quick reply! Looking forward to the update.
Have a nice weekend :-)

@Joschiller Have a great weekend, I fixed it in orm_flutter:0.3.1 version.
Sorry, I can't fix it in 0.2.x, because my time and energy only allow me to maintain the latest version branch at the moment.

Just installed and tested the fix. It's all working :-)
Thanks again for your effort!


Usage of a dev-version of freva-deployment

There is no description of how to try out a dev-version of freva-deployment. What I came up with was  installing it locally via
pip install .

After that, the dev-version was available, but it seemed, that some template-files were missing:
FileNotFoundError: [Errno 2] No such file or directory: '/home/etor/.conda/envs/freva-deploy/share/freva/deployment/config/evaluation_system.conf.tmpl'

Going further from this error message we have downloaded the File (I forgot where I found this command, but I am relatively sure that I didn't came up with it myself):
wget https://raw.githubusercontent.com/FREVA-CLINT/freva/main/assets/evaluation_system.conf -O assets/share/freva/deployment/config/evaluation_system.conf.tmpl

And copied the whole config-Folder into the location mentioned in the error message.

This error was most likely introduced when I switched from setuptools to flint as a build system. I had a custom setuptools build class that downloaded the file when installing in develop mode. I'll check if a similar approach is possible with flint. If not we will have to resort back to a makefile.


Add me

cc @dcampbell24
Not sure about LICENSE

MIT license is pretty liberal, but could add you or could also change the license to "IUP Rust Authors" and create an AUTHORS.txt file like the main Rust project has. That would probably scale better if there are more authors that come along in the future.


Web: Number widget in settings GUI look bad

Testing #83296:
Visual Studio Code - Insiders
Version: 1.40.0-insider
Commit: b7b21844e760e632f39bedf651da2bc93f54b0cc
Date: 2019-10-29T06:38:51.389Z
Browser: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0
Settings editor - number widget looks bad with the default control on Firefox Linux:

//cc @misolori similar to the select dropdown issue we have

Since I looked into this from https://github.com/microsoft/vscode/issues/83556 I pushed a change.
@roblourens please note: https://github.com/microsoft/vscode/commit/b7e9344c5de8d87d92751d9eef867b2a42d5f1c1


Terminal text for task output becomes invisible after some time

VSCode Version: 1.37.1
OS Version: macOS 10.14.6

Steps to Reproduce:

Use VSCode for a while (generally, at least four hours, but sometimes less).
Run many tasks that have terminal output
At some point, notice that most of the terminal output is invisible. The cursor can be seen to move when some process is producing output. Clearing the terminal does not help. Neither does closing it entirely and letting it be re-opened automatically by a task. Only restarting "fixes" the problem for a time.

Does this issue occur when all extensions are disabled?: Haven't checked yet
I'm sorry that this report is vague. Indeed I have not run the application with extensions disabled, because I wouldn't be able to do much, unfortunately — and it does seem to be necessary to actually use and interact with the application, not just have it open.
I've looked through existing reports that look similar, but nothing I've found quite matches this.
I'll try the Insiders build next, but as I haven't seen any related issues, I felt like it might be okay to go ahead and raise this.
I'm hoping there might be some diagnostic I could produce, or something else to try, when I next see this behavior.
I don't know whether it's worth mentioning, but I do have my tasks set to clear the terminal before they run.

/duplicate https://github.com/Microsoft/vscode/issues/69665


Tooltip and Popover on btn-group

http://jsfiddle.net/JEBdk/5/
tooltip or popover on btn-group. the btn-group is not display correctly!
the last btn have not round corners and between the btn's is a space.
my suggestion, by generating the tip or popover on the end of document .

This issue is still present in Bootstrap 3.

@win32re Please read our docs:

Popovers in button groups and input groups require special setting
When using popovers on elements within a .btn-group or an .input-group, you'll have to specify the option container: 'body' (documented below) to avoid unwanted side effects (such as the element growing wider and/or losing its rounded corners when the popover is triggered).


removed duplicated rules from .jscsrc

Removed 5 duplicated rules from .jscrc (pattern of previous commits seemed to favor eslint).

LGTM!

Released as 2.4.1.


Remove superfluous inlines.

Reading more of the source I stumbled over a few more uses of inline that aren't needed.

Coverage remained the same at 100.0% when pulling 6b3301d17c5447b9197f86d1ae9a7e1e85ba94ce on ColinH:master into 09ff43ef0a9bc782eec42b8b2b10818ccc16f1e9 on skypjack:master.


Mobile: update arrow in navigation dropdown

Preview is available here:http://laboratory-pr1182.previews.kube001.services.stellar-ops.com/


cef package is too large

The volume of the cef package is too large, which leads to the application, and the overall package volume reaches 400M+. Is there a way to reduce it?

@tenove maybe, it depends.
Unzipped CEF binaries for v96 is about 315 MB.
This is the current layout:

So on CEF binaries, you can investigate what files and resources are required for your app and see those you can remove. This wiki should help - https://bitbucket.org/chromiumembedded/cef/wiki/Tutorial.md
For .NET Core/.NET 6 dll files, you can see if you can trim them. There are various options online to do that.
Compression/decompression may be helpful too.

You don't need cef_sandbox.lib (or any of the lib files). I wouldn't exclude/delete anything else.


HOMEBREW_PREFIX already set, exiting after upgrading to v0.2.0

After upgrading to v0.2.0 I get this error every time I run xontrib load homebrew:
xontrib-Homebrew: HOMEBREW_PREFIX already set, exiting

When switching back to v0.1.0 it works fine again:
xpip install xontrib-homebrew==0.1.0

My .xonshrc contains a bunch of other things but only one call to xontrib load homebrew and I do not define this env variable manually in it.

Version 0.2.0 autoloads itself, so you your'e loading it the second time manually xontrib load homebrew and thus getting the log message (though this is not an error message, it's a log message)
Also, it should only appear if you have manually set $XONTRIB_HOMEBREW_LOGLEVEL a higher level  (3 print more verbose messages, while the default is 1), can you check that and set it to something less verbose?
Otherwise it's not a big deal to load it twice, the xontrib does nothing and just exists

@eugenesvk yes I did have the $XONTRIB_HOMEBREW_LOGLEVEL flag on - I added it previously when trying to debug some issue.
I tested again and figured out the problem. I noticed that I set the $PATH manually with all of the values I wanted.
I guess this xontrib loads automatically first, before my .xonshrc is run, right?
I changed my .xonshrc to prepend instead and now it gives a similar result to what I used to have with 0.1.0.
Thanks!

I guess this xontrib loads automatically first, before my .xonshrc is run, right?

Most likely, and it might actually be an issue as  autoloading ignores your environment variables, so I've made a separate branch that disables it, check out
https://github.com/eugenesvk/xontrib-homebrew#known-issues for how to install it if you encounter such issues (and maybe upvote the xonsh bug report linked there)
Anyway, given that there are two solutions:

disable the log verbosity
install the non-auto-loaded alternative

think this issue is resolved

Great, thanks @eugenesvk 🙏


how could i use CLIENT_NICKNAME?

hello everyone. i ll use CLIENT_NICKNAME in TSC_CLIENT_QUERYDATA. how? example pls. i ll get player's ts nickname, channel id.

i think ll use TSC_QueryClientData or TSC_QueryChannelData. i dont understand params.
(clientid, TSC_CLIENT_QUERYDATA:data, const callback[], const format[] = "", {Float, _}:...);
clientid = player client id,
data = the data i want,
callback = data will be transferred callback,
format = ? (some info plz)


Is from_json intended to ignore the key argument if newline is True?

Seems like it: to_json raises an error if both are specified.