pythonPackages.protobuf doesn't install native libraries

I've noticed this accidentially while working on https://github.com/NixOS/nixpkgs/pull/13035 -- installFlags on this line doesn't actually do anything. I don't know how to fix this because the flag needs to be used for setup.py and I don't know how to propagate it.
cc @domenkozar

I think it's supposed to be --install-option="--cpp_implementation" since we're using pip to install the wheel. https://github.com/google/protobuf/issues/539

Fixed in #13165


Library doesn't compile with project on IOS

I get the below error message when I try to compile my project in IOS. The project works fine on android. I have followed the steps as given in the installation section.

Hi @sadhwaniyash6 , thank you for your report. Could you try install the latest version 0.6.1 from npm ?  According to your screenshot, seems the package you've installed is not latest version the latest RNFetchBlobResp.h should look like this. I believe the bug has already been fixed on 0.5.7. This bug can also be fixed by upgrade your Xcode to version >=7.3, please see #31 #30 for more information.
If the error still occurs after upgrade the package and Xcode, please let me know, thank you.

Closing this Issue closed due to no response from author, besides this issue should have been fixed in previous release.


Allow sending customStyleMap to convertFromHTML

I tried digging into convertFromHTML and found no way to pass something which will read inline styles.
The function I found that parses inline styles doesn't receive any outside mapping and maps only: bold, italic, underline and strikethrough.
My wish is to use the Colorful Editor example together with convertFromHTML.
Is there something I'm missing?
Can the DOMBuilder or blockRenderMap arguments help me with that?

Yeah, this is pretty restrictive right now. I think it would make sense to have better control over inline style handling for pastes. @clowbrow, any chance you've spent time thinking about this?


[DUE: SAT] Go through app to ensure consistent wording

User story

As a user
I want / need to use an app with consistent wording
So that I am helped to become familiar with the digital landscape of the product

Further description:
We want to make sure that we are consistent in wording. Consistence also covers minor wording improvements that sound more natural, spelling mistakes, improved sentence structure and more meaningful words.
Acceptance criteria

[ ] Go through the app on iOS and identify inconsistencies in wording
[ ] Go through the app on Android and identify inconsistencies in wording
[ ] Provide a summary of identified inconsistencies in the comments of this issue and already suggest an improvement.

Definition of done (DoD)

It builds without errors
code has been reviewed
Build has been made and deployed on a testing environment
Code coverage >75%
All acceptance criteria were met
Documentation updated whenever needed within planning documents (Tabs: Bill of materials, Documentations)
Branch/feature was merged into dev and then main

DoD general criteria

Feature has been fully implemented
Feature has been merged into the mainline
All acceptance criteria were met
Product owner approved features
All tests are passing
Developers agreed to release

This is the structure to use for the result presentation of this issue:

Location of inconsistency
Description of inconsistency
On iOS?
On Android?
Suggested improvement

Placeholder
Placeholder
[Y or N]
[Y or N]
Placeholder

Placeholder
Placeholder
[Y or N]
[Y or N]
Placeholder

Placeholder
Placeholder
[Y or N]
[Y or N]
Placeholder

Placeholder
Placeholder
[Y or N]
[Y or N]
Placeholder

Placeholder
Placeholder
[Y or N]
[Y or N]
Placeholder

Placeholder
Placeholder
[Y or N]
[Y or N]
Placeholder

Placeholder
Placeholder
[Y or N]
[Y or N]
Placeholder

@Interperle and @domi1504: Please bear in mind that you have to do this issue after the merge tomorrow evening and before our next meeting on Saturday.

Location of inconsistency
Description of inconsistency
On iOS?
On Android?
Suggested improvement

Title of Strength Search/Map Search page and Label in Bottom Menu
The titles on the pages doesn't contain a blank/space and the Labels in the bottom Menu does (e.g. 'StrengthSearch' vs 'Strength Search'
Y
[Y or N]
Use the wording without Blank

Not really a wording issue, but how about disabling landscape orientation?

Location of inconsistency
Description of inconsistency
On iOS?
On Android?
Suggested improvement

Select device button
If no device selected: "Click here to select a device", else: "Click to select a different device"
[Y or N]
Y
Change text to "Click to select a device"

Decisions made in team meeting:

Strength Search --> Blank between both words, for both the heading and the appearing popup when signal is received.
Name the "AboutPage" also in the backend "HomePage" --> No change here!
Disable landscape mode for both Android and iOS
"Click to select a different device"


Issues checkpointing optimizer state using Optax, nnx.Optimizer, and Orbax

I am running into an error while trying to checkpoint an Optax optimizer state, wrapped as an nnx.Optimizer, using the Orbax checkpointing library.
ValueError: Unsupported type: <class 'flax.nnx.training.optimizer.OptArray'> for key: ('0', 'count'). Supported types are (<class 'int'>, <class 'float'>, <class 'numpy.ndarray'>, <class 'jax.Array'>).

I am using packages:

flax 0.10.2
jax 0.4.36
jax-cuda12-pjrt 0.4.36
jax-cuda12-plugin 0.4.36
jaxlib 0.4.36
optax 0.2.4
orbax-checkpoint 0.10.2

Minimal repro:
from flax import nnx
import numpy as np
import orbax.checkpoint as ocp
import optax
import os
import pathlib

class MyModel(nnx.Module):
  def __init__(self, rngs):
    self.linear = nnx.Linear(in_features=4, out_features=1, rngs=rngs)

  def __call__(self, x):
    return self.linear(x)

rngs = nnx.Rngs(0)
model = MyModel(rngs)

tx = optax.adam(1e-3)
optimizer = nnx.Optimizer(model, tx)

checkpointDir = pathlib.Path('/tmp/my-checkpoints/')
checkpointer = ocp.StandardCheckpointer()

checkpointer.save(checkpointDir, optimizer.opt_state, force=True)

I see yall have documentation about using Orbax for model checkpointing, but don't see any official info about optimizer state checkpointing.
I see an older github issue where the Optax folks recommended cloudpickle: https://github.com/google-deepmind/optax/discussions/180
I tried adding a custom serialization/deserialization for nnx.training.optimizer.OptArray via https://orbax.readthedocs.io/en/latest/guides/checkpoint/custom_handlers.html#custom-serialization-deserialization, but then just ran into the next error:
ValueError: TypeHandler lookup failed for: type=<class 'flax.nnx.training.optimizer.OptVariable'>

Maybe I'd need to also add a type handler for OptVariable as well as maybe even Variable? That seems quite annoying. Am I missing something?
Thanks for your time!

It seems that also adding a type handler for nnx.training.optimizer.OptVariable is sufficient. I'd rather not need to look into the details of these classes to write serializers/deserializers though.

@SandSnip3r
Im hoping I could ask if you could share what sounds like the solution was. I also have been having issues with this.

Hey @SandSnip3r, try checkpointhing the State instead:
checkpoint = nnx.state(optimizer)
# optional but works better
checkpoint = checkpoint.to_pure_dict()

checkpointer.save(checkpointDir, checkpoint, force=True)

Then to load it you
checkpoint = checkpointer.load(...)
nnx.update(optimizer, checkpoint)

Thanks @cgarciae! I'll give this a try when I get time.
Why does .to_pure_dict() work better?
What about the arguments to checkpoint.load? The checkpointer requires a structure to load the data into, right? Like an abstract tree state? What would I use for this?

Why does .to_pure_dict() work better?

It removes the VariableState objects that contain metadata and leaves only the Arrays. It also makes checkpoint a pure dict which is usually easier to serialize.

The checkpointer requires a structure to load the data into

I think if you have a pure dictionary you don't need the target structure. Else just recreate the checkpoint structure using nnx.eval_shape e.g.
abstract_optimizer = nnx.eval_shape(lambda: create_optimizer())
target = nnx.state(abstract_optimizer).to_pure_dict()
...

@cgarciae Honestly great explanation. Based on this conversation I was able to make this. Its not perfect but this works pretty well.
class CheckpointManager:
    def __init__(self, checkpoint_dir: str, keep_n: int = 3):
        self.checkpoint_dir = checkpoint_dir
        self.checkpointer = orbax.checkpoint.PyTreeCheckpointer()
        self.keep_n = keep_n

    def _cleanup_old_checkpoints(self, step: int):
        # Get all checkpoint steps except 'best'
        checkpoints = []
        for path in self.checkpoint_dir.glob("model-*"):
            if "best" not in str(path):
                try:
                    ckpt_step = int(path.name.split("-")[-1])
                    checkpoints.append(ckpt_step)
                except ValueError:
                    continue

        # Sort and remove old checkpoints while keeping newest n
        checkpoints.sort(reverse=True)
        for old_step in checkpoints[self.keep_n :]:
            for prefix in ["model", "optimizer", "metrics"]:
                path = self.checkpoint_dir / f"{prefix}-{old_step}"
                if path.exists():
                    shutil.rmtree(path)

    def save_model(self, model: Any, step: int, is_best: bool = False):
        state = nnx.state(model).to_pure_dict()
        self._save(obj=state, filename=f"model-{step}")
        if is_best:
            self._save(obj=state, filename="model-best")
        self._cleanup_old_checkpoints(step)

    def save_optimizer(self, optimizer: nnx.Optimizer, step: int):
        state = nnx.state(optimizer).to_pure_dict()
        self._save(obj=state, filename=f"optimizer-{step}")

    def save_training_state(self, step: int, total_tokens: int, metrics: dict = None):
        metrics = {"step": step, "total_tokens": total_tokens, "metrics": metrics}
        self._save(obj=metrics, filename=f"metrics-{step}")

    def restore_model(self, model_cls: Any, model_config: dict, mesh: Mesh, step: int) -> Any:
        abs_model = nnx.eval_shape(lambda: model_cls(**model_config, rngs=nnx.Rngs(0)))
        abs_state = nnx.state(abs_model).to_pure_dict()

        target = jax.tree.map(
            lambda a, s: jax.ShapeDtypeStruct(a.shape, a.dtype, sharding=s),
            abs_state,
            nnx.get_named_sharding(nnx.state(abs_model), mesh),
        )

        state = self._restore(filename=f"model-{step}", target=target)
        model = nnx.merge(abs_model, state)
        return model

    def restore_optimizer(
        self, model: Any, train_config: TrainingConfig, step: int
    ) -> nnx.Optimizer:
        tx = create_tx(train_config) # function where I construct my optax optimizer
        abs_optimizer = nnx.eval_shape(lambda: nnx.Optimizer(model, tx))
        abs_state = nnx.state(abs_optimizer).to_pure_dict()

        state = self._restore(filename=f"optimizer-{step}", target=abs_state)
        optimizer = nnx.Optimizer(model, tx)
        nnx.update(optimizer, state)
        return optimizer

    def restore_training_state(self, step: int) -> dict:
        """Restore training metrics and token count."""
        return self._restore(filename=f"metrics-{step}")

    def _save(self, obj: Any, filename: str):
        path = self.checkpoint_dir.absolute() / filename
        save_args = orbax_utils.save_args_from_target(obj)
        self.checkpointer.save(str(Path(path)), obj, save_args=save_args)

    def _restore(self, filename: str, target: Any = None) -> Any:
        path = self.checkpoint_dir.absolute() / filename
        return self.checkpointer.restore(str(Path(path)), target=target)

As described, this also works without using to_pure_dict() by simply changing the last line of my repro:
checkpointer.save(checkpointDir, nnx.state(optimizer), force=True)

Then restoring can be done with
tx = optax.adam(1e-3)
optimizer = nnx.Optimizer(model, tx)
abstractOptStateTree = jax.tree_util.tree_map(ocp.utils.to_shape_dtype_struct, nnx.state(optimizer))
optimizerState = checkpointer.restore(checkpointDir, abstractOptStateTree)
nnx.update(optimizer, optimizerState)

Thanks a lot, @cgarciae.
What do you think about adding a small blurb about saving & restoring optimizer states in the flax documentation section about checkpointing? https://flax.readthedocs.io/en/latest/guides/checkpointing.html#save-checkpoints I think this would be nice, especially since flax.nnx is offering an API for optimizers.


Get IP from within handler

Is there a way to find out the IP of the other end from within a connection handler?  I'm a new to GO but not seeing how this is possible as-is.

golang-p2p has not the functionality.
But I am working under the next major version and I am about to add it

"Is there a way to find out the IP of the other end from within a connection handler? I'm a new to GO but not seeing how this is possible as-is."

I don't know if you discover how, but I think you could be:

(local) Network discovery to identify active IP addresses
For each active IP address, send a request to the endpoint
Wait for the response to determine if the platform is installed


Hide Some Sliders with jQuery

Let me tell you that it's a great dynamic library for sliders, but I tried to hide some sliders using jQuery.hide() method but found that sliders (cover flow in my case) have been resized to a smaller size and the hidden sliders place still being but the sliders itself is hidden. I have searched found that I should remove (swiper-slider) class, I did that but something strange is here: the slider does not respond to my code until I resize the document.Can you make hide method in your library to able us to hide any slide with a certain class or data attributes?
My code is here :
$('.vc_automation').hide();
//                Swiper Slider Editing
if($(".swiper-slide").is(".vc_automation")){
$(".vc_automation").removeClass("swiper-slide");
};

something strange is here: the slider does not respond to my code until I resize the document

Swiper needs to update it size to do the proper things, try to call its .update method when you hide/show slides or try to enable observer and observeParents parameters

swiper.update()


Update melodic readme to only reflect ROS 1

Small documentation tweak.  I think after this, we should also make ros2 the default branch.
Signed-off-by: Michael Carroll<EMAIL_ADDRESS>

CI is in a bad state, but I don't believe that it has anything to do with updating READMEs


Added Hot Fix For Shipment #7281

Issue Reference
Link: https://github.com/bagisto/bagisto/issues/7281

@devansh-webkul
This is working fine.


App Delivery pre-day at Kubecon Chicago

This issue is to track planning for coordinated pre-day events for CNCF's app delivery-related projects at Kubecon Chicago.
I think we should try to coordinate the project meetings to make them more productive for all of us by enabling projects to learn from each other. Facilitating cross-project pollination and collaboration should be a focus at Kubecon when we actually have people together!
To help plan, here's a list of the project meetings from Amsterdam: https://kccnceu2023.sched.com/overview/type/Project+Meeting
And from Detroit: https://kccncna2022.sched.com/overview/type/Project+Meeting
As a strawman for debate: let's run an 8-hour App Delivery meeting on Monday November 6 and give ~30 minutes to each delivery-related CNCF project (that wants to) to present their latest innovations and answer questions from other projects. Let's also include time for lightning talks on new sandbox projects and ideas; and time for each WG to talk about their domains.
The target audience would be project and TAG maintainers, not the general dev/ops audience at the rest of Kubecon.
We could start by reaching out to the delivery projects that held meetings in Amsterdam such as OpenGitOps, OpenFeature, Buildpacks, Knative, Backstage, LitmusChaos, Flux, Harbor, Carvel, and Keptn and asking them if they'd like to have 30-45 minutes in a shared program rather than their own meetings.
I started a related discussion in Slack at https://cloud-native.slack.com/archives/CL3SL0CP5/p1685976894161329 if you prefer to chime in there. Thanks!

A suggestion was to hold project meetings on Friday November 10 after Kubecon rather than Monday November 6 before Kubecon. The idea would be to make the pre-day about end users, and the post-day about maintainers. What do folks think?

I really appreciate the goal of supporting both end users and maintainers as a part of the KubeCon activities.
IME KubeCon raises a lot of really interesting maintainer questions/opportunities in large part because of the end user conversations. But also, the framing of those conversations can have a bit impact.
For that reason I am really torn on moving the maintainer conversations to the end. My experience in Amsterdam bringing people together to raise the goals of this maintainer group for the next 6-12 months energised me to seek out conversations at the event that would help me learn more about use cases and interests in that area. At the same time, I wonder if that generated a bit of an echo chamber and allowing an end of conference get together would have let the needs of the community surface better.
All of that to say, I don't know 😅 But maybe there is a way to have a short meal/coffee/open space on the opposite day to the one where we invest most heavily in maintainer conversations so as to counteract the risks I raised?

Hey all, as mentioned at the Platforms WG call this week i have had some people raise interest in "Birds of a Feather (BoF) style conversations in Chicago. The topics raised so far are IMO TAG level chats (secrets w/GitOps, patterns for "package managing" in Kubernetes).
One idea is to set a meeting place during the conference that takes advantage of lunch tables out of hours for something like this.
WDYT both about the concept and if you think we should go forward with something like this how to implement it successfully to raise the right topics with a reasonable group of interested people given the amount of events during the week.

That's a great idea @abangser
Another idea could be taking advantage of the project pavilion and rotating through different project booths? Like a mini-TAG project pavilion crawl at set times other projects are available.

That's a great idea @abangser
Another idea could be taking advantage of the project pavilion and rotating through different project booths? Like a mini-TAG project pavilion crawl at set times other projects are available.

Love it! Given we are trying to get more involved in the project reviews etc, maybe we could get a sponsor for a prize if people get stamps from every project? 🤔

Here's an update on what we've been allocated. Wanted to share this info so others can help with planning - thank you all!
First, TAG App Delivery has been allocated a meeting room at the conference center on Monday from 8-12pm (not 8 hours as I proposed above :laughing:). The meeting space will hold about 20 people and is intended for us to discuss work in our TAG, WGs and governed projects. Let's discuss the agenda here, such as soliciting some of our projects to come chat with us and each other; and getting updates from WGs like Platforms and Artifacts.
Second, the TAG will also will have a kiosk in the project pavilion in the PM; this kiosk includes a monitor, a small desk and some display space. We can have small presentations there and share info about the TAG, WG and governed projects.
Here are the times for the booth, I'll be looking to folks to help staff it:

Tuesday, November 7: 3:30 - 8:00 PM
Wednesday, November 8: 2:00 - 5:00 PM
Thursday, November 9: 12:30 - 2:30 PM

Thanks again for your help!


with the latest version, examples/list-gnp/main.go compiling fails

The version I am using: v0.0.0-20230125230755-7aed9df0e47b
$ go mod tidy
go: finding module for package k8s.io/client-go/tools/clientcmd
go: finding module for package k8s.io/apimachinery/pkg/apis/meta/v1
go: finding module for package github.com/projectcalico/api/pkg/client/clientset_generated/clientset
go: found github.com/projectcalico/api/pkg/client/clientset_generated/clientset in github.com/projectcalico/api v0.0.0-20230125230755-7aed9df0e47b
go: found k8s.io/apimachinery/pkg/apis/meta/v1 in k8s.io/apimachinery v0.26.1
go: found k8s.io/client-go/tools/clientcmd in k8s.io/client-go v0.26.1

$ go mod vendor

$ go run .
# github.com/projectcalico/api/pkg/client/clientset_generated/clientset/typed/projectcalico/v3
vendor/github.com/projectcalico/api/pkg/client/clientset_generated/clientset/typed/projectcalico/v3/projectcalico_client.go:19:2: undefined: BlockAffinitiesGetter
vendor/github.com/projectcalico/api/pkg/client/clientset_generated/clientset/typed/projectcalico/v3/projectcalico_client.go:26:2: undefined: IPAMConfigurationsGetter
vendor/github.com/projectcalico/api/pkg/client/clientset_generated/clientset/typed/projectcalico/v3/projectcalico_client.go:48:51: undefined: BlockAffinityInterface
vendor/github.com/projectcalico/api/pkg/client/clientset_generated/clientset/typed/projectcalico/v3/projectcalico_client.go:49:9: undefined: newBlockAffinities
vendor/github.com/projectcalico/api/pkg/client/clientset_generated/clientset/typed/projectcalico/v3/projectcalico_client.go:76:54: undefined: IPAMConfigurationInterface
vendor/github.com/projectcalico/api/pkg/client/clientset_generated/clientset/typed/projectcalico/v3/projectcalico_client.go:77:9: undefined: newIPAMConfigurations

Am I missing something?
By looking into the code, I didn't find the BlockAffinitiesGetter definition.

@mgleung Is it related with your latest commit? Appreciate if you can help to check.

@zongzw seems like something didn't update properly (since my last update should have added BlockAffinities). Let me take a look.

Also seeing this issue

@zongzw @danieljmt sorry for the delay. I just pushed some of the missing files so this should be fixed now. Please let me know if you still see the issue


Sanitize ext_authz path_prefix

Commit Message: ext_auth: Sanitize http path_prefix
Additional Description: Documentation does not indicate that path must start with '/' then in case a user set the config without it, it is added, if not 404 is return due to address concatenation issue ex: server:9000prefix/
Risk Level: Low
Testing: yes
Docs Changes: n/a
Release Notes: n/a
Platform Specific Features: n/a

/retest

/retest


Is this available for Vue 3?

I just wonder if there are any plan to support vue 3?

Hi!
Yes. It will be a new package at the end of July

Duplicates #46


DefaultMQProducter.start()注释

What is the purpose of the change
XXXXX
Brief changelog
XX
Verifying this change
XXXX
Follow this checklist to help us incorporate your contribution quickly and easily. Notice, it would be helpful if you could finish the following 5 checklist(the last one is not necessary)before request the community to review your PR.

[x] Make sure there is a Github issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[x] Format the pull request title like [ISSUE #123] Fix UnknownException when host config not exist. Each commit in the pull request should have a meaningful subject line and body.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Write necessary unit-test(over 80% coverage) to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
[x] Run mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle to make sure basic checks pass. Run mvn clean install -DskipITs to make sure unit-test pass. Run mvn clean test-compile failsafe:integration-test  to make sure integration-test pass.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

@Xuzf880707 感谢贡献，但是有两个问题：
（1）首先如果需要提PR，应该先提一个ISSUE，然后与这个ISSUE关联。
（2）另外，补充注释的话建议在自己的分支里面里面补充，然后分享出来，而且rocketmq不建议使用中文。


Doesn't work. Nothing happens.

I'm using Pop!_OS 22.04 / Ulancher v5.15.3
I have created a personal access token in Asana (50 chars, starts with a single digit followed by a forward-slash) which is registered in the /apps section of my Asana account)
I have installed the extension, and pasted the same token into Ulancher, under Extensions > Asana > API Token.
I have quit and restarted Ulauncher.
Opening UL and typing a  opens the menu "create a task", "search projects" etc, but nothing happens with any of these options-- no indication that it is connected with my account in any way. I can only assume the connection/authorization has failed for some reason. I've no idea if any errors are logged, or where to look for them (?)
Pity, I was really looking forward to using what seemed like a great extension :/

I guess this project is dead, sadly :(

I guess this project is dead, sadly :(

@inigopascall The project was dead but together we have revived it ;) See here https://github.com/cacciaresi/ulauncher-asana/pull/4


How to disable vertical bouncing?

I can't find the way to do it, I only want to use the horizontal scrolling. Thanks, by the way JTAppleCalendar is so great!

@wilsonbalderrama what version of the calendar are you using? Ther should not be any "bouncing" :/

hey @patchthecode thanks for the quick response, I'm using the 6.0.2 version.
I've copied from the README this sentence into my podfile:
pod 'JTAppleCalendar', '~> 6.0'

Can you put this into your podfile instead? Many fixes have been done.
  pod 'JTAppleCalendar', :git => 'https://github.com/patchthecode/JTAppleCalendar.git'

I've put it and now I have the 6.1.1 version and still see the vertical bouncing.

i'll be online for some time today. Can you join me here -> https://gitter.im/patchthecode/JTAppleCalendar

I had to apply this solution https://patchthecode.github.io/CommonProblems/, to my root view controller that is a UITabBarController afterwards everything was ok. thank you @patchthecode for pointing me to the link :)


Bug on start

latest.log
The mod does not allow the server to start

Hi!
Fix in 1.3.1 version.
It was just a misregistration of some mixins.
Now the server should launch correctly!


Add py.typed marker file for PEP 561 typing

This is so that mypy will check the types declared in this project instead of complaining about missing types.

Thanks @kuimono!


Update translations

Once again, translations need to be updated. @RyDroid, @smarquespt and @pejakm, please, update the translations or leave a comment if you can not do so.
Status:

[X] French
[ ] Portuguese
[x] Serbian

@smarquespt, this is the last one before 2.2.0 release. Please, will you do it ? Let me know even if you are not going to do it.

Sorry for late reply. Here(http://ge.tt/5YEaT4R2/v/0) it is.

Thank you guys for doing this once again.


Add sorting to landuse intercut.

Refs #198.
Also adds missing config from #142, also updated to do the sorting based on the existing order parameter.
@rmarianski could you review, please?

:+1:


Batches for text only branches is wrong too

Following this issue https://github.com/InternLM/InternLM-XComposer/issues/404
For some reason, we I feed text only batch with per_gpu_batch =1 and batch_size = 5, the modeling.py only accepts one sample.
 === forward should be of shape 1 x BS ===  ==== data_mix.py ====
5 492 1
 === forward should be of shape 1 x BS ===  ==== data_mix.py ====
5 637 1
=== forward should be of shape 1 x BS ===  ==== modeling.py ====
1 492 1
=== forward should be of shape 1 x BS ===  ==== modeling.py ====
1 637 1

For image inputs
 === forward should be of shape 1 x BS ===  ==== data_mix.py ====
5 11485 1
 === forward should be of shape 1 x BS ===  ==== data_mix.py ====
5 15905 1
=== forward should be of shape 1 x BS ===  ==== modeling.py ====
1 5 11485
 === forward should be of shape 1 x BS ===  ==== data_mix.py ====
5 15905 1
=== forward should be of shape 1 x BS ===  ==== modeling.py ====
1 5 15905
 === forward should be of shape 1 x BS ===  ==== data_mix.py ====
5 11485 1
=== forward should be of shape 1 x BS ===  ==== modeling.py ====
1 5 15905

I found this is the issue with collator

In the text_only model and text_image mode, encoder/or collator is inconsistent, can we unify them?
the text-image mode requires the input to be [1, bs, SEQLEN], while the text-only mode requires input to be [1, SEQLEN], can we unify them?
I understand in MLLM, images in batches requires a nested batch, I would suggest the outer batch size, the batch per gpu is always 1.
    def forward(self,
                input_ids: torch.LongTensor = None,
                attention_mask: Optional[torch.Tensor] = None,
                position_ids: Optional[torch.LongTensor] = None,
                past_key_values: Optional[List[torch.FloatTensor]] = None,
                inputs_embeds: Optional[torch.FloatTensor] = None,
                labels: Optional[torch.LongTensor] = None,
                use_cache: Optional[bool] = None,
                output_attentions: Optional[bool] = None,
                output_hidden_states: Optional[bool] = None,
                return_dict: Optional[bool] = None,
                **kwargs) -> Union[Tuple, CausalLMOutputWithPast]:
        r"""
        Args:
            labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
                Labels for computing the masked language modeling loss. Indices should either be in `[0, ...,
                config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored
                (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`.
        Returns:
        """

        samples = kwargs.get('samples', None)
        if samples:
            infer_mode = samples.get('infer_mode', 'base')
            if samples['data_type'][0] == 'text':
                has_img = False
            elif samples['data_type'][0] == 'multi':
                has_img = True
            else:
                raise NotImplementedError

            # encode text
            text = samples['text_input']
            # encode image
            if has_img:
                image = samples['image'][0]
                bs = len(samples['text_input'][0])
                image_nums = []
                temp_image = []
                for im in image:
                    if type(im) is list:
                        image_nums.append(len(im))
                        temp_image.extend(im)
                    else:
                        image_nums.append(1)
                        temp_image.append(im)
                image = temp_image
                assert type(image) is list and len(image_nums) == bs

                to_regress_embeds, attention_mask, targets, im_mask = self.interleav_wrap(
                    image, text, image_nums)
            else:
                to_regress_tokens, targets = self.text2emb(
                    text, add_special_tokens=True)
                to_regress_embeds = self.model.tok_embeddings(
                    to_regress_tokens.input_ids)
                attention_mask = to_regress_tokens.attention_mask
                im_mask = torch.zeros(to_regress_embeds.shape[:2]).cuda()


Cannot use GitHub backend

I try to use the GitHub backend but sveltiacms just shows the ability to "Work with Local Repository" instead of a login button.
My config.yml (all capitalized words normally are replaced with my credentials):
backend:
name: "github"
repo: "ME/MYREPO"
branch: "main" # Branch to update (optional; defaults to master)
base_url: "https://api.MYURL.com"
auth_endpoint: "auth"

Thanks for reporting the issue. Hmm, not sure what’s happening. I just tested with my client’s Sveltia CMS instance, but it showed the Sign In with GitHub button as expected.
Just in case: Are you using Sveltia CMS remotely, right? If you’re running it locally, the Work with Local Repository button will always appear. (Technically you can use GitHub directly from a local instance, but there is no UI for that)

Ah got it! Then it’s rather better to work with a local repository because changes will not be committed 🙂
Well, I’ll create a demo site once the “test” backend is supported!


Track the missing 2% success after switching to sorted Vec<IpNet> and binary_search

After we switched to use a binary search on each AS's routes for filter_as_set, the number of reported errors increased.

[ ] Make a linear search version and find example missing successes.

Test using linear search shows no difference. The cause should be something other than the binary search and break on !is_sibling.

Using filter_as_num from 33e40445 does not make a difference neither.

Cannot find the cause, then. Closing this.


vscode extension breaks error reporting when editing non-Civet TypeScript files

Notice below how, as I press "space" before a squiggly red underline, an old copy of the error shifts around.
This is while editing a TypeScript file in a completely Civet-free project.
https://github.com/DanielXMoore/Civet/assets/643799/60ea9758-0c92-40fe-8ffb-8bd95a2e4fd7
The only way to get rid of the extra copy of the error is to restart VS Code. (Restarting the TS server doesn't help.)
Very surprising to me that the Civet extension messes up non-Civet files!
Thanks.

This is definitely a bug in the Civet LSP and we should work to fix it.
The reason Civet extension affects non-Civet files is that the TypeScript Server doesn't provide a capable enough plugin system to resolve non-ts files (like .civet) so our LSP needs to be enable for all files that could import .civet files as well as .civet files.
A quicker fix than restarting VSCode would be to either restart the extension host or restart the Civet LSP.
Thanks for the report!

I think this is fixed by https://github.com/DanielXMoore/Civet/pull/1044


Specify extra on.push.paths in chunks configuration

If we add the code-style entries as expected on.push.paths, that would let us assert an exact match for the on.push.paths instead of just checking that the expected entries are present. That would be a lot more foolproof as otherwise we might have extra incorrect entries.
We could specify the extra expected entries in .github/config/chunks.yaml.

This isn't needed as only Maven extra entries are validated after #465


CLI morphometrics doesn't generate index images

Describe the bug
Using the axondeepseg_morphometrics script and specifying folders with the -i option produces an unwanted behavior.

we get the following message (which is peculiar because the xlsx file is saved and look normal)

Cannot save morphometrics data in file 'paasu_img2_axon_morphometrics.xlsx'.`

"index" images are not generated

The first point was reported by @mathieuboudreau last week when he tried to run morphometrics via the CLI with subfolders, similar to what is described here.
When running the script directly in the folder containing the segmentation, we do not encounter this problem and the index files are generated as expected.
Platform (please complete the following information):

Linux 5.10.52-3-rt47-MANJARO
AxonDeepSeg v3.2.0

I'm looking at the section of code that starts here.
It's a bit hard to follow how variables with paths are used. I admit I haven't tested the -i option when implementing the index images.


[WIP] more JaggedArray.getitem cases in awk-cpp

To work with None input in python strides (and to actually unpack a slice, which I don't think is supported in py::slice), I receive the where input on the python side and then send that data to C++ with proper default values. After many many trials and errors, JaggedArray slicing should work on the python side exactly how a normal array would.
Also, I found out that I don't need to add any code to handle negative strides, because slicing an array with negative strides automatically has its start pointer move to the end, which is really convenient:
>>> test = numpy.arange(5)
>>> test[::1].ctypes.data
109515352
>>> test[::-1].ctypes.data
109515368

That means I already properly handle negative strides, because I step through arrays like this:
int N = starts_info.strides[0] / starts_info.itemsize;
for (ssize_t i = 0; i < starts_info.size; i++) {
    if (starts_ptr[i * N] < 0) {
        // ...
    }
}

It's great to hear that the code doesn't have to change for negative indexes, but do you have an explicit test?

@jpivarski I've been working on implementing fromiter() so I can start copying over identical tests from test_numba to test_cpp, but I came across this problem:
>>> test = awkward.cpp.JaggedArray.fromiter([[[1, 4, 5, 4], [1, 5, 5]], [[], [3]], []])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: The maximum of starts for non-empty elements must be less than the length of content

The above array looks like it should be a valid jagged array, but the specifications document does say that the maximum of starts must be less than the length of content.
Is that okay as it is, or should I change something to allow that sort of jagged array to be created?

I've got another problem:
Right now, I'm overriding the __getitem__() method on the python side to be able to unbox slices and accept None type in slices. However, calling fromcounts() creates a new JaggedArray instead of the python-side JaggedArrayCpp which inherits from it. As a result, that object does not override the __getitem()__ function anymore.
I would say my options are:

Implement virtual functions through pybind11 so that __getitem__() calls from the python side
Figure out how to unpack a py::slice (if there even is a way) in C++, and move everything over to the C++ side

I feel like 2 would be much simpler, but I don't know if it's even possible, from what I've seen. The documentation for the py::slice class is in pybind11/pytypes.h.
Here is some code showing what the issue is:
>>> a = awkward.cpp.JaggedArray([0, 3, 4, 4], [3, 4, 4, 6], numpy.arange(8))
>>> a
<JaggedArray [[0 1 2] [3] [] [4 5]] at 0x71c0940>
>>> b = awkward.cpp.JaggedArray.fromcounts([3, 1, 0, 2], numpy.arange(8))
>>> b
<JaggedArray [[0 1 2] [3] [] [4 5]] at 0x71c0b68>
>>> a[0:2]
<JaggedArray [[0 1 2] [3]] at 0x71c0e38>
>>> b[0:2]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: __getitem__(): incompatible function arguments. The following argument types are supported:
    1. (self: awkward.cpp.array.array_impl.JaggedArray, arg0: int) -> object
    2. (self: awkward.cpp.array.array_impl.JaggedArray, arg0: int, arg1: int, arg2: int) -> object

Invoked with: <JaggedArray [[0 1 2] [3] [] [4 5]] at 0x71c0b68>, slice(0, 2, None)

Where a is a JaggedArrayCpp (Python side) which intercepts __getitem__ before being passed to C++, and b is a JaggedArray (C++ superclass).

@jpivarski I've been working on implementing fromiter() so I can start copying over identical tests from test_numba to test_cpp

Don't start on fromiter. It's a major project that will involve all of the awkward classes, and whether the fromiter implementation is in Python or C++ is independent of whether the awkward classes are in Python or C++. For example, using the existing fromiter, we can already populate custom classes:
>>> import awkward, awkward.numba
>>> awkward.fromiter([[[1, 4, 5, 4], [1, 5, 5]], [[], [3]], []], awkwardlib=awkward.numba)
<JaggedArrayNumba [[[1 4 5 4] [1 5 5]] [[] [3]] []] at 0x7f0bd26c6048>

The awkwardlib parameter determines which module is used as a source of JaggedArray, Table, etc. Without specifying anything, it's awkward.JaggedArray, awkward.Table, etc., but since I put awkwardlib=awkward.numba, it's awkward.numba.JaggedArray, awkward.numba.Table, etc. That's why we're putting all of the classes into awkward.cpp with generic names.
Most or all of the functions that aren't attached to an awkward array but produce awkward arrays have an awkwardlib parameter (so that arrays read from files and such can be any flavor the user needs), and so does uproot.

Oh, but your error is probably due to the fact that the last inner array is empty. Empty arrays can be represented by any start, stop pair in which start == stop, but the natural way (and the only way if you're coming from offsets) is for an empty inner array at the end to have stop[i - 1] == start[i] == stop[i] where i is the index of the last inner array. Thus, start[i] is equal to len(content), which you might be excluding because there's no data there.
It doesn't matter that there's no data there because the length of the inner array is zero.
So instead of assert start[i] < len(content), the rule should be assert start[i] == stop[i] or start[i] < len(content).

Implement virtual functions through pybind11 so that __getitem__() calls from the python side
Figure out how to unpack a py::slice (if there even is a way) in C++, and move everything over to the C++ side

I feel like 2 would be much simpler, but I don't know if it's even possible, from what I've seen. The documentation for the py::slice class is in pybind11/pytypes.h.

From those error messages, you need to have a clearer separation between the Python objects and the C++ objects, and they need to go through a boundary layer that wraps and unwraps the Pythonness.
So first off, the C++ function that does getitem and takes different signatures for different index types (e.g. number, slice, ...) should not be named __getitem__ because that's the Python name. You can make your life easier by having a clear distinction between the C++ names and the Python names, so that an error message tells you which side the problem is on.
The function (probably written on the C++ side, but not necessarily) that interprets Pythonic indexes (numbers, slices, arrays, tuples of all of the above) should take a generic py::object and unpack it manually using Python functions. (You can write using Python functions in C++, it's just a little more difficult.) Then, after you've unpacked it and you know whether it's an index or a slice or whatever, you can call the appropriate C++ method.
You might not want to write constructors (like fromcounts) on the C++ side. You've already written counts2offsets on the C++ side (the part that can benefit from C++ speed), maybe then just wrap the resulting starts and stops in a Python object on the Python side.
There are many ways to do this. The two you mentioned don't sound like the easiest. Here are others.

Make the C++ side just a bucket of accelerated functions, which the Python side calls into when it needs a speedup, but most of the objects are defined on the Python side. (That's the approach I was thinking about before this project started.)
Make a C++ JaggedArray class usable on the C++ side, but the Python side wraps all of its entrances and exits with a subclass. (That's the approach we've been trying to follow up to now. It would be nice to just assign to the array.__class__ to change the array's class as you would if it were a pure Python object, but I don't think you can do that to objects made in C++.)
Make a C++ JaggedArray class usable on the C++ side, but the Python encloses it and wraps all of its entrances and exits with an unrelated class (neither superclass nor subclass). By "enclosing," I mean the Python JaggedArrayCpp has one member: the pybind11 jagged array. Every method and property that the JaggedArrayCpp must provide is implemented by calling the C++ object. In that case, the Python-side object should probably also do the Pythonic unpacking, like determining whether a __getitem__ index is a number, slice, etc. and call a specialized getitem (no underscores) C++ method for each. Slices, in particular, would get translated to a three argument method (start, stop, step separately) or maybe six arguments to have a place for boolean hasstart, hasstop, hasstep. Or maybe the Python side fills in a concrete start, stop, and step because it can see the length of the array.

Don't make things difficult on yourself by trying to make it all happen on the C++ side.

I figured out how to unpack slices on the C++ side, and it's so conveniently made that it gives the start, stop, step, and length based on the len() value of the class it's being called on. Because of this, I got to delete a lot of unnecessary code and it really cleaned things up.

I also figured out how to create default arguments in the binding code, so I carried over the other python-side function as well. Now everything is written in C++.

Just to make sure:
In getitem(int array), the code should not allow negative values in the array, right? I can handle negative indices like python if necessary, but from the numba tests, it looks like it's only designed for exact indices.

You'll want to have at least one getitem interface that's extremely lightweight, with no bounds checking or negative index handling (and hopefully the compiler just inlines that), and you'll need to do these checks in another interface (to make sure the user doesn't do something dumb). Where you put them is up to you.

Do you think it's worth it to make the C++ function inherit functions directly from awkward.array.jagged instead of having a python child do it? As shown here, it's possible to have a pybind11-bound C++ class inherit methods from a python class, and that would remove any need to have an in-between now that I have everything implemented on the C++ side.

That example you're pointing to is just invoking Python statements from C++. It's equivalent to just writing the Python, but it might save you from having two classes to carry around.

I think I've decided not to mess with the strategy shown in that other GitHub thread. The most perfect solution for creating a class which implements awkward array in C++ would be to fully recreate every function listed for awkward array in the C++ JaggedArray class, and have the __init__.py file point to that directly rather than the Python JaggedArrayCpp class which inherits from both C++ and Python implementations of jagged arrays.
While that would be nice to see, it would not be possible until far in the future, so until then I'll just continue translating jagged array methods from awkward-array to awkward-cpp.

I've spent a long time trying to figure out (from test_numba) how the argument input works for __getitem__(tuple), and I thought I had it, until I got to the bool array tests and then I got lost.
Here's how I thought it was working:
Taking in a tuple as an argument, the code identifies the type of the [0] index and calls that specific function on the jagged array. For each element it selects through that function, it calls the next index in the tuple. This continues until the end of the tuple. The only time that a returned array has a reduced number of dimensions is when it's calling the single-int getitem function.
Now, I'm pretty sure that was wrong. I wrote out that process on the examples in the bool array test, and I got far different results from the listed output, so I'm more or less back to square one on this.
How exactly should these functions be called? Is it recursive, or just case-by-case? The bool array one is especially stumping me:
def test_numba_getitem_tuple_slice_boolarray(self):
    a = numpy.arange(36).reshape(4, 3, 3)
    a2 = awkward.fromiter(a)
    @numba.njit
    def test1(x, i):
        return x[1:3, i]
    assert test1(a, numpy.array([True, False, True])).tolist() == [[[9, 10, 11], [15, 16, 17]], [[18, 19, 20], [24, 25, 26]]]
    assert test1(a2, numpy.array([True, False, True])).tolist() == [[[9, 10, 11], [15, 16, 17]], [[18, 19, 20], [24, 25, 26]]]
    @numba.njit
    def test2(x, i, j):
        return x[1:3, i, j]
    assert test2.py_func(a, numpy.array([True, False, True]), numpy.array([True, True, False])).tolist() == [[9, 16], [18, 25]]
    assert test2(a2, numpy.array([True, False, True]), numpy.array([True, True, False])).tolist() == [[9, 16], [18, 25]]

@jpivarski What do you think of my most recent commit?
I created a new C header file called CPU_methods.h which implements its own byte swap and native endian handling, along with creating four new structs: C_array_8, C_array_16, C_array_32, and C_array_64. After that, I rewrote offsets2parents_int64 using only variables being passed by reference.
In the C++ file, I made a method called numpy_to_c which converts a NumPy array to a C_array struct (for now just the 64-bit version), and then I replaced the looping content of offsets2parents with offsets2parents_int64.
I believe the C header file is C-compiled, because of the extern "C" { border I put around the contents of the file. If this looks correct, I can start carrying more functions over to C tomorrow.

Also, travis is being picky about data types because I guess my long long int = int64 assumption wasn't universally correct. I'll have to look into hardcoding how many bytes each integer has, rather than using the long, short, and int keywords.

extern "C" doesn't mean "C compiled," it just means that the interface is C (FFI). We can take advantage of that, because it isn't necessary for the functions to be implemented in C, just that they present an interface that I can use in Numba.
For standard integer sizes, you should use the stdint.h header. It corrects C's mistake of having integer sizes be platform-dependent.
Also, there are built-in headers for byte swapping—I just don't know what they're called. In your project, it's more important to find these standards than to correctly write your own, because we'll be using standards in the long run.

The byte-swapping headers for C are all system-dependent (to make it more efficient), so I don't know if it's necessary to find something better than my utility functions.

What about this? https://linux.die.net/man/3/htonl

I think that's just for Linux systems. I looked up where to get that header and the solutions was that Microsoft has its own library for that purpose. <arpa/inet.h> is apparently included in an amd64 package.

Okay, fine. I guess this is simple enough to not need a header. I had just thought it would be in the same place on all systems (instead, we'd probably find ourselves putting in preprocessor directives like, "#if Windows, do this...").
https://stackoverflow.com/a/10346046

Just to make sure: I can design this to only use signed types, right?

I think I need to draw out a better structure for myself to visualize how this is going to be set up, because what I had before got too messy and I made some mistakes with the polymorphism. Generally, I'm thinking the CPU methods should only cover the looping, computation-intensive sections of code, with everything else continuing to exist how it has been in the C++.
In addition to what is in awkward-cpp right now, I would expect there to be a CPU_methods.h header of some sort that contains the C functions, and then another header, CPU_cpp.h or something, that wraps the C in C++ code.
What I covered yesterday and today was good practice, but I was too ambitious and more or less broke my code, so I reverted and will start again from where I was on Monday.

We can focus on just signed types. It doesn't exclude much available range.

@jpivarski I'd like your opinion on the solution I found for this bug:
The issue was that I was taking the pointers from the beginnings of the strides and shape vectors to be used in the c_array struct, but those vectors didn't have a reference and so it would sometimes corrupt the data. I tried just creating vector variables and setting the strides and shape to those variables, but it didn't work. My solution was to copy the values over to arrays and attach those arrays to the c_array struct instead:
struct c_array py2c(py::array input) {
    py::buffer_info info = input.request();

    if (info.ndim > 15) {
        throw std::invalid_argument("Array cannot exceed 15 dimensions");
    }

    char format[15];
    strcpy(format, info.format.c_str());
    ssize_t shape[15];
    std::copy(info.shape.begin(), info.shape.end(), shape);
    ssize_t strides[15];
    std::copy(info.strides.begin(), info.strides.end(), strides);

    struct c_array out = {
        info.ptr,
        info.itemsize,
        info.size,
        format,
        info.ndim,
        &shape[0],
        &strides[0]
    };
    return out;
}

I feel like the copying being done here is too much strain on processing time and memory. Can you think of a more efficient way to deal with this issue?

Also, is this PR good to merge? The tests that are failing are all in test_arrow.py. I don't know why they're failing, but my tests are all okay.

To fix the Arrow tests, do a git merge master. Arrow updated in a backward-incompatible way and I put corrections into master. Then you should get a clean bill of health.
I'll have to look at your Numpy fix later—it's directly relevant for data ownership, which is what you should be focusing on.

@EscottC

The issue was that I was taking the pointers from the beginnings of the strides and shape vectors to be used in the c_array struct, but those vectors didn't have a reference and so it would sometimes corrupt the data.
...
I feel like the copying being done here is too much strain on processing time and memory. Can you think of a more efficient way to deal with this issue?

Okay, I've read this comment and this is not a good solution. You're taking ownership of the data by copying it. Whether this is a performance problem for a specific application depends on the application, but some applications are going to want zero-copy adoption of memory, so that's something we want to learn how to do.
Python defines data ownership in terms of a reference count. In the pure C API, there are Py_INCREF and Py_DECREF macros to increase and decrease this integer, which extension developers have to manage manually. I've been told that one of the big advantages of Cython and pybind11 is that they do this in an automated or semi-automated way. It's certainly a mess when an extension developer doesn't handle the reference count correctly (segmentation faults in Python!)
When an object is created in Python, its reference count increments to 1. If that object is in an expression that never gets assigned to a name (e.g. myvar = make_whatever()), then it gets decremented to 0. If it gets assigned to several names, it gets several increments. When names get deleted with the Python del operator or they go out of scope because they were defined in a function, they get decremented. Some of these names might be attributes of other objects, like myobj.attribute = make_whatever(). Deleting an object decrements all the objects that are attached to it.
When an object's reference count is 0, there is no way to access it. Such a value in C or C++ would be a memory leak, but Python keeps a global list of objects and occasionally checks their reference counts for 0 and deletes those (garbage collection). It's a little more complicated than that for cyclic references: A could have B as an attribute, which has A as an attribute, and so both A and B would have a reference count of 1 even if they're both inaccessible—Python has to do a more computationally intensive search for such cases, but it does them. (In fact, this is a performance tip for Python: avoid cyclic references to make memory management quicker!)
This tutorial describes Python reference counting and introduces the relevant terminology: "new reference," "borrowed reference," and "stolen reference." These are the English terms we use when the language doesn't track ownership for us, as Python does and as C++ shared_ptr does. There's probably a one-to-one relationship between Python's reference counting and C++ smart pointers that pybind11 does automatically, which is precisely what we want to learn. (Remember that in the new architecture, the pure C functions do the work, the C++ layer does object ownership, and the Python object ownership is one-to-one coupled to C++. Your focus is this C++/Python layer, so your project is all about learning how this works.)
Numpy adds another layer to this. A Numpy array is a Python object that points to raw bytes somewhere in memory. The Python object has a reference count, like any other Python object, and it also has an owndata flag to say whether it "owns" the raw bytes. When a Numpy object's reference count is 0 and it gets garbage-collected, it will delete its memory buffer if and only if it owns it.
Here's an example of two Numpy arrays, one owns the data, another does not.
>>> import numpy
>>> a = numpy.arange(10)
>>> b = a[::2]
>>> a.flags.owndata
True
>>> b.flags.owndata
False
>>> a.base, b.base      # base is None or the array that an array is derived from
(None, array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]))
>>> b.base is a         # Python's "is" means they are the same object in memory
True
>>> id(b.base), id(a)   # Python's id(x) is a unique integer for each object
(138444522648560,<PHONE_NUMBER>48560)
>>> id(b.base) == id(a)
True
>>> bcopy = b.copy()
>>> bcopy.flags.owndata
True
>>> print(bcopy.base)
None

Another way that a Numpy array can not own its data is if we got it from somewhere else, some arbitrary bytes that Python generically calls a "buffer." A buffer is not a Python type, but a type in the Python C API—objects that support the buffer protocol are anything that has a pointer to raw bytes somewhere, including strings, bytestrings, and Numpy arrays. Python 3 has a similar concept called a memoryview—I don't know the differences between buffers and memoryviews, but they're pretty similar.
>>> x = b"got this from somewhere"
>>> y = numpy.frombuffer(x, dtype=numpy.uint8)
>>> y
array([103, 111, 116,  32, 116, 104, 105, 115,  32, 102, 114, 111, 109,
        32, 115, 111, 109, 101, 119, 104, 101, 114, 101], dtype=uint8)
>>> y.flags.owndata
False

We will frequently be viewing Apache Arrow buffers as arrays. This is something we want to do a lot, and we want to do it zero-copy. It's one of the selling points for awkward-array.
So in your implementation, you really should not copy the data but take ownership of it. The Numpyish way to do it is to make our objects in C++ be reference-counted Python objects, hopefully by some easy connection between shared_ptr and Python reference counts through pybind11, then to add a reference-counted attribute named base that points back to the original Numpy array. That way, even if the original Numpy array goes out of scope, we have a reference to it named base that will keep Python's garbage collection from erasing that array or the raw bytes it points to.
How exactly that works is up to you, but this is the requirement: own the data without copying it.


Update java.rb

Here is a java patch

https://github.com/caskroom/homebrew-cask/pull/38152#issuecomment-326192869


Build(deps-dev): Bump eslint-plugin-react from 7.29.2 to 7.29.3 (#3029)

Pull request opened by the merge tool on behalf of #3029

Merged in 24a59234019a685cc8f1d5e3e935d60056673c79


LG LSWD100 Signature

Hi, my washer/dryer shows up in HK.
I don't know how I can help you but it would be nice if there will be support for the integrated tank system of the detergent and  softener.

homekit do not provide official supported for washer/dyer accessory at this time, I don't know if have any homekit characteristic can simulate tank system


Upload config to GCS and tensorboard

Adding flag save_config_to_gcs. When it is set to True, MaxText will upload the config params to GCS as config.yml.
Write config params and libtpu_init_args to tensorboard

I've been wanting this feature for some time, thank you!

These changes were merged in https://github.com/google/maxtext/commit/2fdf8a2dfa553b928d52052711219ca9cbcf4ec7. Closing out this PR now.


How to make this plugin work on html.heex templates from Phoenix

Hey, i am using Elixir Phoenix for an Project but this Plugins seems to not really work with heex files.

commenting for visibility! Would also be nice for other embedded languages if erb files for ruby on rails!

I figured it out, I have two separate config files, one to load the plugin and call setup which is lazy loaded
nvim-ts-autotag.lua
return {
  "windwp/nvim-ts-autotag",
  config = function()
    require('nvim-ts-autotag').setup({})
  end,
  lazy = true
}

and then my nvim-treesitter.lua has
return {
  'nvim-treesitter/nvim-treesitter',
  lazy = false,
  priority = 999,
  build = ":TSUpdate",
  cmd = { "TSUpdateSync" },
  dependencies = {
    "windwp/nvim-ts-autotag",
    "tpope/vim-endwise",
    "HiPhish/nvim-ts-rainbow2",
    "nvim-treesitter/nvim-treesitter-textobjects",
  },
  config = function()
    require 'nvim-treesitter.configs'.setup {
      ensure_installed = { "elixir", "heex", "eex", "ruby" },
      autotag = {
        enable = true,
        filetypes = { "html", "xml", "eruby", "heex", "elixir", "embedded_template" },
      },
    }
  end
}(rest of config)

@sharkby7e  could you solve the Issue?

@sharkby7e so i could try to add the autotag to nvimtreesitter

@jpx40 yeah that's what fixed it for me!

@sharkby7e  Can you help me fix my config? It doesn't work yet

@jpx40  sure, post ur config!

@sharkby7e https://github.com/jpx40/dotfiles/tree/main/dot_config/nvim

It's based on LazyVim, so i kinda don't made the most my self.

@jpx40 I think ur missing the treesitter config. Check out my config here https://github.com/sharkby7e/nvim

@jpx40  did the pr you pushed fix this?

I don't know if the commit is right way since templ is an custom file. I figured out that autotag is pre-configured in Lazyvim. The easiest way would be an fork.

hey guys, not sure if this is the right issue for it, but I'm trying to make autotag work with ~H sigil inside ex files, but for some reason it does not. I'm using lunarvim and my current config for autotag looks like this:
  {
    "windwp/nvim-ts-autotag",
    config = function()
      require("nvim-ts-autotag").setup({
        enable = true,
        filetypes = { "html", "xml", "eruby", "heex", "elixir", "embedded_template" }
      })
    end,
    lazy = true,
  },

@Vagab sry, i still have similar problems my knowledge about this plugin and neovim is not good enough to help you. If i figure something out i will help you.


Add authorization directory in Reference.

istio/istio#6377

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: yangminzhu
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: sdake
Assign the PR to them by writing /assign @sdake in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

content/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

See here for how to fix spelling errors: https://github.com/istio/istio.github.io#linting


Make tall code blocks scrollable

Very tall code blocks make the thread hard to follow and requires a lot of scrolling. Ideally, users should just post a Gist link, but they don't. Instead, we could limit the height and make it scrollable.
Example: https://github.com/sindresorhus/gifski-app/issues/12#issuecomment-369313344

@sindresorhus thats a quote > not ``

Perhaps both blockquotes and pre should be scrollable

Yes

How would we determine whats a tall piece of code? I was thinking maybe using the clientHeight of the pre or blockquote element and if the height is greater than (X) we can make it scrollable

No need, we can just add a max-height and overflow: auto via CSS

what do yo guys think is a good number for the max-height?
I tested 30em and it looks good. I think it is big enough for small code snippets as well.

Just make sure that it clips in the middle of a line, so the clipping has a better chance of being noticed.

I am not sure what you mean by clipping in the middle of a line? Do you want me to set a default position of scrollbar to middle so the users can see the scroll bar?

I am not sure what you mean by clipping in the middle of a line

@bfred-it meant to use max height like 30.5em to make only a part of the line visible, so that the user knows that there is more content to scroll (in case of invisible scroll bars, when clipped to a line, the content looks complete even when there is content to scroll).

Testing:

import 'webext-dynamic-content-scripts';
import onAjaxedPages from 'github-injection';
import select from 'select-dom';
import domLoaded from 'dom-loaded';

import markUnread from './features/mark-unread';
import addOpenAllNotificationsButton from './features/open-all-notifications';
import addUploadBtn from './features/upload-button';
import enableCopyOnY from './features/copy-on-y';
import addReactionParticipants from './features/reactions-avatars';
import showRealNames from './features/show-names';
import addCopyFilePathToPRs from './features/copy-file-path';
import addFileCopyButton from './features/copy-file';
// - import copyMarkdown from './features/copy-markdown';
import linkifyCode from './features/linkify-urls-in-code';
import autoLoadMoreNews from './features/auto-load-more-news';
import addOPLabels from './features/op-labels';
import addMoreDropdown from './features/more-dropdown';
import addReleasesTab from './features/add-releases-tab';
import addGistsLink from './features/add-gists-link-to-profile';
import addTimeMachineLinksToComments from './features/add-time-machine-links-to-comments';
import removeUploadFilesButton from './features/remove-upload-files-button';
import scrollToTopOnCollapse from './features/scroll-to-top-on-collapse';
import removeDiffSigns from './features/remove-diff-signs';
import linkifyBranchRefs from './features/linkify-branch-refs';
import hideEmptyMeta from './features/hide-empty-meta';
import hideOwnStars from './features/hide-own-stars';
import moveMarketplaceLinkToProfileDropdown from './features/move-marketplace-link-to-profile-dropdown';
import addYourRepoLinkToProfileDropdown from './features/add-your-repositories-link-to-profile-dropdown';
import addTrendingMenuItem from './features/add-trending-menu-item';
import addProfileHotkey from './features/add-profile-hotkey';
import addYoursMenuItem from './features/add-yours-menu-item';
import addCommentedMenuItem from './features/add-commented-menu-item';
import addToggleFilesButton from './features/add-toggle-files-button';
import addReadmeButtons from './features/add-readme-buttons';
import addBranchButtons from './features/add-branch-buttons';
import addDeleteForkLink from './features/add-delete-fork-link';
import linkifyIssuesInTitles from './features/linkify-issues-in-titles';
import addPatchDiffLinks from './features/add-patch-diff-links';
import markMergeCommitsInList from './features/mark-merge-commits-in-list';
import showRecentlyPushedBranches from './features/show-recently-pushed-branches';
import addDiffViewWithoutWhitespaceOption from './features/add-diff-view-without-whitespace-option';
import preserveWhitespaceOptionInNav from './features/preserve-whitespace-option-in-nav';
import addMilestoneNavigation from './features/add-milestone-navigation';
import addFilterCommentsByYou from './features/add-filter-comments-by-you';
import removeProjectsTab from './features/remove-projects-tab';
import fixSquashAndMergeTitle from './features/fix-squash-and-merge-title';
import addTitleToEmojis from './features/add-title-to-emojis';
import sortMilestonesByClosestDueDate from './features/sort-milestones-by-closest-due-date';
import moveAccountSwitcherToSidebar from './features/move-account-switcher-to-sidebar';
import openCIDetailsInNewTab from './features/open-ci-details-in-new-tab';
import focusConfirmationButtons from './features/focus-confirmation-buttons';
import addKeyboardShortcutsToCommentFields from './features/add-keyboard-shortcuts-to-comment-fields';
import addConfirmationToCommentCancellation from './features/add-confirmation-to-comment-cancellation';
import addCILink from './features/add-ci-link';
import embedGistInline from './features/embed-gist-inline';
import extendIssueStatusLabel from './features/extend-issue-status-label';
import highlightClosingPrsInOpenIssues from './features/highlight-closing-prs-in-open-issues';
import toggleAllThingsWithAlt from './features/toggle-all-things-with-alt';
import addJumpToBottomLink from './features/add-jump-to-bottom-link';
import addQuickReviewButtons from './features/add-quick-review-buttons';
import extendDiffExpander from './features/extend-diff-expander';
import sortIssuesByUpdateTime from './features/sort-issues-by-update-time';
import makeDiscussionSidebarSticky from './features/make-discussion-sidebar-sticky';
import shortenLinks from './features/shorten-links';
import waitForBuild from './features/wait-for-build';
import addDownloadFolderButton from './features/add-download-folder-button';
import hideUselessNewsfeedEvents from './features/hide-useless-newsfeed-events';
import closeOutOfViewModals from './features/close-out-of-view-modals';
import addScopedSearchOnUserProfile from './features/add-scoped-search-on-user-profile';
import monospaceTextareas from './features/monospace-textareas';
import improveShortcutHelp from './features/improve-shortcut-help';

import * as pageDetect from './libs/page-detect';
import {safeElementReady, enableFeature} from './libs/utils';
import observeEl from './libs/simplified-element-observer';

// Add globals for easier debugging
window.select = select;

async function init() {
	await safeElementReady('body');

	if (pageDetect.is404() || pageDetect.is500()) {
		return;
	}

	if (document.body.classList.contains('logged-out')) {
		console.warn('%cRefined GitHub%c only works when you’re logged in to GitHub.', 'font-weight: bold', '');
		return;
	}

	if (select.exists('html.refined-github')) {
		console.warn('Refined GitHub has been loaded twice. If you didn’t install the developer version, this may be a bug. Please report it to: https://github.com/sindresorhus/refined-github/issues/565');
		return;
	}

	document.documentElement.classList.add('refined-github');

	if (!pageDetect.isGist()) {
		enableFeature(addTrendingMenuItem);
	}

	if (pageDetect.isDashboard()) {
		enableFeature(moveAccountSwitcherToSidebar);
		enableFeature(hideUselessNewsfeedEvents);
	}

	if (pageDetect.isRepo()) {
		onAjaxedPages(async () => {
			// Wait for the tab bar to be loaded
			await safeElementReady('.pagehead + *');
			enableFeature(addMoreDropdown);
			enableFeature(addReleasesTab);
			enableFeature(removeProjectsTab);
		});
	}

	if (pageDetect.isUserProfile()) {
		enableFeature(addScopedSearchOnUserProfile);
	}

	enableFeature(focusConfirmationButtons);
	enableFeature(addKeyboardShortcutsToCommentFields);
	enableFeature(addConfirmationToCommentCancellation);
	enableFeature(monospaceTextareas);

	// TODO: Enable this when we've improved how copying Markdown works
	// See #522
	// delegate('.markdown-body', 'copy', copyMarkdown);

	await domLoaded;
	onDomReady();
}

function onDomReady() {
	enableFeature(markUnread);
	enableFeature(addOpenAllNotificationsButton);
	enableFeature(enableCopyOnY);
	enableFeature(addProfileHotkey);
	enableFeature(makeDiscussionSidebarSticky);
	enableFeature(closeOutOfViewModals);
	enableFeature(improveShortcutHelp);

	if (!pageDetect.isGist()) {
		enableFeature(moveMarketplaceLinkToProfileDropdown);
		enableFeature(addYourRepoLinkToProfileDropdown);
	}

	if (pageDetect.isGist()) {
		enableFeature(addFileCopyButton);
	}

	if (pageDetect.isDashboard()) {
		enableFeature(hideOwnStars);
		enableFeature(autoLoadMoreNews);
	}

	onAjaxedPages(ajaxedPagesHandler);
}

function ajaxedPagesHandler() {
	enableFeature(hideEmptyMeta);
	enableFeature(removeUploadFilesButton);
	enableFeature(addTitleToEmojis);
	enableFeature(shortenLinks);
	enableFeature(linkifyCode);
	enableFeature(addDownloadFolderButton);
	enableFeature(linkifyBranchRefs);

	if (pageDetect.isIssueSearch() || pageDetect.isPRSearch()) {
		enableFeature(addYoursMenuItem);
		enableFeature(addCommentedMenuItem);
	}

	enableFeature(sortIssuesByUpdateTime); // Must be after addYoursMenuItem + addCommentedMenuItem

	if (pageDetect.isMilestone()) {
		enableFeature(addMilestoneNavigation); // Needs to be before sortMilestonesByClosestDueDate
	}

	if (pageDetect.isRepo()) {
		enableFeature(addReadmeButtons);
		enableFeature(addBranchButtons);
		enableFeature(addDiffViewWithoutWhitespaceOption);
		enableFeature(removeDiffSigns);
		enableFeature(addCILink);
		enableFeature(sortMilestonesByClosestDueDate); // Needs to be after addMilestoneNavigation
	}

	if (pageDetect.isRepoRoot()) {
		enableFeature(addToggleFilesButton);
	}

	if (pageDetect.isPR()) {
		enableFeature(scrollToTopOnCollapse);
		enableFeature(addDeleteForkLink);
		enableFeature(fixSquashAndMergeTitle);
		enableFeature(openCIDetailsInNewTab);
		enableFeature(waitForBuild);
		enableFeature(toggleAllThingsWithAlt);
	}

	if (pageDetect.isPR() || pageDetect.isIssue()) {
		enableFeature(linkifyIssuesInTitles);
		enableFeature(addUploadBtn);
		enableFeature(embedGistInline);
		enableFeature(extendIssueStatusLabel);
		enableFeature(highlightClosingPrsInOpenIssues);

		observeEl('.new-discussion-timeline', () => {
			enableFeature(addOPLabels);
			enableFeature(addTimeMachineLinksToComments);
		});
	}

	if (pageDetect.isNewIssue()) {
		enableFeature(addUploadBtn);
	}

	if (pageDetect.isIssue() || pageDetect.isPRConversation()) {
		enableFeature(addJumpToBottomLink);
	}

	if (pageDetect.isPRList() || pageDetect.isIssueList()) {
		enableFeature(addFilterCommentsByYou);
	}

	if (pageDetect.isPRList() || pageDetect.isIssueList() || pageDetect.isPR() || pageDetect.isIssue()) {
		enableFeature(showRecentlyPushedBranches);
	}

	if (pageDetect.isCommit()) {
		enableFeature(addPatchDiffLinks);
		enableFeature(toggleAllThingsWithAlt);
	}

	if (pageDetect.isCompare()) {
		enableFeature(toggleAllThingsWithAlt);
	}

	if (pageDetect.isPR() || pageDetect.isIssue() || pageDetect.isCommit() || pageDetect.isDiscussion()) {
		enableFeature(addReactionParticipants);
		enableFeature(showRealNames);
	}

	if (pageDetect.isCommitList()) {
		enableFeature(markMergeCommitsInList);
	}

	if (pageDetect.isPRFiles() || pageDetect.isPRCommit()) {
		enableFeature(addCopyFilePathToPRs);
		enableFeature(preserveWhitespaceOptionInNav);
	}

	if (pageDetect.isPRFiles()) {
		enableFeature(addQuickReviewButtons);
		enableFeature(extendDiffExpander);
	}

	if (pageDetect.isSingleFile()) {
		enableFeature(addFileCopyButton);
	}

	if (pageDetect.isUserProfile()) {
		enableFeature(addGistsLink);
	}
}

init();

Resolution: needs to use JS https://github.com/sindresorhus/refined-github/pull/1146#issuecomment-383282295

For people's reference → to disable this feature, add this to the extension's custom css:
.comment-body blockquote,
.comment-body pre {
  max-height: none;
}

For people's reference → to disable this feature, add this to the extension's custom css:

Thanks, it would be nice to have an option flag for this. :+1:

@dorian-marchal I think you can now disable all features in the settings. Just look into the list for the feature and uncheck the checkbox.

@mohe2015 	this one is only via the CSS option

I found this issue trying to disable this again (settings keep getting reset somehow).
https://github.com/sindresorhus/refined-github/pull/1146#issuecomment-475872046
.comment-body blockquote,
.comment-body pre {
  max-height: none !important;
}

Sorry, but it is not clear to me. Hw can I make long code blocks scrollable? Much like StackOverflow?

I don't know what you're referring to exactly, but this repo is for a Chrome extension: https://github.com/refined-github/refined-github
Unless you want to make the code scrollable *here on GitHub, you might want to ask the question on StackOverflow.


DBBACKUP_STORAGE_OPTIONS possible credential leak through debugging

Hello
Django protects in the debug mode some variables which are containing KEY, SECRET, PASSWORD and so on.
Because of the reason DBBACKUP_STORAGE_OPTIONS  is using lower case, django does not protect these variables.

I suggest to change DBBACKUP_STORAGE_OPTIONS to upper case for protecting these sensitive credentials.
Thanks and greets
Manuel

Hello @ManuelGysin,
Sorry for my latency, Yep great suggestion. I'll implement it soon.
It will be in next version.

This is fixed with https://github.com/django-dbbackup/django-dbbackup/blob/2dee62a5bcd8560cbdf6945bd29d762012c9d8ee/dbbackup/storage.py#L65