rules_ios mono-repo for improved developer experience

A proposal to merge XCHammer core and Tulsi core into rules_ios to improve developer experience, consequently onboard more folks, and set precedent for coming changes.
Background
Originally rules_ios pulled rules from bazelbuild org like rules_apple and Tulsi. These rules are maintained with regard for and optimized to accept up-stream commits from a small handful of companies: e.g. copybara PRs. Using said rules was gainful in the early days of Bazel where the community was much smaller and 1-2 companies took on the majority of maintenance burden and owned architecture decisions. In the early days it was an amazing way to use Bazel with iOS development but came at a cost.
In the bazel-ios org, we're working to make an amazing iOS build system by scaling out contributors and have enlisted multiple industries as maintainers. We have all long outgrown the model where 1 or 2 companies dictated how iOS development tools worked and improvements were made by forking, binary patching, or code injecting. Because Bazel has aten the world/top app list, we now have effectively achieved this goal: more teams are improving rules_ios and iOS developers have successfully migrated to Bazel by using rules_ios.
Mainlining key dependencies to improve them
This change pulls in dependencies to iterate freely on them in tree. The main reasons reasons are architecture limitations, mono-repo developer experience, and diversifying the contributor base amongst multiple maintainers/industries.
Remove architecture restrictions and complexity
We have simplified iOS Bazel builds by reducing the surface area and supported features. By having a smaller amount of test cases we can make sure things like the debugger is compatible with the rules.
We're often focused on making the rules work better with how an exiting Xcode project works and adopting common idioms. Often  features in Xcode and core compilers need heavy modification or redesigning to work at scale.
This vertical integration is why you can build with rules_ios, generate an an Xcode project, and LLDB works end to end in. We even test complicated features like this end to end. The users of rules_ios don't have the same problems as other users because of it. Importantly, we're able to create optimizations in both build system and IDE by making assumptions that you're using rules_ios. For instance, index while building V2 and virtualized frameworks. These features have a non trival end to end consideration in the rules, Bazel core, Xcode project, and remote execution configuration.
Aligning with industry standard conventions makes it easier to onboard a dev to a codebase, even when it isn't ideal to implement these features inside of Bazel or bazelbuild or repositories.  The key takeaway is that making these features work helped us achieve scale and mass adoption. Said changes are often riddled with tradeoffs and complexity. Therefore, they are often zero-sum with an organization that has already re-designed code to work with Bazel or has it working well.
We are effectively locked into the architecture that bazelbuild org uses which is optimized for both a small handful of companies and upstreaming changes from copybara. In practice, the implementation of these rules has made it very difficult for broad adoption of Bazel and required hacks and forks evolve the state of the art.
Maintainer velocity and alignment
We often change dependencies to work better with the Xcode way. This is at a zero sum where bazelbuild maintainers architecting the software in a way that they can integrate with copybara. This limitation prevents us from evolving the state of the art: often, we have to design features and make tradeoffs in design that aren't aligned with the use cases of the bazelbuild org.
Xcode updates: for varying reasons, a maintainer may need to adopt Xcode sooner than another. We'll have a better shot to be up-to-date by having multiple contributors working to update Xcode in the open in a tested mono-repo. Finally, because we have a subset of supported features and test cases to do things, it's easier to maintain overall and we can more easily land commits upstream which are gated by an open source test suite and several maintainers with aligned use cases.
Mono-repo developer experience
Today it is painful to bump tags, commits, and it is significantly easier to reason about how a change will work in a mono-repo. Even bzlmod is prolific, testing rules_ios end to end is still easier in a mono-repo. Versioning has been a problem in the last several years, and we can side-step it by testing our rules in a single repository.
Conclusion
This proposal improves the developer experience, makes it easier to test, manage users, and consequently enable more iOS developers adopt Bazel.

Thanks for writing this up Jerry! I think a lot of this makes sense, and having a monorepo certainly helps with upstreaming changes to the underlying rules / tools. I've been trying to contribute more to rules_ios and did find it difficult to understand were using a fork of XCHammer for example. If these all lived in this repository it would make it more obvious what tools we're using and how we can continue to update them.
Some questions:

How would we ensure any improvements to the main XCHammer repo also make it here?

Would we archive that one in favor of this or something else?

What does it mean for engineers using those other repositories, should they be informed that it wont be maintained (at least not in that repo anymore)?

Overall I'm in favor of this change and agree with a lot of what you laid out.
Some other things to maybe consider:

How will we ensure we keep up with issues / PRs etc? Should we set up a working group that can figure out how best to serve the community on this?
Are we at all worried about deviating from the rules_apple rulesets and what that might mean in terms of splitting up the community / splitting up our efforts?

You've got it! Yeah I'm hoping that we can make rules_ios easier to add features to it as a result

How would we ensure any improvements to the main XCHammer repo also make it here?

I believe if we get the core into rules_ios I imagine folks would use our optimized version of it.

What does it mean for engineers using those other repositories, should they be informed that it wont be maintained (at least not in that repo anymore)?

The maintainers of those repositories can still support / maintain them or pull our code instead. The issue is that we want to simplify, unify, and redesign the code in a way that won't require us to hard fork. Tulsi was a long running fork because there are many changes upstream-able and architecting it breaks copybara support. rules_apple is in a similar boat.

Are we at all worried about deviating from the rules_apple rulesets and what that might mean in terms of splitting up the community / splitting up our efforts?

The original proposal is only for Xcode generator unification but I imagine the idea will improve the situation for rules_apple and eventually re-write it in tree / cut the dependency. We had to deeply integrate it into the repository and it is very difficult to update as a result. e.g. look at all the internal imports.
This splitting of the community exists because some folks tweaked their repos to work with Bazel as-is and other folks made different bazel rules to be compatible with how their builds worked. The historical issue is the two approaches are both difficult and add unwanted complexity, abstractions, when people tried to unify. IMHO splitting is beneficial now that both ways have critical mass. The overall Bazel iOS community has multiple and approaches to building an iOS app with open source build technology!

How will we ensure we keep up with issues / PRs etc? Should we set up a working group that can figure out how best to serve the community on this?

Today we use github and iterate on this in the open e.g. this issue 😉  . It's TBD but tentatively folks are kicking around making the first ever rules_ios con later this year!

I'd also want us to think how rules_xcodeproj fits into this picture.
I'm not clear on what the benefits of importing XCHammer and Tulsi into rules_ios are. Maybe it'd be worth elaborating more on what you have in mind... like are you talking about just copying them in and losing the history? Using git submodules? I see we already have them forked under the bazel-ios org, so explaining why that isn't sufficient to iterate on them would be a good start.
Although the rules_ios community has grown, all of us are also part of the broader community of "people who build iOS apps with Bazel" and my main concern would be that this broader community is also small enough that splitting it can divide resources and make it harder for each project to move forward.

Thanks you all for the thoughtful comments

Can you clarify how rules_apple and rules_swift would fit into this setup?

This proposal is mainly focused around XCHammer and Tulsi but does set precedent for others. I suspect rules_apple can benefit from the proposal as well: an in-tree version of bundling functionality that drops Copybara compatibility, is fully tested in a mono-repo, and adopts bazel-ios's community based ownership model.
In this case, in-tree rules would be written and tested as an atomic unit instead of separate parts. Folks that pull rules_apple and rules_swift today would instead pull the functionality as a single SHA that was tested as a unit along with the rest of the required functionality. We'll have end to end test coverage and more maintainers on this code. As a Bazel user pulling a single SHA is an improvement from a release and testing perspective over what we have today: a special incantation tens of git commit SHA256's /github URLS to update Xcode and generate a project that maybe work together.
As a developer and maintainer of the rules, it's easier for me to have a common set of test cases at hand, see how it all fits together in the same repo, and explain how something works in the same tree to a colleague.

I'd also want us to think how rules_xcodeproj fits into this picture.

buildbuiddy github org has different Xcode project generator for Bazel. rules_ios originally had a similar thing: create an Xcodegen generator rule like XCHammer and without Tulsi's core. There was already a sequence of github PRs to fork XCHammer, update it to Bazel 5, fix M1 issues, add rules_ios support, and align to one generator. The additional apsect this proposal adds is a mono-repo and optimizing towards for community maintainers and diversified ownership.
As a Bazel user, it's easier for me to pull a single tested SHA than worry about what incantation of rules_apple, rules_swift, rules_ios, Xcode generator, and Bazel I need to unlock an Xcode update.

my main concern would be that this broader community is also small enough that splitting it

This proposal fixes splits that exist already in the community: specifically with rules_ios, XCHammer, Tulsi. Mainly, aligning rules_ios to use XCHammer and Tulsi. @thiagohmcruz might be able to weigh in. These changes may break copybara comapt by re-architect aspects of the generator to work better with the rest of the ecosystem
It's significant portion of top apps and contributors aligned here now in regards to dedicated developers on Bazel. As a Bazel ecosystem contributor, it's easier to contribute to a mono-repo that the top apps have clearly aligned on.

I'm not clear on what the benefits of importing XCHammer and Tulsi into rules_ios are.

TDLR is it's easier for owners to iterate on and test end to end as a unit, bump a single sha instead of tens, unify efforts and diversify ownership of XCHammer/Tulsi amongst multiple maintainers/companies/industries.
The main issue with making a complex rule like a generator is having to accept PRs in the open source that lack all test cases. By developing as a unit we can more easily achieve this and have better end to end testing.
Finally, like other rules, versioning all required functionality to build a Swift/ObjC app into one mono-repo fixes many issues that exist Bazel ecosystem and further aligns key maintainers. Finally, if you need to fork it, you can fork a single repo instead of forking 10 and easily test your fork end to end.

and adopts bazel-ios's community based ownership model.

To be clear rules_apple, rules_swift, apple_support, and tulsi are all community owned as well. To clarify any misconceptions I've documented the process to become a maintainer here: https://github.com/bazelbuild/rules_apple/blob/master/MAINTENANCE.md#how-to-become-a-maintainer

In this case, in-tree rules would be written and tested as an atomic unit instead of separate parts.

If you're interested in something like this I'm sure we could setup some CI integration on the other repos that automatically ran upstream rules_ios CI on your contributions to those repos. I definitely see the value in a single atomic change being fully tested downstream, but I think if you copy those rules into this repo and "drop copybara support" what I think you're saying is you're creating a hard fork for rules_ios?

The additional aspect this proposal adds is a mono-repo and optimizing for community maintainers and diversified ownership.

Maybe @brentleyjones can comment here but I know with the initial interest around rules_xcodeproj there are some folks contributing there as well, I imagine as that project matures it be community maintained similar to rules_apple and friends.

which is optimized for both a small handful of companies

The upstream rules are definitely not intended to be optimizing for a small handful of companies. I'm definitely supportive of moving things into those rules that are generally applicable! I've mentioned this in the past with headermap support, and maybe there are some other clear things that rules_ios does that also fall into that category

and upstreaming changes from copybara.

Are there examples of major breaking changes you would like us to make that have been blocked on this? We have been considering being less strict about taking upstream changes, but also so far I think we've tried to balance divergence reasonable, so if there's a case where you don't think we've made the right call I'd love that example.

what I think you're saying is you're creating a hard fork for rules_ios

Yeah I'm proposing that we copy in a subset of XCHammer and Tulsi into the tree and align on that instead of keeping a fork of Tulsi, a fork of XCHammer, and rules_ios's other Xcodegen rule. People can pull a tested and up-to-date version instead of rebuilding their own. rules_ios makes Bazel for iOS an easier sell 😉

To be clear rules_apple, rules_swift, apple_support, and tulsi are all community owned as well. To clarify any misconceptions I've documented the process to become a maintainer here: https://github.com/bazelbuild/rules_apple/blob/master/MAINTENANCE.md#how-to-become-a-maintainer

IMO we'd still need to fix the iteration speed issues that moving it in addresses: e.g. complexity issues, lack of API, dropping capybara compatibility, CI automation, and removing micro-repo layout.

I'm sure we could setup some CI integration on the other repos that automatically ran upstream rules_ios CI on your contributions to those repos

Bazel For iOS is expensive and harder to maintain because of the millions of untested feature permutations in Tulsi, rules_apple, or rules_swift. IMHO there isn't enough code amongst Tulsi, rules_ios, rules_apple, rules_swift, and xchammer have so many separate repos, CI processes, and contributor models. rules_ios doesn't need all the complex configuration that integrating the bazelbuild repos requires and the code will fundamentally simpler because we can remove a large portion of it and related abstractions required in rules_ios to use these repos.

The upstream rules are definitely not intended to be optimizing for a small handful of companies.

After "handing it off" historically the idea was Copybara PRs sent to a separate branch and 1-2 blessed maintainers cherry-picked them to the main branch. Because of this ( and I imagine isolating contributors in other ways ), velocity of that repo rules slowed because it was 1-2 companies maintaining it where the maintainers might have isolated contributors and it wasn't possible to refactor or rewrite in a way that breaks Copybara. People have had to do everything ad-hoc or in forks.
Maintainers in this bazel-ios org have been continuously unable to upstream changes to bazelbuild org even to add for support basic stuff like Xcode updates. I suspect the unmerged PRs outpaced the ones we landed and then implemented in ad-hoc ways elsewhere. The ad-hoc patching is expensive for the community who wants to be on the latest tech: e.g. to use Bazel with M1 l was originally required to do it an edge-case riddled hacky way in multiple forks to pull it off or change rules_ios in a way that doesn't work with Bazel core. Case in point, the bazelbuild org maintainers didn't need support for M1 out of the gate and they weren't structured for mass contribution and wouldn't accept our changes.

know with the initial interest

I think it's amazing that other people are adding more rules to the ecosystem - the existence of alternatives and startups in the space further legitimizes the Bazel community! AFAIK making another bazel to pbxproj generator pattern does split the community further. I'm biased towards a simpler way and one repo!

if there's a case where you don't think we've made the right call I'd love that example.

I'm not suggesting that anyone has made a problematic decision BTW - it's more around the current maintainer model, "internal" APIs we have to/will have call, and unmaintainable attractions we need to use rules_apple and rules_swift as they are today 🔨

While I agree with the desire to simplify and make the rules easier to develop on and maintain, I'm not sure having hard forks of dependencies is the way to go. Maybe you're seeing it differently than I am though... would it help to make a list of what we hope to accomplish that is more concrete? E.g. you mention:

we'd still need to fix the iteration speed issues that moving it in addresses: e.g. complexity issues, lack of API, dropping capybara compatibility, CI automation, and removing micro-repo layout.

Would it help to be more explicit about what PRs we've submitted that didn't get merged? Or what complexity we hope to eliminate? Have we tried proposing a better API for consumers of rules_apple/rules_swift? Or asking for some private providers to be made public?
I'm also a bit confused about:

Bazel For iOS is expensive and harder to maintain because of the millions of untested feature permutations in Tulsi, rules_apple, or rules_swift.

Do you mean we'd just delete this features from rules_swift? How do we decide which features to keep and which to remove?

Maintainers in this bazel-ios org have been continuously unable to upstream changes to bazelbuild org even to add for support basic stuff like Xcode updates.

It would help to have some examples of this. For instance the M1 stuff that you mention...
FWIW, I'm not trying to be difficult... in my experience hard forks end up being maintenance burdens so I'd want us to consider this carefully.

Thank you for the thoughtful comments!

Do you mean we'd just delete this features from rules_swift? How do we decide which features to keep and which to remove?

For sure and unifying the existing iOS rule ecosystem to a subset mono-repo is a direct goal! We can define what's in-scope in with principals. IMO if something is used by us, has an open test case on rules_ios CI, it's beneficial to have in this repo. Otherwise fine/ideal to consider removing it. I believe such principals are necessary to maintain velocity and easily accept PRs.
Importantly, if we keep our approach aligned with how people build iOS software, then it continues to benefit us and Bazel at scale. We can come employ an interface to integrate other ecosystems.

I'm not sure having hard forks of dependencies is the way to go.

I agree with you 100%. The intention behind this proposal is removing hard forks of Tulsi and XCHammer. If the updated/maintained version is moved in tree, I'd suggest to redirect/archive the originals and use our community mono-repo. We cannot force hands, however we can continue to gain momentum by making rules_ios amazing from a build performance, maintainability, and contribution perspective 🚀

FWIW, I'm not trying to be difficult... in my experience hard forks end up being maintenance burdens

bazelbuild org has the opposite contributor model of what I am proposing. Dare I call status quo closed development/ open source. Today the maintenance burden is on the community ( e.g. mainly one person I owe tremendous gratitude ) to keep parity with and cherry pick Copybara PRs. In other-words, we are already forked. Pulling the dependencies in-tree allows us to drop un-needed-by-rules-ios-complexity and make it easier to update and integrate with for the community.

Have we tried proposing a better API for consumers of rules_apple/rules_swift?  Or asking for some private providers to be made public?

I would advocate to simplify the code instead. Bazel has rules, actions, and build files as a general way to compile code. Going beyond this adds more opinions and complexity to bag.
The closest thing we've seen is the ecosystem using native providers in Bazel core. These providers maintain a lot of fields and require refactoring issues on all consumers: take apple_common.ObjcProvider as one attempt. Case in point, in Bazel 5 there was a copybara PR to remove module_map field from it but who is to say that change benefitted the community: e.g. say we want it because clang still has a -fmodule-map-file argument and had issues updating to Bazel 5 because the shared provider.
We can always bridge external rules into rules_ios. For swift we'd need to rely on SwiftInfo provider, or just make a BUILD file that compiles code with our own rule.

Would it help to be more explicit about what PRs we've submitted that didn't get merged?

This proposal improves the developer experience by bringing it into a mono-repo and not having to worry about breaking ad-hoc test case when you submit. I think it's somewhat ortoganl

For now, we have an issue to address removing forks of XCHammer and Tulsi: https://github.com/bazel-ios/rules_ios/issues/498
Most of the discourse here is on rules_apple and rules_swift so will leave this open incase anyone suggestions or proposals about rules_apple or rules_swift 🚀


[CELEBORN-1399] MR job CelebornMapOutputCollector should checkExcepti…

…on after flush

What changes were proposed in this pull request?
check exception and throw it if exist after CelebornMapOutputCollector flush
Why are the changes needed?
For small shuffle mr job, shuffle push maybe only happend one time when map task finished deal all task records and do flush before close  MapOutputCollector
In this case, CelebornMapOutputCollector should checkException after flush, and throw exceptions when flush has exception, if not, job status is wrong
Does this PR introduce any user-facing change?
no
How was this patch tested?
test use mr job

I think you need to check exception in close method as well.

I think you need to check exception in close method as well.

done

Merged into main(v0.5.0) and branch-0.4(v0.4.1).


dot (.) rendering issue

Environment
Windows build number: Microsoft Windows [Version 10.0.18363.1256]
Windows Terminal version: Windows Terminal Preview v1.6.10272.0

Any other software?: Not applicable

Steps to reproduce
Type dots repeatedly.
Spacing around the dots keeps on changing till the first 4 dots. Afterwards it is fine.

The effect you are seeing is the result of font ligatures - when multiple characters are combined to render as a single glyph. If you're using the Cascadia Code font, you can switch to Cascadia Mono for the equivalent style but without the ligatures.
For more information see https://docs.microsoft.com/en-us/windows/terminal/cascadia-code

The effect you are seeing is the result of font ligatures - when multiple characters are combined to render as a single glyph. If you're using the Cascadia Code font, you can switch to Cascadia Mono for the equivalent style but without the ligatures.
For more information see https://docs.microsoft.com/en-us/windows/terminal/cascadia-code

Yea, this is just how ligatures work. Let us know if you have any other questions!


Clusters being produced with no members assigned?

Dear team Terminus,
After running Salmon I invoked Terminus like so:
terminus group -d ${SALMONDIR} -o ${TERMINUSDIR} -m 0.05 --tolerance 0.01
terminus collapse --threads 16 -d ${SALMONDIR} -o ${TERMINUSDIR}

To my surprise, I'm getting clusters with plenty of reads assigned but no transcripts belonging to them.
For instance, from the Terminus quant.sf:
NewTr25922 | 1155 | 975.4305 | 240.3131 | 1563.851
This cluster does not show up in clusters.txt at all. What gives? Am I invoking Terminus incorrectly?
Version information:
salmon 1.4.0 (installed via conda)
Terminus 0.1.0 (installed via conda)

Turns out, these clusters probably represent all the reads that could not be assigned to anything else. Happened in my case because I had mapped against a subset of all the sequences I had.


Removed the style setting margin/padding/height on body and html.  It is...

... too intrusive.

If there was a good reason for that line of CSS, then there must be a better way than removing padding from the html and body tags.


Add equivalent of torch.Tensor.flatten

It would be convenient for jax.Array to have the equivalent of the torch.Tensor.flatten method. (For example, when flattening an input with arbitrary batch dimensions in a convolutional neural network.) Here is an example implementation:
from jax import numpy as jnp, Array

def flatten_axes(a: Array, start: int = 0, end: int = -1) -> Array:
    return a.reshape(a.shape[:start] + (-1,) + a.shape[end:][1:])

def main():
    a = jnp.arange(2 * 3 * 4 * 5 * 6).reshape(2, 3, 4, 5, 6)
    assert flatten_axes(a, 1, -2).shape == (2, 3 * 4 * 5, 6)
    assert flatten_axes(a, -3).shape == (2, 3, 4 * 5 * 6)

if __name__ == '__main__':
    main()

Seems reasonable to me! Thanks for the suggestion.
@jakevdp do you foresee any issue with this? If not, I can add it.

I think lax.collapse would work.

@soraros Thanks for the suggestion. lax.collapse is less convenient in that

It is not a method attached to jax.Array, like torch.Tensor.flatten. This could be resolved by attaching it as a method.
Its axis arguments are required rather than optional, and its end axis argument is exclusive rather than inclusive. For example, "flatten the last 3 axes" must be written as lax.collapse(a, -3, a.ndim) rather than simply a.collapse(-3). This could be resolved by making the start and end axis arguments optional, defaulting to 0 and a.ndim respectively.

Thoughts?
from jax import numpy as jnp, Array, lax

def flatten_axes(a: Array, start: int = 0, end: int = -1) -> Array:
    return a.reshape(a.shape[:start] + (-1,) + a.shape[end:][1:])

def main():
    a = jnp.arange(2 * 3 * 4 * 5 * 6).reshape(2, 3, 4, 5, 6)
    assert flatten_axes(a, 1, -2).shape == (2, 3 * 4 * 5, 6)
    assert flatten_axes(a, -3).shape == (2, 3, 4 * 5 * 6)
    assert lax.collapse(a, -3, a.ndim).shape == (2, 3, 4 * 5 * 6)

if __name__ == '__main__':
    main()

We could probably extend the existing jax.Array.flatten method with optional arguments. The only issue would be if numpy someday adds conflicting arguments to the method.

Well, we'd probably want to make start and end keyword-only

Personally, I think einops.rearrange is a better way to write these sort of operations.
I definitely don't think this is fundamental enough to be worth adding as a method. JAX and NumPy already have too many array methods.

@shoyer How would you write this flatten function in terms of einops.rearrange?

e.g.
einops.rearrange(images, 'b h w c -> (b h) w c')

This flattens b, h axes into (b, h) axis. However, not sure if rearrange will compile down to primitive JAX ops. @shoyer are you are if it does?

e.g.
einops.rearrange(images, 'b h w c -> (b h) w c')

This flattens b, h axes into (b, h) axis. However, not sure if rearrange will compile down to primitive JAX ops. @shoyer are you aware if it does?

Yes, it does, which you can verify with jax.make_jaxpr:
import einops
import jax.numpy as jnp
import jax

jax.make_jaxpr(
    lambda images: einops.rearrange(images, 'b h w c -> (b h) w c')
)(jnp.zeros((2, 3, 4, 5)))

Prints:
{ lambda ; a:f32[2,3,4,5]. let
    b:f32[6,4,5] = reshape[dimensions=None new_sizes=(6, 4, 5)] a
  in (b,) }

@jon-chuang What I meant was: Is there a way to express the operation "flatten axes from start to end" in einops, where start and end are arbitrary integers?
@shoyer

I definitely don't think this is fundamental enough to be worth adding as a method.

Flattening seems like a pretty fundamental array operation to me.

Coming back to this: I'm a bit wary of adding additional arguments to array.flatten that diverge from NumPy. I think given that we have an existing API that accomplishes this, we can close this request:
In [1]: import jax
   ...: x = jax.numpy.zeros((2, 3, 4, 5))
   ...: jax.lax.flatten(x, 1, -1)
Out[1]: (2, 12, 5)

Coming back to this: I'm a bit wary of adding additional arguments to array.flatten that diverge from NumPy. I think given that we have an existing API that accomplishes this, we can close this request:
In [1]: import jax
   ...: x = jax.numpy.zeros((2, 3, 4, 5))
   ...: jax.lax.flatten(x, 1, -1)
Out[1]: (2, 12, 5)

>>> x = jax.numpy.zeros((2, 3, 4, 5))
>>> jax.lax.flatten
AttributeError: module 'jax.lax' has no attribute 'flatten'
>>> jax.lax.collapse(x, 1, -1).shape
(2, 12, 5)

Thanks - I should have written collapse instead of flatten.


swift2.0 compatibility

Are there any plans to make this available for swift2.0?

Thanks for your feedback. I'll do in 1-2 days.

Hi,
Sorry for late reply.
There's new branch "XCode 7 beta" that has been tested with Swift 2.0 in XCode 7. You can try out.

Thanks!
I got the old code working with just a few tweaks - but I will definitely give the "official" version a try.  :)


Polymorphic deserialization of list items

I have the following structure:
@Parcel
public class Zoo {
  List<Animal> animals;
  List<Tiger> tigers;
}

@Parcel
public class Animal {

}

@Parcel
public class Tiger  extends Animal {

}

I put objects of different Animal subclasses at runtime. When i deserialize the parcel, i only get back Animal objects, and not Parrot, Tiger etc. I tried to implementations field of the Parcel annotations on Animal, but it fails with the following exception:
Caused by: org.parceler.transfuse.util.TransfuseRuntimeException: Unable to perform code generation
        at org.parceler.transfuse.transaction.CodeGenerationScopedTransactionWorker.innerRun(CodeGenerationScopedTransactionWorker.java:53)
        at org.parceler.transfuse.transaction.AbstractCompletionTransactionWorker.run(AbstractCompletionTransactionWorker.java:35)
        at org.parceler.transfuse.transaction.ScopedTransactionWorker.run(ScopedTransactionWorker.java:55)
        at org.parceler.transfuse.transaction.Transaction.run(Transaction.java:77)
        at org.parceler.guava.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:299)
        at org.parceler.transfuse.transaction.TransactionProcessorPool.execute(TransactionProcessorPool.java:60)
        at org.parceler.transfuse.transaction.TransactionProcessorComposite.execute(TransactionProcessorComposite.java:37)
        at org.parceler.transfuse.transaction.TransactionProcessorChain.execute(TransactionProcessorChain.java:38)
        at org.parceler.internal.ParcelProcessor.execute(ParcelProcessor.java:83)
        at org.parceler.ParcelAnnotationProcessor.process(ParcelAnnotationProcessor.java:84)
        at com.sun.tools.javac.processing.JavacProcessingEnvironment.callProcessor(JavacProcessingEnvironment.java:794)
        at com.sun.tools.javac.processing.JavacProcessingEnvironment.discoverAndRunProcs(JavacProcessingEnvironment.java:705)
        at com.sun.tools.javac.processing.JavacProcessingEnvironment.access$1800(JavacProcessingEnvironment.java:91)
        at com.sun.tools.javac.processing.JavacProcessingEnvironment$Round.run(JavacProcessingEnvironment.java:1035)
        at com.sun.tools.javac.processing.JavacProcessingEnvironment.doProcessing(JavacProcessingEnvironment.java:1176)
        at com.sun.tools.javac.main.JavaCompiler.processAnnotations(JavaCompiler.java:1170)
        at com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:856)
        at com.sun.tools.javac.main.Main.compile(Main.java:523)
        ... 88 more
Caused by: javax.annotation.processing.FilerException: Attempt to recreate a file for type `zoo.Tiger`$$Parcelable
        at com.sun.tools.javac.processing.JavacFiler.checkNameAndExistence(JavacFiler.java:522)
        at com.sun.tools.javac.processing.JavacFiler.createSourceOrClassFile(JavacFiler.java:396)
        at com.sun.tools.javac.processing.JavacFiler.createSourceFile(JavacFiler.java:378)
        at org.parceler.transfuse.gen.FilerSourceCodeWriter.openBinary(FilerSourceCodeWriter.java:48)
        at org.parceler.codemodel.CodeWriter.openSource(CodeWriter.java:100)
        at org.parceler.codemodel.JPackage.createJavaSourceFileWriter(JPackage.java:486)
        at org.parceler.codemodel.JPackage.build(JPackage.java:437)
        at org.parceler.codemodel.JCodeModel.build(JCodeModel.java:311)
        at org.parceler.transfuse.transaction.CodeGenerationScopedTransactionWorker.innerRun(CodeGenerationScopedTransactionWorker.java:49)
        ... 105 more

If i remove the @Parcel annotation from Tiger, i got this
Parceler: Unable to find read/write generator for type java.util.List<zoo.Tiger> for zoo.tigers

What version are you using @WonderCsabo?

I forgot to mention that: 1.1.6 .

Hmm, looks like I need to lock down the implementations field a bit more.
By default, Parceler doesn't perform a runtime class check on the contents of your list.  It will treat it as the declared type.
There is a way around this, however.  You could use a converter that uses the Parcels.wrap/unwrap to perform the runtime check.  It is a bit slower, but probably not noticeable:
@Parcel
public class Zoo {
  @ParcelPropertyConverter(ListParcelsWrapConverter.class)
  List<Animal> animals;
  @ParcelPropertyConverter(ListParcelsWrapConverter.class)
  List<Tiger> tigers;
}

public class ListParcelsWrapConverter extends ArrayListParcelConverter<Object> {

    @Override
    public void itemToParcel(Object input, Parcel parcel) {
        parcel.writeParcelable(Parcels.wrap(input), 0);
    }

    @Override
    public Object itemFromParcel(Parcel parcel) {
        return Parcels.unwrap(parcel.readParcelable(ListParcelsWrapConverter.class.getClassLoader()));
    }
}

We touch on this in the docs: http://parceler.org/#polymorphism and http://parceler.org/#custom_serialization
I am always open to ideas here if you have a better technique or thoughts.

Ehh, i read README file, but i did not know you have a fancy webpage. Thanks!
This is an acceptable workaround, thanks! I am not sure if it is feasible, but i think a new annotation, or a parameter to @Parcel would be uesful to mark the class as polymorphic. I think most users would want the concrete objects back.

Eh, i was too happy. This general class will not work.
I got this compilation error:
Zoo$$Parcelable.java
Error:(53, 107) error: incompatible types: List<Animal> cannot be converted to Collection<Object>
Error:(80, 119) error: incompatible types: ArrayList<Object> cannot be converted to List<Animal>
Zoo$$Parcelable.java
Error:(53, 109) error: incompatible types: List<Tiger> cannot be converted to Collection<Object>
Error:(80, 121) error: incompatible types: ArrayList<Object> cannot be converted to List<Tiger>

The obvious fix is that i change to ListParcelsWrapConverter extends ArrayListParcelConverter<Animal>. But i also have to create a different class for ListParcelsWrapConverter extends ArrayListParcelConverter<Tiger>. 😢

Ugh, sorry, I was too quick in giving an example... yeah, you're right you'd have to create a different converter for each type.  I need to fix the converter validation and casting to the given type.

Cool, I created the two classes. Actually I could create a generic base
class which provides the method implementations, and the subclasses only
differ in the class signature (generic argument). Maybe you can provide the
base class in the library if it is a common problem.

One idea brought up before was to add an annotation property to trigger Parceler to use the Parcels.wrap/unwrap for polymetric behavior instead of having to deal with a converter.  Something like this:
@Parcel
public class Zoo {
  @ParcelPropertyConverter(polymetric = true)
  List<Animal> animals;
}

I do want to keep away from using too much reflection (.getClass() in particular) as it is a slower opreation.
Oh, and if you're impressed by the website, you might be interested in whats backing it.  I've used Jekyll, Asciidoc, and Travis to generate the site directly from Github.  Makes editing the site easy because its just a PR or push to master.  In fact, you can edit the contents via github's web interface.
Here's the github repo:https://github.com/johncarl81/parceler-site
And the base "JAQ" project that I help maintain: https://github.com/asciidoctor/jekyll-asciidoc-quickstart

Yep, I also mentioned the annotation, but not this way. It would be great!
I think I'd you earn the user about reflection, it is ok. There is no way
to work around polymorphism I guess.
Your documentation setup is indeed impressive! Thanks for the info.


npm out of sync

Hello, it appears that the latest version on npm is 0.5.2, which has problems with React 16. I noticed it's been fixed here in the repo in version 0.6.2. Could someone update the package on npm for me?
Thanks in advance!
I also noticed that ImageHeaderScrollView.js is still importing PropTypes from prop-types even though it's not being used, and is not a dependency in package.json, so it's likely to cause some troubles :)

Oops, I published a patch for the 0.5.1 version which was used in one of my project. I guess npm used it as default.
It should be fixed now. I published a new version.


Add helper Makefile for more easier integration tests

Looks like you have failing functional tests.

I'm going to go ahead and merge this - we can figure out if we want to somehow merge this with the main Makefile tests later.  One thing to keep in mind - this changes the way the tests are run now quite a bit, making it easier to run a single test.  Please see the readme @ejholmes @russellballestrini


oauth support?

I would try this if I don't have to type my google username/password into your app :)

It should be perfectly safe - the desktop player is essentially just a specialized web browser. Signing in to it should be as safe as signing in in Chrome/Firefox.

Basically what @FISHunderscore said :+1:

It'd be nice if users didn't have to trust you, though.  That's why systems like oAuth exist.  Not everyone can understand the source well enough to be comfortable entering their credentials.
If you're not happy with that option, an alternative is what Radiant Player for Mac does -- you can log in to Google via Safari, and since Radiant uses Safari to render the page, the cookie can be safely reused and the user doesn't have to trust another application.  I'm not familiar with Electron, but if it's based on Chromium, could we do something similar?

Wait... So you're perfectly happy that an application can just steal your cookies from another application?!?!?  But logging in to a separate application securely is somehow bad?
In any case, that is not possible as chromium/chrome encrypts cookie storage.
Also, oAuth is not available for Google play music as there is no API for it.  It just isn't possible

Nothing is stolen, it's Safari using its own cookie. It's not transferred outside the Safari context.
"logging in to a separate application securely" is the part that requires trust. Unless it's a Google signed binary, or you know the code, you have to take a leap.
I understand your reaction, but please consider that there may be some bias since you wrote the application. It'd be nice to highlight security in the messaging if people are to trust their (valuable) credentials to the software.

I get what you're saying, and yeah I am probably a bit biased.  But let's be real here.

If you are worried about security as much to not trust a Open Source application that is code signed going through googles login process you probably have 2 factor authentication so you're fine
If you don't have 2FA then you could be password sniffed through thousands of ways more discreet then a google play music desktop application.

Although you are right this code is not signed by google, my signing cert has a trust rating that comes with it.  (Windows SmartScreen automatically assigns a "trust" value) and no one has EVER reported any issues before.
Like I said, if I could use oAuth I would.  But unfortunately I just can't :cry: Blame Google for that one.
Is there anything else you can think of (excluding oAuth and using chromes cookies) that would increase the trust some users will need to use this application?

Ah you're right, there's no oauth for google music. they do have API for google play. weird that that doesn't extend into music :\

This conversation provides great insight for the security conscious person deciding whether or not to use your app. Crafting a small blurb (in the spirit of your responses) and adding it to the readme would help increase your user base even further.

It would also be a convenience to the security minded developers to link to the code you are using for the login screen.

There is literally no code for the login screen. GPM sits inside of a webview which can be thought of as an iframe.


Initial Rook documentation (#886)

Initial implementation of:
https://github.com/osism/issues/issues/886

@berendt The two markdown check 404's come from unmerged changes to the main branch of ansible-collections-services and should be fine.

Testbed test running. Will merge this tomorrow after a successful run.


[dv/csr] Handle reset in the middle of CSR transactions

The DV code dealing with CSR accesses gets confused if a reset happens while there is an ongoing CSR access. In order to avoid this confusion tests are restricted to avoid resets while there are ongoing CSR accesses. This is undesirable on two accounts:

It prevents testing IP unit behavior when it gets reset in the middle of a CSR access, which is a pretty realistic scenario.
It complicates the generation of testing scenarios in order to prevent these cases.

Correction: the scenario is actually generated, but the problem is that the sequence cannot terminate in the middle of a transaction. Perhaps csr_rd/wr should immediately return if cst_utils_pkg::under_reset is on to avoid sequences getting wedged?
In any case it would be smart to document this behavior to avoid time wastage.

@weicaiyang
@sriyerg
might need your help to gauge whether this is something that's M2 gating or not.
It sounds like to me it's closer to V3 / M3 ..

I have discussed this offline with @matutem.
We don't restrict a sequence to issue a reset in the middle of a CSR access, but RAL access can't be killed before it finishes, otherwise, RAL is locked up.
We could either wait until all CSR accesses are done, before killing the sequence, or use the stress_all_with_rand_reset approach. @matutem LMK if you have any problem on this issue.
I think it's ok to close this or defer to V3. stress_all_with_rand_reset is a V3 test.

personally i think killing after completion is probably better...
but sounds good, let's table this to M3 for now.
If anyone objects, please add to the thread.

Triage: not considering this for M2.5 currently.

Rationale: With the current reset architecture, we have ndmreset and POR, which both reset blocks with CSRs as well as Ibex. There are also SW-controlled resets (e.g., for spi_device), but when Ibex does a reset through that, it cannot simultaneously do a CSR access during the reset.

Should this be reassigned?

Should this be reassigned?

Yeah, it's currently scheduled for M3, so I think we'll assign it when focusing on M3


Errors with 2017-2019 integrated feature tables?

This issue was initiated wrt the 2017-2019 integrated feature tables, which were deployed in late Nov 2020 / early Dec 2020, but appears to apply more generally to all tables. The tables are missing data, with column headers either not represented or empty.

A quick check of the underlying patient tables for counts of patients with CervicalCancerDx=1 reveals the following:
year 2010, N =16
year 2011, N = 19
year 2012, N = 24
year 2013, N = 37
year 2014, N = 64
year 2015, N = 85
year 2016, N = 64
(Note that I did not have the 2017-2019 tables available locally when I ran this test.)


Issue with Timezones when writing timestamp data type

When inserting timestamp data to AstraDB via Stargate REST API, the only timezone that works is Z.
No other timezones or timezone offsets work. Here are the ones I've tested:
Working:

2022-11-08T10:30:00Z
2022-11-08T10:30:00.000Z

Not Working

Bad request: Invalid Timestamp value: Text '2022-11-08T10:30:00.000' could not be parsed at index 23
Bad request: Invalid Timestamp value: Text '2022-11-08T10:30:00.000+11' could not be parsed at index 23
Bad request: Invalid Timestamp value: Text '2022-11-08T10:30:00.000+1100' could not be parsed at index 23
Bad request: Invalid Timestamp value: Text '2022-11-08T10:30:00.000+11:00' could not be parsed at index 23
Bad request: Invalid Timestamp value: Text '2022-11-08T10:30:00' could not be parsed at index 19
Bad request: Invalid Timestamp value: Text '2022-11-08T10:30:00+11' could not be parsed at index 19
Bad request: Invalid Timestamp value: Text '2022-11-08T10:30:00+1100' could not be parsed at index 19
Bad request: Invalid Timestamp value: Text '2022-11-08T10:30:00+11:00' could not be parsed at index 19

Test Environment
Table definition
CREATE TABLE demo.sendemail (
    timestamp timestamp PRIMARY KEY,
    emaildata text
)

API POST
curl --location --request POST 'https://1b44200e-93df-4b4e-b250-005c20557cde-australia-southeast1.apps.astra.datastax.com/api/rest/v2/keyspaces/demo/sendemail' \
--header 'X-Cassandra-Token: <token>' \
--header 'Content-Type: application/json' \
--data-raw '{
    "timestamp": "2022-11-08T10:30:00.000",
    "emaildata": "this is my mail message"
}'

Is this more a better documentation request or support all timestamps?

This is more a request to support all timezones.

Starting to investigate this now; here are some initial findings from modifying an existing Integration Test:
First, it looks like Stargate V1 does indeed fail on offsets, with or without colon.
Stargate V2 REST appears to accept offset with colon (+02:00, -03:00) but not without.
I will see if I can create unit tests for converters to dig deeper wrt Stargate V2 handling.

Ok, I found a way to improve REST implementation in Stargate V2 (main branch), added tests, PR is #2241. But I think I better try to cover all tests cases included -- the only I am not sure about is lacking timezone/offset indicator altogether as that seems like a dangerous usage.
But if that would result in same behavior as with Z maybe that is acceptable.


[ENHANCEMENT] - Option to merge graph lanes

Problem
When dealing with large Git repositories that contains a lot of branches, it is very commun to get visual bottle necks when a lot of branches share the same parent commit and start to accumulate at one particular place in the graph.
Proposal
An option to merge the graph lanes that share the same parent.
It have two benefices :

avoid lanes stacking
greatly reduce the horizontal graph size

Such option would make a new branch to only appear when its first commit is created instead of going on without any useful information to give to the user.
Like so, the graph complexity is diluted and informations appears only where they are relevant.
Mockup
The only example of this feature I can think of is in GitExtensions :

On the left :

The graph behave just like the one in SourceGit and the lanes are stacked at one place, where branches share a common parent.
This produce a lot of visually empty branches that goes up until their first commit happen. But like so, they simply pollute the space without much information to give.
Moreover, branches that starts earlier in the graph are trying to avoid this stack and therefore add more complexity to this area by agraving this behavior.
This is becoming hard to know at first glance which branch start on the previous commit or if it start at a lower level in the graph.

On the right :

The setting is activated and produce a gradual split of branches along the original one. They only appear when their first commit is created.
Notice how much the horizontal graph size is reduced, and so, without losing any information.

SourceGit for reference :

As always, thanks a lot for your hard work !!

Currently, if there's no other commits between the commit and it parent, we use short link curve, just like the line pointed by RED arrow. Otherwise we use a standalone curve for it, see the line pointed by GREEN arrow.
This design is also used by Fork.

If I understand correctly, there is currently nothing that could potentially be used to create a branch from elsewhere than a commit ?
Is there any chance that this design could be improved to include this behavior ?
If this proposal cannot be implemented, could it be possible to address this issue through another solution ?
I think this is really important to improve the graph readability ! Right now, I have to filter branches out to see clearly but it prevent me to understand the repository organization as a whole which is kinda lame..

Before:

After enable merge lines:

I don't think it's any clearer with merge branch lines enabled...


False Link for Git Repos

In your Documentation ist the link for clone the github false.
git clone https://github.com/example-user/my-drupal8-site

Well, it's an example, not false. Would it be more descriptive if it were https://github.com/example/example-site-repo.git ? How can we communicate that it's an example, standing in for your own site?
If you want to clone a starter Drupal site, you're better with the composer example in the quickstart.
Again, give us a hint how to make this more obvious that it's an example.


fix unexpected size_limit_exceeded

Codecov Report

Merging #1959 into canary will decrease coverage by <.01%.
The diff coverage is 0%.

@@            Coverage Diff            @@
##           canary   #1959      +/-   ##
=========================================
- Coverage    6.32%   6.31%   -0.01%
=========================================
  Files          90      90
  Lines        4082    4087       +5
=========================================
  Hits          258     258
- Misses       3824    3829       +5

Impacted Files
Coverage Δ

src/commands/deploy/latest.js
0% <0%> (ø)
:arrow_up:

src/util/report-error.js
0% <0%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0bfbd40...dcc420f. Read the comment docs.


fix(helm) change name length limit to 63 for helm create

since kubernetes 1.4, the upper limit for resource name is 63 characters,
so it is better to change the template when running helm create accordingly.

I guess this makes sense for master now that it targets k8s v1.4 and 1.3 support is in the v2.0 releases.
Unofficial LGTM

There was an issue for this, but I can't find it. If anyone stumbles on it, please link it back here.

https://github.com/kubernetes/helm/issues/1831?


unable to capture system audio.

There is no good way to record system sound

Electron doesn't support system audio recorder since its a chromium-based app.
and currently, there is no chrome API for recording system audio

audio means, not able to record the microphone or speaker output as well.

you mean that this App that you make  , did not record audio or streaming the audio(microphon) yes ?


⚠ Bofrost.de has degraded performance

In a0bde7a, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 705 ms

Resolved: Bofrost.de performance has improved in d4655e3 after 8 minutes.


ENH: added association measurements Pearsons Contingency Coefficient, Cramer's V, Tschuprow's T, Phi to scipy.contingency

Added measures of association for nominal data to contingency stats module. Measures added include Pearsons Contingency Coefficient, Tschuprow's T, Cramer's V and Phi. Measures are individual specified by values of the 'stat' argument (as 'C', 'T', 'V' or 'phi', respectively) within function scipy.stats.contingency.association_test.
These measures can provide useful information about variable independence to those performing chi-squared tests or categorical data analysis in general. Each of the measures is empirically estimated using the chi-squared statistic and are valid for 2d arrays.
@rgommers I thought I could delete the resolve_conflicts branch after checking out on the new one but I guess I was wrong. Either way here is the PR - squeaky clean.

@rgommers Thanks for referencing the previous PR and shepherding this project - I really appreciate it. As far as the failed tests I am not sure what is going on here, best I can tell I am getting a permissions error when the docs try to build - is this something I need to fix in my PR?

@dtemkin, your pull request is from your master branch.  It is recommended that you work from a separate branch when you submit a PR.  NumPy has a recommended git workflow, which you can also use when contributing to SciPy--just change numpy to scipy where appropriate.  The steps to create a new branch for the PR are described in the subsection "2 Develop your contribution", where, after updating your master branch, you create a new branch to work in with the command
git checkout -b new-branch-name

That creates and checks out a new branch called new-branch-name.  For your PR, you could call this branch, say,  assoc:
git checkout -b assoc

Having said that, you can continue working with your master branch, but it means you can't easily track concurrent changes taking place in the master branch of scipy.
Off the top of my head, I don't know the simplest set of commands that you would use to move your existing work to a separate branch and reset your master branch to the branch in the SciPy repo.    Does anyone else already have a recommended procedure for this?  New contributors often end up working in their master branch on their first PR, so it would be nice to have a standard set of instructions for fixing that.

New contributors often end up working in their master branch on their first PR, so it would be nice to have a standard set of instructions for fixing that.

No, best just to point it out for next time and not try to fix it for the work-in-progress PR. Otherwise any pushes are going to be more complex, like git push origin my-new-branch:master, which has a counterproductive effect.

Let's just review this PR and merge it in a reasonable amount of time.

Closes gh-12299

@dtemkin, thanks for the updates.  There are a few more changes to the code that I requested.  Can you take a look at those?
Also, we'll need to add association to the list __all__ at the top of contingency.py.  Doing that might trigger the same problem with the refguide check that I ran into in https://github.com/scipy/scipy/pull/11352.  To fix that, you can make the same change as I made in this commit: https://github.com/scipy/scipy/pull/11352/commits/462bf0bcb1b4843e9cf288128758c447b2d45dc0.

So I got the docs build to work but only after removing the argument description from the two arguments I took from chi2_contingency() not sure why a reference power_divergence was a problem in my function doc string.

The latest test failures are not related to the changes in this PR.

I polished off a few PEP-8 issues and merged.  Thanks @dtemkin!
I added a note about the new function in the Release notes for 1.7.0.

@WarrenWeckesser Awesome! Thanks so much for all your help.


'trace purp' command produces incorrect purpose

'trace purp' command produces syntactically incorrect purpose when the last (?) action is quiescence, by introducing STOP before HIT:
TXS >>  >-> Result !<PHONE_NUMBER>
TXS >>  >-> Action ! Plus(-3084,-2147475794)
TXS >>  >-> Result ! -2147478878
TXS >>  >-> Action ! Plus(-2147480901,6122)
TXS >>  >-> Result ! -2147474779
TXS >>  >-> Action ! Minus(0,-2147479461)
TXS >>  >-> Result !<PHONE_NUMBER>
TXS >>  >-> Action ! Minus(-2147482075,-2147479688)
TXS >>  >-> STOP
TXS >>  >-> HIT
TXS >>  ENDDEF
STOP should be replaced by QSTEP.

@tretmans could you give a model (txs-file) and the torxakis commands to reproduce this bug?


proposal: os/exec: inherit process startup info for new commands on windows.

My proposal is for new processes spawned with the exec library to inherit the parents startup info. This is a windows specific proposal.
Currently, if I were to try and spawn a process in a specific desktop it would not work due to the startup info property not being inherited from the parent.
Thanks!

What do you mean inherited? bInheritHandles flag in STARTUPINFO?

Yes, meaning if I start a Go process in C with special STARTUPINFO then when I call exec in the Go process it should get that special STARTUPINFO.

The place for this kind of thing is syscall.SysProcAttr.  Is the existing CreationFlags field sufficient for this?  Could you just set a flag there?

here?

How would I set this field?

For exmaple: https://github.com/mattn/goreman/blob/e9150e84f13c37dff0a79b8faed5b86522f3eb8e/proc_windows.go#L13

@Brett-Kavanaugh, given that you can control the flags in syscall.SysProcAttr, is that enough? If not, what specifically should we be looking at adding to syscall.SysProcAttr? Thanks.

Closing per lack of response. If you have more to add, please comment.
Per discussion with @golang/proposal-review


Cloudwatch log group should expire after a set amount of time time in CI

Enabler

So that CI logs are searchable, we want to expire Cloudwatch logs after a [group agreed] set amount of time. Logs created by CI are currently set to never expire. This adds up overtime, making searching difficult (AWS only loads up to 10,000 log groups by default). More logs will be created as the product grows, which will also incur additional costs to keep them forever.
Acceptance Criteria

[ ] Logs expire after X amount of time
[ ] Most recent logs (~ last 3 weeks) are easily searchable via Cloudwatch (AWS Console)

What about ElasticSearch? Do we need logs stored at all?

Logs going to ElasticSearch is managed here, I think:
https://github.com/linz/elasticsearch-config/tree/master/config/li/ilm


Test/disable feature procedure

:tada: This PR is included in version 2.0.1-beta.61 :tada:
The release is available on:

npm package (@beta dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Unable show Scrollbar On Google Chrome(and Safari) which does not display scroll bars by default

Hi, I used this react component a few days ago, It's very efficient!
But, today I find a compatibility problem. It does't work when Google Chrome is set to not show the scroll bar by default.
The cause of the problem is the getScrollbarWidth.ts file.  You will get number 0 by function getScrollbarWidth when Google Chrome is set to not show the scroll bar by default. Because the value of div.offsetWidth equal to the value of div.clientWidth.

Hi, @ellis-s, I have a similar issue.

Google Chrome is set to not show the scroll bar by default

Are you using Chrome feature flag to enable/disable scrollbar by default?
I had to set Show scroll bars to Always on macOS Monterey (v12.5). Then the scrollbar started to appear where expected on all browsers.


chore: packaging and development improvements

Types of changes

[ ] Bug fix (a non-breaking change which fixes an issue)
[ ] New feature (a non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Description
After #4051 we needed a build step again. This PR eliminates that step again.
Checklist:

[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly. (PR: #)
[ ] I have added tests to cover my changes (if not applicable, please state why)
[ ] All new and existing tests are passing.

Codecov Report

Merging #4193 into dev will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##              dev    #4193   +/-   ##
=======================================
  Coverage   88.94%   88.94%
=======================================
  Files          37       37
  Lines        1646     1646
  Branches      430      430
=======================================
  Hits         1464     1464
  Misses        155      155
  Partials       27       27

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0669b68...a8c9cb2. Read the comment docs.


Messages icon in wrong the place

The standard messages+ notification number has shifted one icon to the left, It is supposed to be over the right-most icon where you check the messages.
Sry to write so many on this, but I am hoping you might be able to address these at one time.

I was testing on a clean install, so I never noticed this.
I will have a look together with the other issues. Thanks for the heads up.

I just noticed that it is only the new IM indicator (green) which appears to the upper left of the shield that is off.  The right-side indicator for various alerts is in the right place.

This should be fixed with the latest release. Let me know if you still have this issue.

I tested issues #1, #2 and #3.   It is looking just awesome!  Thank you so much!

Great! Thanks.


Violating Content Security Policy directive

I've implemented the code in the project and after running, it ends up with this error:
Refused to load the script 'https://us1.clevertap-prod.com/a?t=96&type=page&d=N4IglgJiBcICoFEDKcC0BOAHJ1B2AWgKyoBKALAGz4gA0IADgOYwBMdAxkzCABYAufetAD0wgDYB7dgEMxPCQGc%2B0AMwAGDeImMwAO1ogA6vhIBpAPoBhAIIBZAAoxQAdwC29GGrp8x7GAG0AXQBfOmkAMycQMTAAI25nAFNY1AUIAGtUADcARgA6fJYQYOCgAA%3D&rn=4&i=1649835178&sn=0&useIP=true&r=1649835180740' because it violates the following Content Security Policy directive: "script-src 'nonce-9U7THwOdqNKAWeKl7glt3y==' 'self' https://maps.googleapis.com https://www.googletagmanager.com https://www.google.com/recaptcha/api.js https://cdn.jsdelivr.net/npm/clevertap-web-sdk/clevertap.min.js". Note that 'script-src-elem' was not explicitly set, so 'script-src' is used as a fallback.
I'm new to ReactJS app, need your help.
Thanks in advance.

Hello Divya. Is the issue resolved ?
In your index.html try adding the following line

Do revert if issue still persist.

Hello Kamble, Solution didn't work for me but I have applied another patch to overcome this issue.

Ok. Thank you. Should I close the issue then ?

Before closing this issue. We need a robust solution for this. Because we are dealing with the finance on our web so we have blocked all the inline scripts and this is the basic security requirement. In the clever tap library, all the scripts are inline which will create an issue in the future also.

Divya, In React App you can use sdk by importing it as npm pacakge. Or you can create external js file which will import min.js link.


coming again, about Converts

now, in my code like this:
rem(75)
.banner {
    height: 75px;
    .image {
        width: 1rem;
     }
}

My code may have some weird,  I selected all to convert to rem.
the "height" to 1rem and the "width"  to 75px;
I have to converts one by one,
I think it might be more convenient if there are two hotkeys,
one converted to rem and one converted to px.
If you convert to rem, you may need to filter out px
I don't know if my thoughts are unreasonable.

Maybe px needs to be converted to rem is a decimal, rem is converted to the integer required by px

You can actually add the keyboard shortcuts yourself

Go to File > Preferences > Keyboard Shortcuts
Type px in the searchbox
Assign a keyboard shortcut to the operation you want to perform px>rem or rem>px

My fault. thanks a lot!


Convert to west extension

https://docs.zephyrproject.org/latest/develop/west/extensions.html#adding-a-west-extension
This is a very convenient script, thank you for creating it! As I start to use it I felt like it would be easier for the user if it was a proper west extension. At the same time, west has most (all?) the context already of things like compiler path, source directory and build directory, so the extension should be able to automatically figure it out.

Sorry for the late response. I was not an expert of west when I worked on this script.
And thank you for the interesting suggestion. I will give it a try.

Did you get a chance to try? Would love to test it!


[karma-esm] fails to test

Description
Running tests produces a TypeError : 'map of undefined'. This occurs when setup-es-dev-server.js:30 searches for instances of script tags on the karma page
Steps to Reproduce
git clone git@github.com:bennypowers/stripe-elements.git
cd stripe-elements
git checkout create-source
npm ci
npm t

Expected
Tests!
Actual
~/D/stripe-elements $ npm t
><EMAIL_ADDRESS>test /Users/bennyp/Documents/stripe-elements
> karma start --coverage

START:
15 12 2019 18:40:55.406:WARN [filelist]: Pattern "/Users/bennyp/Documents/stripe-elements/__snapshots__/**/*.md" does not match any file.
15 12 2019 18:40:55.418:INFO [karma-server]: Karma v4.4.1 server started at http://<IP_ADDRESS>:9876/
15 12 2019 18:40:55.418:INFO [launcher]: Launching browsers ChromeHeadlessNoSandbox with concurrency unlimited
15 12 2019 18:40:55.421:INFO [launcher]: Starting browser ChromeHeadless
15 12 2019 18:40:55.950:INFO [HeadlessChrome 79.0.3945 (Mac OS X 10.14.6)]: Connected on socket IpLKwVR6PB7COQ4AAAAA with id 6825825

  TypeError: Cannot read property 'map' of null
      at fetchKarmaHTML (/Users/bennyp/Documents/stripe-elements/node_modules/@open-wc/karma-esm/src/setup-es-dev-server.js:62:6)
      at processTicksAndRejections (internal/process/task_queues.js:89:5)
      at async responseTransformMiddlewareConfig (/Users/bennyp/Documents/stripe-elements/node_modules/es-dev-server/dist/middleware/response-transform.js:65:22)
      at async transformIndexHTMLMiddleware (/Users/bennyp/Documents/stripe-elements/node_modules/es-dev-server/dist/middleware/transform-index-html.js:102:5)
      at async watchServedFilesMiddleware (/Users/bennyp/Documents/stripe-elements/node_modules/es-dev-server/dist/middleware/watch-served-files.js:40:5)
      at async responseBodyCacheMiddleware (/Users/bennyp/Documents/stripe-elements/node_modules/es-dev-server/dist/middleware/response-body-cache.js:118:5)
      at async etagCacheMiddleware (/Users/bennyp/Documents/stripe-elements/node_modules/es-dev-server/dist/middleware/etag-cache.js:14:5)
      at async /Users/bennyp/Documents/stripe-elements/node_modules/es-dev-server/dist/create-middlewares.js:117:5
      at async log404 (/Users/bennyp/Documents/stripe-elements/node_modules/@open-wc/karma-esm/src/esm-config.js:38:7)
      at async /Users/bennyp/Documents/stripe-elements/node_modules/koa-compress/index.js:38:5

Correcting that line (String#match returns null, instead of an empty array, if no matches are found) results in:
 ~/D/stripe-elements $ npm t
><EMAIL_ADDRESS>test /Users/bennyp/Documents/stripe-elements
> karma start --coverage

START:
15 12 2019 18:47:40.073:WARN [filelist]: Pattern "/Users/bennyp/Documents/stripe-elements/__snapshots__/**/*.md" does not match any file.
15 12 2019 18:47:40.085:INFO [karma-server]: Karma v4.4.1 server started at http://<IP_ADDRESS>:9876/
15 12 2019 18:47:40.086:INFO [launcher]: Launching browsers ChromeHeadlessNoSandbox with concurrency unlimited
15 12 2019 18:47:40.090:INFO [launcher]: Starting browser ChromeHeadless
15 12 2019 18:47:40.629:INFO [HeadlessChrome 79.0.3945 (Mac OS X 10.14.6)]: Connected on socket 7-JlLNLf2b8TnZXSAAAA with id 6158904
Dynamic import can only be supported when transforming ES modules to AMD, CommonJS or SystemJS. Only the parser plugin will be enabled.
HeadlessChrome 79.0.3945 (Mac OS X 10.14.6) ERROR
  Uncaught SyntaxError: Cannot use import statement outside a module
  at src/stripe-elements.test.js:1:1

  SyntaxError: Cannot use import statement outside a module
HeadlessChrome 79.0.3945 (Mac OS X 10.14.6) ERROR
  Uncaught SyntaxError: Cannot use import statement outside a module
  at src/stripe-elements.test.js:1:1

  SyntaxError: Cannot use import statement outside a module

Finished in 0.434 secs / 0 secs @ 18:47:41 GMT+0200 (Israel Standard Time)

SUMMARY:
✔ 0 tests completed

=============================== Coverage summary ===============================
Statements   : Unknown% ( 0/0 )
Branches     : Unknown% ( 0/0 )
Functions    : Unknown% ( 0/0 )
Lines        : Unknown% ( 0/0 )
================================================================================
npm ERR! Test failed.  See above for more details.

The karma page for this test shows the error SyntaxError: import declarations may only appear at top level of a module in console.

The problem is that you're missing type="module" in your test pattern. We definitely need to add better error logging for this.


Update test vectors

Test vectors for #267 and #268.
Breaking changes from the PRs

From #267

hash_to_scalar:  removed count. There is no longer a loop and the input to expand_message is just msg_octets instead of msg_prime = msg_octets || I2OSP(count, 8)
creat_generators: removed n (and the check at step 8) and went from I2OSP(n, 4) to I2OSP(i, 8) in the input to expand_message when calculating v.

From #268

ciphersuite_id: added HM2S_ at the end (the MAP_TO_SCALAR_ID of the map_to_scalar operation).

Note that the test vectors use variable length messages now (with one of the messages being empty). Addresses one of the comments in #223.

Discussed on the WG call of the 10th of July. Fixtures are cross validated. Mergin


doc: update architecture.md file

Replacing the reference and link to BDR with reference and link to
EDB Postgres Distributed (pgd) because of the name-change at EDB.
Closes #261
Signed-off-by: Jan Karremans<EMAIL_ADDRESS>

/ok-to-merge


Raw HTML header tags get rendered in a paragraph tag...

If I put a raw header tag in the markdown, it gets rendered inside a paragraph.  For example,

Markdown

gets rendered as

Markdown

I would expect it to be rendered like this:

Markdown

I've tried other html block tags (not exhaustively), and they don't seem to have this problem.

This is still happening with "markdown-it": "^8.4.1",

@puzrin why you closed this issue?
@cristiano-belloni did found a workaround?

I also figured out that it converts '' to '\n' which seems random 😨


Update vendor/cache

vendor/cache somehow got out of sync, probably from the recent upgrade to Ruby 2.4. For this patch, I simply ran bundle install and committed the results.
/cc @github/hooks

Looks like CI is still running under Ruby 2.1.7

CI shouldn't even be running Janky for this public repository, I don't think. No one externally can even see it.

We actually have a travis.yml file too, huh: https://github.com/github/github-services/blob/master/.travis.yml


Save final model

If the val_check_interval is not a factor of the total number of iterations, then the last few iterations will be wasted.  Also, if the val_check_interval is bigger than the total number of iterations, then no model is saved at all.  We should make sure the final model always gets saved.

Also related to #291.


Update dependency list to remove peer warning

Latest version of @material-ui/icons is 2.x.x and a peer dependency warning pops up looking for 1.x.x
npm WARN<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>but none is installed. You must install peer dependencies yourself.

+1 for this

we are now at allow<EMAIL_ADDRESS>as a peer dep


Arrange in alp. order and Duplicate fix

Checklist

[x] I made something good today 💐.
[x] I put links in the correct format: [Tool](link) — description.
[x] My description starts with the lower-case letter and ends with the full stop.
[x] I put the tool in the alphabetical order.

Just did a light cleanup and maintenance, this whole project came out good so far 😄🎉
Optional

[x] I added the appropriate labels: free, open-source or mac.
[x] I put the tool only to one category.
[x] I followed Contribution Guidelines.

@oliver-gomes thanks for the improvements!


Color chooser dialog support?

I managed to rebuild your standalone morphing dialog to my liking, however I found out the creator of MaterialDialogs also has a color picker dialog which I would like to use morphingly as well. However, upon inspection of the code he appears to use a fragment and the standard android Dialog system for this. We cannot simply "getView()" and put it inside the morphing activity. Any ideas of how to manage to do this?

I reckon there's about 3 ways to do this.

Abandon activity/fragment plumbing altogether and just mold the animation to work with two Views, this would require recreating the color chooser dialog as a standalone view and decoupling it from it's current fragment/Dialog. Then it's mostly a matter of doing the circular reveal + arc animation from a FABs position to the color dialog's final dimensions.
Recreate the color picker without fragment (most likely as a compound view) such that getView() does work (i.e. using a similar format as to what the library's MorphDialogActivity does for plain MaterialDialog)
I haven't looked into this one, but if shared element transitions for fragments work in a manner similar to activities, then simply hooking up the shared element transition animation to the color chooser dialog fragment could be the easiest way to do this.


cleaned up html structure and clarified experience section

Organized an HTML file section (list of technologies and development methodologies)

Let me know if I wasn't successful at resolving my merge conflict.  This has been excellent practice! Thank you for the opportunity.

Hi, looks like there's still conflict that might however be because I merged some other pull requests before this one :)

OK - trying again.  Thanks!

Nice! Great work 👍


New 'pull_production' rake task fails for me

The new 'pull_production' rake task fails for me.  Trying to run it does this:
pg_dump: saving database definition
pg_dump: dumping contents of table projects
pg_dump: [archiver (db)] query failed: SSL SYSCALL error: Connection timed out
pg_dump: [archiver (db)] query was: COPY public.projects .... TO stdout;
pg_restore: connecting to database for restore
Password:
pg_restore: [archiver (db)] connection to database "development" failed: fe_sendauth: no password supplied
Password:
psql: FATAL:  password authentication failed for user "dwheeler"
FATAL:  password authentication failed for user "dwheeler"

I had tried this earlier and never got this approach to work for me.  There aren't many users of this rake task (!), so I think the quick solution is to just restore in addition the rake task that actually works for me:
desc 'Copy database from production into development (requires access privs)'
task :pull_database do
  puts 'Getting database'
  # This fails, probably because we use Unix sockets with local db:
  # sh 'rake db:drop &&
  #     heroku pg:pull DATABASE_URL development --app production-bestpractices'
  sh 'heroku pg:backups capture --app production-bestpractices && \
      curl -o db/latest.dump `heroku pg:backups public-url \
        --app production-bestpractices` && \
      rake db:reset && \
      pg_restore --verbose --clean --no-acl --no-owner -U `whoami` \
        -d development db/latest.dump'
  puts 'You may need to run "rake db:migrate".'
end

Before doing that, please spend a minute trying to get pg:pull working. The issue, I believe, is that you aren't using a default password on your local postgres. See: https://github.com/heroku/heroku-pg-extras/issues/42#issuecomment-47258173
Alternatively, try running the the following:
PGUSER=some_user PGPASSWORD=some_password heroku pg:pull

Okay, makes sense.  I ran:
sudo -u postgres createdb `whoami`

and am trying again.  It take a long time to pull, which is odd because the database doesn't really have a lot of data.  (Granted, all the old versions take some real space.. but not THAT much space.)

Huh.  Now the "psql" command works just fine, but the download still fails:
.... crypto_certificate_verification_justification, crypto_verification_private_status, crypto_verification_private_justification, hardened_site_status, hardened_site_justification, installation_common_status, installation_common_justification, build_reproducible_status, build_reproducible_justification) TO stdout;
pg_restore: connecting to database for restore
Password:
pg_restore: [archiver (db)] connection to database "development" failed: fe_sendauth: no password supplied
Password:
psql: FATAL:  password authentication failed for user "dwheeler"
FATAL:  password authentication failed for user "dwheeler"
psql failed. exit status 512, output: ""
/usr/local/heroku/lib/heroku/command/pg.rb:793:in `exec_sql_on_uri'
/usr/local/heroku/lib/heroku/helpers/pg_dump_restore.rb:114:in `exec_sql_on_uri'
/usr/local/heroku/lib/heroku/helpers/pg_dump_restore.rb:95:in `verify_extensions_match'
/usr/local/heroku/lib/heroku/helpers/pg_dump_restore.rb:28:in `verify'
/usr/local/heroku/lib/heroku/helpers/pg_dump_restore.rb:16:in `execute'
/usr/local/heroku/lib/heroku/command/pg.rb:411:in `block in pull'
/usr/local/heroku/lib/heroku/helpers/heroku_postgresql.rb:77:in `maybe_tunnel'
/usr/local/heroku/lib/heroku/command/pg.rb:406:in `pull'
/usr/local/heroku/lib/heroku/command.rb:212:in `run'
/usr/local/heroku/lib/heroku/cli.rb:33:in `start'
/usr/local/heroku/bin/heroku:25:in `<main>'
 !    Heroku client internal error.
 !    Search for help at: https://help.heroku.com
 !    Or report a bug at: https://github.com/heroku/heroku/issues/new

    Error:       psql failed. exit status 512, output: "" (RuntimeError)
    Command:     heroku pg:pull DATABASE_URL development --app production-bestpractices
    Version:     heroku-toolbelt/3.42.45 (x86_64-linux) ruby/2.3.0

http://stackoverflow.com/questions/7667193/heroku-dbpull-failing-due-to-lack-of-password-when-i-havent-specified-a-passwo
--
Dan Kohn<EMAIL_ADDRESS>Senior Advisor, Core Infrastructure Initiative
tel:+1-415-233-1000
On Wed, May 4, 2016 at 6:43 PM, David A. Wheeler<EMAIL_ADDRESS>wrote:

Huh. Now the "psql" command works just fine, but the download still fails:
.... crypto_certificate_verification_justification, crypto_verification_private_status, crypto_verification_private_justification, hardened_site_status, hardened_site_justification, installation_common_status, installation_common_justification, build_reproducible_status, build_reproducible_justification) TO stdout;
pg_restore: connecting to database for restore
Password:
pg_restore: [archiver (db)] connection to database "development" failed: fe_sendauth: no password supplied
Password:
psql: FATAL:  password authentication failed for user "dwheeler"
FATAL:  password authentication failed for user "dwheeler"
psql failed. exit status 512, output: ""
/usr/local/heroku/lib/heroku/command/pg.rb:793:in exec_sql_on_uri' /usr/local/heroku/lib/heroku/helpers/pg_dump_restore.rb:114:in exec_sql_on_uri'
/usr/local/heroku/lib/heroku/helpers/pg_dump_restore.rb:95:in verify_extensions_match' /usr/local/heroku/lib/heroku/helpers/pg_dump_restore.rb:28:in verify'
/usr/local/heroku/lib/heroku/helpers/pg_dump_restore.rb:16:in execute' /usr/local/heroku/lib/heroku/command/pg.rb:411:in block in pull'
/usr/local/heroku/lib/heroku/helpers/heroku_postgresql.rb:77:in maybe_tunnel' /usr/local/heroku/lib/heroku/command/pg.rb:406:in pull'
/usr/local/heroku/lib/heroku/command.rb:212:in run' /usr/local/heroku/lib/heroku/cli.rb:33:in start'
/usr/local/heroku/bin/heroku:25:in `'
!    Heroku client internal error.
!    Search for help at: https://help.heroku.com
!    Or report a bug at: https://github.com/heroku/heroku/issues/new
Error:       psql failed. exit status 512, output: "" (RuntimeError)
Command:     heroku pg:pull DATABASE_URL development --app production-bestpractices
Version:     heroku-toolbelt/3.42.45 (x86_64-linux) ruby/2.3.0

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
https://github.com/linuxfoundation/cii-best-practices-badge/issues/320#issuecomment-217025867

Did you try these options?

A reinstall of my local environment fixed it long ago. There was probably some weird local corruption. This seems to have been specific to my situation a long time back, so I'm closing this.


Persistent command line history not working on FABulous2.0 branch

When testing the edits made in the recent pull requests, I noticed that the persistent CLI history (being able to recall commands from previous FABulous sessions) introduced in #132 did not work anymore in the FABulous2.0 branch. It still does work for me in the master branch. Can anyone confirm or deny this behaviour? If it can be confirmed, was it removed on purpose or by accident?
Thanks in advance for any help!

I have experienced the same, and it is likely by accident.

When I am working on the release, I will spend some time using LLM to generate tests for the software.


TradeService.cancelOrder(String orderId) can't always be implemented

Hello,
I found out that some exchanges don't provide the API required to implement TradeService.cancelOrder(String orderId). For example, at least Livecoin and Binance require to provide the currency pair, in addition to the order ID. This method being one of the most used, I am not sure what to do here: changing it to TradeService.cancelOrder(LimitOrder order) provides overall compatibility but looks too restrictive since most of the exchanges only need the order ID, whereas leaving it as it is and put some specific code in the involved exchanges implementation trade services is not good for the global use of the library.
Any thoughts?

This has been added, but it might not be supported by the exchange you are using.  The method in question is
public boolean cancelOrder(CancelOrderParams orderParams)
And the parameter can support id and currency pair... if it's been implemented (see TheRockCancelOrderParams for an example).  Implementing it is usually trivial.

Ok thank you I will look into this.


Skip remote Correios tests, they are unreliable.

Because travis tests 12 different versions and one of these tests fails I'd estimate around 10% of the time basic statistics tells us that these tests will break CI 70% of the time.
So we kill them :fire:
@kmcphillips

Great solution to working around an unreliable API.


readme: use PNG for correct webfont rendering

fixes #3
btw this thing is AWESOME. printed it out. :)

hook it up?!!!

Sorry for the delay. Why not?


twitter api calls are all doubled

they all call verify_credentials just to get the screen_name, should cache that

fixed this a while bacon forgot to close!


"No newline" character moves to new line 😂

With Refined GitHub, the "no newline" character is placed on a new line (hehe).

It should probably look more like without the extension enabled:

Reference: https://github.com/microsoft/TypeScript-Website/pull/1/files

That's me 😅.
This is because of: https://github.com/sindresorhus/refined-github/blob/6d852ce1d75bf9ce59bedb902300d9d5a2a8b4e6/source/features/indented-code-wrapping.css#L12
I know that I could've used inline-block but for the love of science I can't exactly remember why I went with block.

Hello there - is this still a problem? I've had a look at the link provided in the description (https://github.com/microsoft/TypeScript-Website/pull/1/files) and it seems fine to me in both Firefox and Chrome with the extension?

@arnaudvalle thanks for checking! indented-code-wrapping has been disabled, that's why the problem doesn't appear.


Support ONVIF cameras that require authentication

Is your feature request related to a problem? Please describe.
Our sample ONVIF broker currently does not support connecting to cameras that require authentication . This support should be added along with documentation on how to pass credentials.
Describe the solution you'd like
A solution should be proposed and made for passing unique credentials such as user/pass to brokers. One solution could be to use a ConfigMap. The ONVIF sample broker should be updated to optionally use authentication with the provided credentials.
Describe alternatives you've considered
Credentials could be passed via kubernetes secrets mounted into broker Pods, but then all brokers would get the same credentials. All discovered IP cameras would need to accept the same user/pass which is unlikely and unsage.
Additional context
Credential passing is currently supported for OPC UA -- aided by the use of x.509v3 certificates which can have a common CA over standard user/pass; however, that is not a complete solution either.
For OPC UA, kubernetes Secrets containing an x.509 certificate for an OPC UA client and the associated certificate authority are set in the broker PodSpec of a Configuration. However, this assumes that all discovered OPC UA servers have the same Certificate Authority as the certificate passed to the broker or have manually accepted the broker's client. It also means that all brokers have the same certificate, which is unideal. A configMap could also be leveraged by OPC UA to give brokers unique certificates. Or the OPC UA brokers could call out to a CA to get a signed certificate.

A little update here. I did some investigation into what ONVIF device management endpoints require authentication. The only endpoint that seems universally public is GetSystemDateAndTime which is why it is currently used for aliveness checks. On the other hand, cameras that require authentication are currently not discovered because Akri's ONVIF discovery handler calls the GetNetworkInterfaces endpoint on the camera in order to receive IP and MAC address information for filtering. Since this endpoint is not publicly accessible, a Received a response status of 401 Unauthorized, expected 200 error is returned and the camera is removed from the discovered list. Previously, we also called GetScopes which is also not publicly accessible; however, we were able to get scope information from the original discovery response to perform scope filtering. The question is whether we can find a similar work around for getting the device MAC address. This unique identifier is needed for creating a unique Akri Instance ID for the camera.
Options for supporting IP cameras with authentication

Akri could support discovering cameras with auth right now by removing the option to filter by MAC address and making the identifier based off of camera name and ip; however, there would be a risk of name collisions.
Akri could resolve the mac address of the cameras via a tool like arp; however, the libpnet crate appears to only support arp for IPv4. There may be other crates out there to support getting the mac address of a device we can use.
Design and implement a way to pass credentials to the Discovery Handler.

Another idea, use mac for ipv4, and ip for ipv6. If there are ipv6 cameras that use temporary ipv6 ips for privacy, or other non stable ipv6 addresses, we could revisit. But chances of ipv6 address conflict are low.
Re arp library, wondering if onvif discovery can only work on the same subnet or further, which would probably break arp discovery.

I had a thought about this where it might be interesting to use a pattern similar to cert-manager. This would allow a discovery handler to create a "creds request" then have the "auth-manager" component be able to reach out to a 3rd party auth location to request creds.
These creds could be encrypted and stored in something like Vault instead of a secret.
We are running into this general issue more frequently than just Akri (Metal^3 for example) so there's likely room for a 3rd party project to implement/standardize this pattern.

I had a thought about this where it might be interesting to use a pattern similar to cert-manager. This would allow a discovery handler to create a "creds request" then have the "auth-manager" component be able to reach out to a 3rd party auth location to request creds.
These creds could be encrypted and stored in something like Vault instead of a secret.
We are running into this general issue more frequently than just Akri (Metal^3 for example) so there's likely room for a 3rd party project to implement/standardize this pattern.

@adithyaj has been also putting thought into this space. The model you describe @agracey sounds ideal where we could have various secret manager backends that the auth manager could reach out to. I also was curious as to whether we could use Dapr for this but haven't been able to set aside time to try it out.

A little update here. I did some investigation into what ONVIF device management endpoints require authentication. The only endpoint that seems universally public is GetSystemDateAndTime which is why it is currently used for aliveness checks. On the other hand, cameras that require authentication are currently not discovered because Akri's ONVIF discovery handler calls the GetNetworkInterfaces endpoint on the camera in order to receive IP and MAC address information for filtering. Since this endpoint is not publicly accessible, a Received a response status of 401 Unauthorized, expected 200 error is returned and the camera is removed from the discovered list. Previously, we also called GetScopes which is also not publicly accessible; however, we were able to get scope information from the original discovery response to perform scope filtering. The question is whether we can find a similar work around for getting the device MAC address. This unique identifier is needed for creating a unique Akri Instance ID for the camera.
Options for supporting IP cameras with authentication

Akri could support discovering cameras with auth right now by removing the option to filter by MAC address and making the identifier based off of camera name and ip; however, there would be a risk of name collisions.
Akri could resolve the mac address of the cameras via a tool like arp; however, the libpnet crate appears to only support arp for IPv4. There may be other crates out there to support getting the mac address of a device we can use.
Design and implement a way to pass credentials to the Discovery Handler.

@kate-goldenring Dear Kate:  The idea is to keep discover() simple and just do discover() at a time. Therefore, shall we decouple the filtering logic to another request?
And a few thoughts built on possible options here

Removing the option to filter by MAC address in the current flow is a way to simplify the discover().
To avoid the risk of name collisions by "Scope and IP", providing a unique identifier in "Scopes" might be an option.
e.g. onvif://www.onvif.org/name/GreatONVIFCamera  -> onvif://www.onvif.org/name/GreatONVIFCamera-deviceID
or   onvif://www.onvif.org/name/GreatONVIFCamera-deviceID + onvif://www.onvif.org/location/locationSpecifier
To get the user credentials in the discovery handler if we really need to do so, just like @agracey mentioned, create a "get creds request" in the discovery handler and delegating the job to the "auth-manager" component might be a way to support platform-agnostic solution.

We discussed this a little in our community meeting today. I think we need to get a unique identifier from the camera (without authentication) in order to have an identifier to pass to a registry to get credentials. Maybe we circle back to looking into how to use a tool like arp (or equivalent library) to get mac addresses

@kate-goldenring Dear Kate, many thanks for the reply, not a problem, I know it's the holiday season around the end of the year. I hope you take some time to relax.
Let's circle back to this.

It would be a fairly big rework to remove filtering logic from the gRPC Discover request since the request passes the filters. There doesnt seem to be a reason for the agent to do the filtering. It would duplicate a lot of work and potentially bloat the Agent.

---> oh, indeed, removing the filtering logic is a burdensome rework.
What if we keep the same discover request interface
but do not support custom helm key: onvif.configuration.discoveryDetails.macAddresses.
As we know, to get a device macAddress via the ONVIF protocol needs the user credentials, which is a pinpoint so far.
Could we leverage http://onvif_host/onvif/device_service as an alternative to distinguish devices?
According to ONVIF core spec sec 5.1.1,

To my understanding, the entry point for an ONVIF-compliant device is fixed to http://onvif_host/onvif/device_service, onvif_host can be either an ipv4 or ipv6 address.
And this entry point can be retrieved from the WS-discovery response(d:XAddrs) as well(here's a response from my physical vivotek cc8160 camera)
How about the ONVIF device service endpoint as the unique identifier?
helm key:
onvif.configuration.discoveryDetails.onvifUrls,action: filter action to take on a set of ONVIF URLs.
onvif.configuration.discoveryDetails.onvifUrls.items: ONVIF URLs that the filter action acts upon

I think we could explore a way to identify cameras without MAC address. I am not sure how common it is for IP cameras to use scopes.

--> Scopes can be retrieved from the WS-Discovery response. However, it might not be a common user scenario
to use scopes for a user. ONVIF device entry point might be an easy one.

I agree that finding a solution to get credentials to the Discovery Handler is probably the best solution and one we have been exploring. This PR takes a look at using a device registry with Akri Implement external device inventory integration #520

--> That's an interesting one, thanks for the information here. Will definitely have a look at it!

@kate-goldenring : This is a really important feature as in most productive systems authentication has to be enable and without this feature Akri ONVIF cannot be used.

We discussed this a little in our community meeting today. I think we need to get a unique identifier from the camera (without authentication) in order to have an identifier to pass to a registry to get credentials. Maybe we circle back to looking into how to use a tool like arp (or equivalent library) to get mac addresses

arp and mac-address might not work for use cases where the cameras are in a different subnet than Akri which is often the case with cluster network and camera network separation.
Agree with @markwu010 , the service endpoint would be a good option.

@markwu010 and @martin-weber using the service endpoint sounds fine to me. Are either of you interested in taking on this work.
The change would be to stop using the result of get_device_ip_and_mac_address when filtering as an identifier and instead use the service endpoints returned by simple_onvif_discover.

According to ONVIF core spec sec 7.3.1,  The combination of an wsadis:Address
and wsadis:ReferenceProperties provide a stable and globally-unique identifier.
We can use "wsadis:Address + wsadis:ReferenceProperties.Address" as the identifier of an Onvif device.
7.3.1 Endpoint reference
A device or an endpoint that takes the client role should use a URN:UUID [RFC4122] as the address property
of its endpoint reference.
The device or an endpoint that takes the client role shall use a stable, globally unique identifier that is constant
across network interfaces as part of its endpoint reference property. The combination of an wsadis:Address
and wsadis:ReferenceProperties provide a stable and globally-unique identifier.

For IP and Mac address filters, we can keep them but the DH can be flexible to skip querying ip/mac address from Onvif devices if neither of them are used (set to Exclude None).  The consequence of not using the ip/mac address filters is Onvif DH won't expose the environment variables: ONVIF_DEVICE_IP_ADDRESS and ONVIF_DEVICE_MAC_ADDRESS_LABEL_ID to brokers.

Just to make sure, these two still allow for the workload (broker) to find the device on the network, right?

Currently the sample broker only uses ONVIF_DEVICE_SERVICE_URL to reach to the device.  ONVIF_DEVICE_IP_ADDRESS and ONVIF_DEVICE_MAC_ADDRESS are not used.  so there is no impact to the sample broker.  but if a workload does rely on the IP and MAC address and, in the future, we change the Onvif DH to only expose ONVIF_DEVICE_IP_ADDRESS and ONVIF_DEVICE_MAC_ADDRESS when possible, it might be an issue for those brokers that rely on IP/MAC address but that can be solved by providing credentials to perform authenticated discovery (PR in review https://github.com/project-akri/akri/pull/619)


GitHub Pages 404s

GitHub Pages have the following 404s:
https://azure.github.io/azure-sdk-for-net/
://azure.github.io/azure-sdk-for-net/
	https://azure.github.io/sdk/core/Azure.Core/README.md
	  \_____ error code: 404 (not found)

https://azure.github.io/azure-sdk-for-net/CONTRIBUTING.html
	https://github.com/Azure/azure-sdk/blob/master/docs/engineering-system/repo-structure.md
	  \_____ error code: 404 (not found)
	https://azure.github.io/azure-sdk-for-net/doc/dev/Versioning.md
	  \_____ error code: 404 (not found)

https://azure.github.io/azure-sdk-for-net/index.html
	https://azure.github.io/sdk/core/Azure.Core/README.md
	  \_____ error code: 404 (not found)

@chidozieononiwu

@chidozieononiwu have a look at the current broken links using the pipeline https://dev.azure.com/azure-sdk/internal/_build/results?buildId=225016&view=logs&j=275f1d19-1bd8-5591-b06b-07d489ea915a&t=e2acf9c4-1534-54a3-1ef0-dea62490483f.
We need to update our doc indexing to replace relative likes in the root readme, similar to what we are doing in other languages, and then make sure we are using relative links instead of links to master in these readmes.

Doc index already uses the replace relative link tool. I fixed the only remaining case with this PR https://github.com/Azure/azure-sdk-for-net/pull/9331

All 404 are now fixed.


Update SILLoop::canDuplicate to correctly handle begin_apply

Leave a warning in isTriviallyDuplicatable so that future updates there will know to also update this code.

LGTM

@rjmccall @aschwaighofer My understanding is that it's valid to have:
loop:
  begin
cond_br exit, tail

tail:
  end
  br loop

exit:
  end

Right.  And in that situation, the loop would contain just loop and tail, right?

@rjmccall Yes.


Update django.po

this -> 此

Thanks for the report, however, translations are handled at Transifex, not through GitHub.