Proposal to introduce notion of environments

It should be possible to define multiple remotes for the same environments. For example, Kafka cluster will consist of number of remote urls:

broker(s)
schema registry
connector instance(s)
KSql instance(s)

The above set of remotes (that logically forms a Kafka cluster) will be repeated for different environments (dev, prod).
Also we should be able to specify some common remote systems (for example Secret manager) per environment.
'servers' can de defined under the environment as well.
Do you think a concept of 'environment' should be introduced on the root level?
environment:
  localDev:
    servers:
        # servers here
    remotes:
      broker:
          # url, protocol, etc
      schemaRegistry:
          # url, protocol, etc
      connect:
          # url, protocol, etc
      secretManager:
          # url, protocol, etc

  staging:
    servers:
        # servers here
    remotes:
      broker:
          # url, protocol, etc
      schemaRegistry:
          # url, protocol, etc
      connect:
          # url, protocol, etc
      secretManager:
          # url, protocol, etc

Issue 65 proposes to solve the same problem by grouping servers.
I would say it may be used complimentary to the environment proposal.
Example:
Kafka cluster usually defines a list of brokers, and any of them may be used by a client application to connect to a cluster.
For this problem I would use grouping to specify that number of servers (within an environment) actually represent brokers.
I would still use environment(s) to distinguish what group of brokers belongs to which environment.
environment:
  staging:
    remotes:
      broker-1:
          group: brokers
          # url, protocol, etc
      broker-2:
          group: brokers
          # url, protocol, etc
      schemaRegistry:
          # url, protocol, etc
      connect:
          # url, protocol, etc
      secretManager:
          # url, protocol, etc

  production:
    remotes:
      broker-1:
          group: brokers
          # url, protocol, etc
      broker-2:
          group: brokers
          # url, protocol, etc
      broker-3:
          group: brokers
          # url, protocol, etc
      schemaRegistry:
          # url, protocol, etc
      connect:
          # url, protocol, etc
      secretManager:
          # url, protocol, etc

One more use of environments is to specify variables to substitute in other definitions. Runtime value for the 'environment' will need to be passed to generator and other tools.
In this example 'x-replicas' value will be different depending on the environment.
environments:
  local:
    variables:
      example-channel-replicas: 1
  dev:
    variables:
      example-channel-replicas: 3

channels:
  example-channel:
    x-messageCompatibility: 'FULL'
    x-message:
      $ref: '#/components/messages/OrganisationFact'
    bindings:
      kafka:
        x-partitions: 9
        x-replicas: '#/environments/{environment}/variables/example-channel-replicas'
        x-configs:
          cleanup.policy: compact
          delete.retention.ms: 0
          confluent.value.schema.validation: 'true'

@ekozynin Hi! Have in mind that https://github.com/asyncapi/spec/pull/531 will most likely be included in the next version of AsyncAPI, so how would you like to combine your idea with the idea of linking channels to servers?
Another thing: Server Reusability. If a server does not change between environments at all, would the user then have to duplicate the server definition for each environment? This seems to me to be something that would create a lot of boilerplate for the user.
The case with variables: I am not in favor of this solution because you are trying to tie variables (perhaps for generation etc) to a specification. Currently you already have this possibility in the current version (two options):

parse the document and paste in the appropriate tokens (like {environment}) the data from the input
use an extension that can be read in the tools.

How about marking the environment at the server level?
servers:
  brokerProd:
    environment: prod
    # url, protocol, etc
  schemaRegistryDev:
    environment: dev
    # url, protocol, etc

Or if you need also grouping then we can introduces labels or annotations (proposal copied from @GeraldLoeffler from first draft of proposal from https://github.com/asyncapi/spec/pull/531 PR) to define dynamically annotations:
brokerProd:
  labels:
    environment: prod
    group: brokers
  # url, protocol, etc
schemaRegistryDev:
  labels:
    environment: dev
    group: schemaRegistries
  # url, protocol, etc

Sorry my example wasn't detailed enough, server definitions are different between environments. Consider the below example.
Linking servers to channels will not help in this scenario, as in case of Kafka, channels (topics) are partitioned and located on different servers by Kafka internals. We need to define multiple brokers in asyncapi because this is how client applications connect to Kafka cluster. In case one broker is down, they will connect to any other broker in the list.
Introducing labels (as in @magicmatatjahu example above) for servers will solve this particular problem, as we will be able clearly mark server's type and environment.
environment:
  staging:
    remotes:
      broker-1:
          group: brokers
          endpoint: staging1.cloud
      broker-2:
          group: brokers
          endpoint: staging2.cloud
      schemaRegistry:
          endpoint: schema-registry-staging.cloud

  production:
    remotes:
      broker-1:
          group: brokers
          endpoint: production1.cloud
      broker-2:
          group: brokers
          endpoint: production2.cloud
      broker-3:
          group: brokers
          endpoint: production3.cloud
      schemaRegistry:
          endpoint: schema-registry-prod.cloud

Since the spec has a notion of servers/remotes, means we are already including our environments/infrastructure into asyncapi file.
Example 1:
Our servers/remotes configuration usually remains the same across multiple environments, except of some small changes, such as remote endpoint. In this scenario, and in case of few environments, we have to copy paste the same definition, just changing one or two attributes. Amount of boilerplate code increases.
Using variables we can define different values for different environments, while keeping a single server/remote definition.
Example 2:
In case of kafka, when we define kafka bindings for the channel as part of asyncapi spec, we may want to create only a single partition for local, while many more partitions for production use. I don't think it is do-able with the current version of asyncapi, nor I think there is anything in place for upcoming version 3.
Using variables we can define different values for different environments, and using them in the bindings section.
For the latter example, I can use custom extension to define variables, parse and substitute them. The question is, should it become part of the asyncapi specification.
We already can define different servers/remotes in the specification (aka different environments) in the spec, so I think we should think how to make it more re-usable and flexible for different use-cases.

I definitely see a lot of value in being able to group related server types together - and, for schema registries in particular, having a proper place to put these details would be fantastic.
In my opinion, though, I don't think Kafka Connect belongs in here. Connect is really just an application, so I think should be treated the same as any other client application.

@ekozynin I recently updated https://github.com/asyncapi/spec/issues/654, becoming mostly a similar suggestion on what you did.
WDYT about @fmvilas suggestion here: https://github.com/asyncapi/spec/issues/654#issuecomment-1074424027
Do you think it makes sense?
I'm happy to move forward and join efforts with you, so we can include this into v2.4.0 release next month.

@fmvilas @derberg could you please remove the stale label? https://github.com/asyncapi/spec/issues/654 might fix this.

Done. Weird, supposedly, just leaving a comment here should remove the stale label 🤔 Or maybe I dreamed it 😄

@fmvilas not dream, it is true 😄 . Just remember that stale workflow runs just once a day. So we have remove-stale-when-updated but you need to wait until next schedule 🤷🏼
Example: https://github.com/asyncapi/spec/issues/738#issuecomment-1185510041
So better to unstale by comment as it works and kinda makes you think about the update info


Add divmod function to cupy namespace

This PR fix this error.

I tried this patch but the error still occurs. The error does not occur after writing cupy.divmod = cupy.core.divmod, so we have to put the symbol to cupy instead of fixing the argument of create_ufunc.

Looks good.
Jenkins, test this please.

It looks NumPy <1.13 does not have divmod.

@beam2d I fixed test.

Thanks, LGTM

@beam2d This pull-request is marked as to-be-backported, but the corresponding backport PR could not be found. Could you check?

It seems the PR added cupy.divmod.  Is this still an enhancement and OK to be backported?

OK, I changed label.

I want backport this PR to support latest fusion function.


[SOLVED] GDAX API unable to call getBalance, http_status 'invalid signature'

Sim and paper trading working fine but as soon as I try live trading I get this error. Been trying to dig into this myself without luck, would appreciate some help or pointers on rectifying this issue. GDAX status site shows everything is fine on their end.
Running CentOS7-1611 Core, all zenbot dependencies installed and up to date that I'm aware of. Everything else working fine, no other errors or warnings. Triple checked that all GDAX API info in conf.js was entered correctly.
I can grab version info on nodejs npm etc. if it will help.
This is the console error output I receive after running zenbot in live trading mode:
GDAX API is down! unable to call getBalance, retrying in 10s { Error: non-200 status: 400     at statusErr (/root/zenbot/extensions/exchanges/gdax/exchange.js:30:17)     at /root/zenbot/extensions/exchanges/gdax/exchange.js:93:25     at Request._callback (/root/zenbot/node_modules/gdax/lib/clients/public.js:45:7)     at Request.self.callback (/root/zenbot/node_modules/gdax/node_modules/request/request.js:187:22)     at emitTwo (events.js:106:13)     at Request.emit (events.js:191:7)     at Request.<anonymous> (/root/zenbot/node_modules/gdax/node_modules/request/request.js:1044:10)     at emitOne (events.js:96:13)     at Request.emit (events.js:188:7)     at IncomingMessage.<anonymous> (/root/zenbot/node_modules/gdax/node_modules/request/request.js:965:12)     at emitNone (events.js:91:20)     at IncomingMessage.emit (events.js:185:7)     at endReadableNT (_stream_readable.js:974:12)     at _combinedTickCallback (internal/process/next_tick.js:80:11)     at process._TickCallback (internal/process/next_tick.js:104:9) code: 'HTTP_STATUS', body: { message: 'invalid signature' } }

GDAX API is down! unable to call getBalance, retrying in 10s
{ Error: non-200 status: 400
at statusErr (/root/zenbot/extensions/exchanges/gdax/exchange.js:30:17)
at /root/zenbot/extensions/exchanges/gdax/exchange.js:93:25
at Request._callback (/root/zenbot/node_modules/gdax/lib/clients/public.js:45:7)
at Request.self.callback (/root/zenbot/node_modules/gdax/node_modules/request/request.js:187:22)
at emitTwo (events.js:106:13)
at Request.emit (events.js:191:7)
at Request. (/root/zenbot/node_modules/gdax/node_modules/request/request.js:1044:10)
at emitOne (events.js:96:13)
at Request.emit (events.js:188:7)
at IncomingMessage. (/root/zenbot/node_modules/gdax/node_modules/request/request.js:965:12)
at emitNone (events.js:91:20)
at IncomingMessage.emit (events.js:185:7)
at endReadableNT (_stream_readable.js:974:12)
at _combinedTickCallback (internal/process/next_tick.js:80:11)
at process._TickCallback (internal/process/next_tick.js:104:9) code: 'HTTP_STATUS', body: { message: 'invalid signature' } }

Problem solved!
I had only enabled "view" and "trade" checkboxes when requesting my original API key on GDAX, when it appears that "manage" is also required. Generated a new key with those three checkboxes marked and now live trading works.
So this is not an issue with zenbot at all.


Updated the README.md.

Changed the flake8 version string to be pip compatible.

@safiyat This can break nox session to validate readme. Can you update that as well?

@safiyat This can break nox session to validate readme. Can you update that as well?

Done. Sorry I missed those changes to be made, earlier.

@safiyat please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.
@microsoft-github-policy-service agree [company="{your company}"]

Options:

(default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.

@microsoft-github-policy-service agree

(when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

@microsoft-github-policy-service agree company="Microsoft"

Contributor License Agreement

@microsoft-github-policy-service agree company="VoerEir AB"

@karthiknadig could you please check now?


chore: 2024-05 update deps

Problem
Just maintenance work keeping deps up to date without including many new transitive deps along the way
Summary of changes
updates:

tonic
prost
opentelemetry
memoffset
hashbrown
rpds
serde_with
serde_assert

Checklist before requesting a review

[ ] I have performed a self-review of my code.
[ ] If it is a core feature, I have added thorough tests.
[ ] Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
[ ] If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

[ ] Do not forget to reformat commit message to not include the above checklist

Or would you rather the split PRs be across multiple releases?

that would be the best tbh. This PR probably causes tens of thousands of lines of diff.


Add bytearray.{lower, upper}

Also I fixed the bug with checking if bytearray is lower. The function bytearray.islower() returns alway true.

Codecov Report

Merging #502 into master will decrease coverage by 0.21%.
The diff coverage is 70.83%.

@@            Coverage Diff             @@
##           master     #502      +/-   ##
==========================================
- Coverage   57.37%   57.15%   -0.22%
==========================================
  Files          67       67
  Lines       12302    12357      +55
  Branches     2908     2930      +22
==========================================
+ Hits         7058     7063       +5
- Misses       3459     3507      +48
- Partials     1785     1787       +2

Impacted Files
Coverage Δ

vm/src/obj/objbytearray.rs
50.49% <70.83%> (-15.5%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ac95b61...150fe18. Read the comment docs.

It looks like this has some conflicts with your other PR I just merged, could you resolve these?

Yes, sure. I rebased this branch with master branch.


Merge common code bases for TdsParserStateObject.cs (3)

Basically the first two commits of #1844. Plus an optional refactoring. Let me know what you think.

I just rebased the branch adding a few commits, in order to follow @David-Engel's suggestion:

On the "breaking it up" suggestion. Troubleshooting the pipeline failures might be easier by breaking it up, too. I'd first try to make changes to the separate TdsParserStateObject.cs files to bring them closer to identical. Ideas for steps in individual PRs:

Rename internal variables/properties to all match.

Reorder methods/member variables to align the file layout as much as possible, keeping identical code together as much as possible. I might even add some regions and/or comments to help future diffs keep their alignment when common code is removed from each file. (No other changes so that reviewers don't have to scrutinize any differences between the large code blocks.)

Move common code into a new common file. Avoid any changes within moved blocks of code.

That's still a lot of changes to review with the care we need. I've gone through them all and made notes so I'll put my notes here to help others, function by function.
TryPeekByte - identical in both
TryReadByteArray - functionally identical, netfx adds reliability assert, uses different memory barrier, uses null check for span empty
TryReadByte - functionally identical, netfx adds reliability assert, uses different memory barrier
TryReadChar - functionally identical, netfx uses _bTmp where netcore uses stackalloc, netcore is better
TryReadInt16 - functionally identical, netfx uses _bTmp where netcore uses stackalloc, netcore is better
TryReadInt32 - mostly identical, netfx uses _bTmp where netcore uses stackalloc, netfx uses BinaryConverter where netcore does manual shifting
TryReadInt64 - functionally identical, netfx adds reliability assert
TryReadUInt16 - functionally identical, netfx uses _bTmp where netcore uses stackalloc, netcore is better, no reliability assert?
TryReadUInt32 - functionally identical, netfx uses _bTmp where netcore uses stackalloc, netfx adds reliability assert, netcore is better
TryReadSingle - functionally identical, netfx adds reliability assert
TryReadDouble - functionally identical, netfx adds reliability assert
TryReadString - functionally identical, netfx adds reliability assert
TryReadStringWithEncoding - functionally identical, netfx adds reliability assert
ReadPlpLength - identical in both
TryReadPlpLength - identical in both
ReadPlpBytesChunk - identical in both
TryReadPlpBytes - mostly the same, netcore uses Array.Empty and has support for cached plpbuffer stored in snapshot, netcore is better
It would be good to understand what the reliability assert is supposed to be doing and see if there a way we can make it compile out completely on netcore.

Good to see you reviewing this @Wraith2, since you wrote #667 and #285. If it helps:

TryReadByteArray: Interlocked.MemoryBarrier is a wrapper for Thread.MemoryBarrier according to its documentation (see Remarks section).
TryReadByteArray: The null-check that changes to Span.IsEmpty is inside a Debug.Assert condition, so it doesn't affect Release behavior. Moreover, a few lines below there is an if statement using Span.IsEmpty rather than the null-check in both netcore and netfx. I didn't go back in commit history to see how it ended up this way yet, will edit this comment if I do.
The reliability assert call is compiled out actually, because I annotated the empty stub with [Conditional("NETFRAMEWORK")]. Since the stub's code is only compiled for netcore, the condition will always be false. The reliability assert in netfx is basically a Debug.Assert, checking the that method is running in the correct context (e.g. some reliability functions have been called beforehand).

My personal preference would be to use Interlocked.MemoryBarrier everywhere if possible. It fits more closely with the logical intention than using the Thread version to me. Either version would work identically so it's not a huge issue.
For spans since they're structs comparing against null has never made sense to me and I'd prefer to have IsEmpty used everywhere if possible. It was probably me that got it wrong but since you're here you can correct it :)
As long as codegen confirms that the reliability call is entirely compiled out i'm fine with it as you've ported it.

My personal preference would be to use Interlocked.MemoryBarrier everywhere if possible. It fits more closely with the logical intention than using the Thread version to me. Either version would work identically so it's not a huge issue.

I agree, that's why I chose Interlocked.MemoryBarrier.

For spans since they're structs comparing against null has never made sense to me and I'd prefer to have IsEmpty used everywhere if possible. It was probably me that got it wrong but since you're here you can correct it :)

Comparing against null is a indeed confusing, because of the implicit conversion. You can use Span.Empty instead. However it isn't exactly the same as checking IsEmpty in case of an empty array. In which case, the null-check triggers Debug.Assert while IsEmpty doesn't. But in the end of the day I think it doesn't really matter, since it is just a debug assertion and the code below it handles both null and empty arrays the same way.

As long as codegen confirms that the reliability call is entirely compiled out i'm fine with it as you've ported it.

I think we could add Debug.Assert or throw an exception inside the stub, to verify it is omitted without having to look at the codegen. An exception would also make tests fail. What do you think?

What was this? A pull and rebase? A merge on top of your previous commits would have been clear. Please avoid anything that requires a force push on a PR under active review. It makes it really difficult to tell what was changed since the last time someone reviewed. It essentially invalidates all reviewed commits as we can't tell what's changed. This causes a delay in getting your PR reviewed and merged.

@David-Engel My bad, you are right. I will force-push the previously reviewed commits and add a merge commit on top.


Fix links, spelling, punctuation, wording, example code

Fix invalid links
Correct spelling
Fix punctuation
Improve wording
Fix incorrect example code

We really appreciate your contributions that come in every weekend, rain or shine or snow, like a clockwork :)

Moved some sections to the main Connections guide and merged manually. Thank you.


adding draft for fixing behaviour for group parameter

Hi all.
This might be more complex than pruning the path in the open_group_as_dict function. It is kind of complex because when we use _iter_zarr_groups,  it yields the root group. I am still working o it

[x] Closes #9665
[ ] Tests added
[ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
[ ] New functions/methods are listed in api.rst

@TomNicholas, I think this solution works, but the  DataTree.from_dict(groups_dict) function will create the root as an empty node. I think we might need to check it out. Let me know your thoughts.

I think we are getting close. However, we are still having some discrepancies when comparing both datatrees using the group parameter and when directly selecting it via path. Example:
print(dtree["/group2/subg1"])
Group: /group2/subg1 <----- different root paths here
│   Dimensions:  (x: 2, y: 3)
│   Inherited coordinates:
│     * x        (x) int64 16B -1 -2
│     * y        (y) int64 24B 0 1 2
│   Data variables:
│       blah     (x) int64 16B 2 3
├── Group: /group2/subg1/subsub1 <----- different paths here
│       Dimensions:  (y: 3)
│       Data variables:
│           var      (y) int64 24B 4 5 6
└── Group: /group2/subg1/subsub2 <----- different  paths here

dt2 = xr.open_datatree("test.zarr",
                          group="/group2/subg1"

print(dt2)
Group: /  <----- different root paths here
│   Dimensions:  (x: 2)
│   Dimensions without coordinates: x
│   Data variables:
│       blah     (x) int64 16B ...
├── Group: /subsub1 <----- different  paths here
│       Dimensions:  (y: 3)
│       Dimensions without coordinates: y
│       Data variables:
│           var      (y) int64 24B ...
└── Group: /subsub2 <----- different  paths here

Any comments on this @TomNicholas @keewis?

this is by design, I think? I'd interpret group="/group2/subgroup1" as saying "give me that group as the new root group". Then tree.encoding["source_group"] can contain the full path of the new root.
(if you know unix commands, this would be similar to chroot)

discrepancies when comparing both datatrees using the group parameter and when directly selecting it via path

The reprs are different because the objects are different: dtree["/group2/subg1"] has a parent, whereas dt2 does not. So this is intended.

The test should look very much like the ones in #9669 - create a tiny nested tree, save it to zarr/netcdf, open it with the group parameter, and check the structure is as expected.

Sorry @aladinor - in fact could we just do this? https://github.com/pydata/xarray/pull/9666#discussion_r1815590340

This looks good! @keewis 's comments are addressed so I'm going to merge it.

(you can still add yourself to the list of contributors to the DataTree entry in the whats-new, @aladinor)

Otherwise @TomNicholas can do that while preparing for the release.

Thanks @TomNicholas and @keewis for your guidance!


Support for ECMAScript module

What version of Ajv you are you using?
6.12.5
What problem do you want to solve?
Hello, I'm opening this issue because I did not find other discussions about that, please redirect me on the proper issue if you are already discussing it. I was looking for any solution for the Angular 10 warnings (CommonJS or AMD dependencies can cause optimization bailouts.), rather that simply ignore them by configuring allowedCommonJsDependencies in angular.json.
What do you think is the correct solution to problem?
I realized that a ECMAScript module would be needed to make Angular happy. Is there any plan to build ajv in that way?
Will you be able to implement it?
Unfortunately not

Does v7 solve it?

I just tried with 7.0.0-beta.3 and the warning is still there
WARNING in validate.ts depends on 'ajv'. CommonJS or AMD dependencies can cause optimization bailouts.
For more info see: https://angular.io/guide/build#configuring-commonjs-dependencies

Probably the new version changed the interface, now I got errors:
ERROR in validate.ts:5:13 - error TS2349: This expression is not callable.
  Type 'typeof import("/app/node_modules/ajv/dist/ajv")' has no call signatures.

5 const ajv = Ajv({

I changed:
import * as Ajv from "ajv";

const ajv = Ajv({
  allErrors: true,
  strictDefaults: true,
  strictKeywords: true,
  strictNumbers: true,
});

into:
import Ajv from "ajv";

const ajv = new Ajv({
  allErrors: true,
});

(strictDefaults, strictKeywords, strictNumbers were no longer recognized)
With this fix the error at compile time disappeared, but then I have troubles at runtime. Please note that I'm trying a fast upgrade from v6 to v7 without studying any migration guide or changelogs, just to verify the commonjs dependencies warning. Probably I'm doing something wrong, but I think that it is no so important to make it work now (I'm waiting the final v7 release before starting the upgrade).
Can I test something more? Do you have any suggestion to change the import or code to prevent the warning?

Type 'typeof import("/app/node_modules/ajv/dist/ajv")' has no call signatures.

Ajv is ES6 class in v7, it is not possible to make it callable without new.

(strictDefaults, strictKeywords, strictNumbers were no longer recognized)

strict mode is now enabled by default, it includes the functionality of all there option and some other restrictions - see docs/strict-mode.md

I have troubles at runtime.

What exactly? It would help to know what fails.
v7 is functionally backwards compatible, excluding draft-04 schemas support, but it has different defaults and formats were split to ajv-formats package. "Fast upgrade" can probably work for trivial cases, but in most cases it would help reviewing the release notes...

Do you have any suggestion to change the import or code to prevent the warning?

import Ajv from "ajv" works from me without any warnings... There may be some changes needed in tsconfig.json?
What is the code that generates the warning?
Ajv is exported in this way, to be backwards compatible:
export default class Ajv { /* ... */ }
module.exports = Ajv
module.exports.default = Ajv

The issue I recently discovered and not yet sure how best to change that these exports overwrite all other exports that are not types.
For example, even though I have these exports in Ajv:
export {_, str, stringify, nil, Name, Code, CodeGen, CodeGenOptions} from "./compile/codegen"

I can import types Name etc., but not other things from this export.
I didn't look yet how best to address it.

Thank you for the help @epoberezkin, I would like to be more helpful in my reporting but...

What exactly? It would help to know what fails.

The problem is that I don't really understand the runtime error.
This is the stack:
Observable.js:53 TypeError: You provided 'null' where a stream was expected. You can provide an Observable, Promise, Array, or Iterable.
    at subscribeTo (subscribeTo.js:27)
    at innerSubscribe (innerSubscribe.js:69)
    at CatchSubscriber.error (catchError.js:36)
    at MapSubscriber._error (Subscriber.js:75)
    at MapSubscriber.error (Subscriber.js:55)
    at CatchSubscriber.notifyError (innerSubscribe.js:45)
    at SimpleInnerSubscriber._error (innerSubscribe.js:13)
    at SimpleInnerSubscriber.error (Subscriber.js:55)
    at Observable._Observable__WEBPACK_IMPORTED_MODULE_0__.Observable.scheduler.schedule.error.error [as _subscribe] (throwError.js:4)
    at Observable._trySubscribe (Observable.js:42)

And the offending line seems to be this:
  const validator = ajv.getSchema("#/definitions/" + ref);

if I stop my validation function just before this line no exception is raised. It works fine with 6.12.6 (I only changed the version in package.json, changed the import and change the object creation by adding the new).

What is the code that generates the warning?

I'm not very helpful again... I tried to remove both the lines from dist/ajv.js (is the correct file?) but the warning is still there
However, I don't want to waste your time with badly reported issues and confused errors. If my reports are helpful to you to understand the issue, we can continue, otherwise I may back to you with better reporting once migrated my app to ajv 7 (but I would like to do this after the official release)
Thank you again for your help!

The error is unrelated to ajv, it is because ajv.getSchema("#/definitions/" + ref); returns undefined, but it's not clear why - what are Ajv options and which schema is returned by this call in v6?

I may back to you with better reporting once migrated my app to ajv 7 (but I would like to do this after the official release)

Sure - I will keep the issue open for now

Once back to v6 I put a couple of console.log around the ajv.getSchema
 48  console.log(ref);
 49  const validator = ajv.getSchema("#/definitions/" + ref);
 50  console.log(validator);

->
validate.ts:48 User

validate.ts:50 ƒ (data, dataPath, parentData, parentDataProperty, rootData) { 'use strict';  var vErrors = null;  var errors = 0;      if (rootData === undefined) rootData = data;  if ((data && typeof data === "objec…

"#/definitions/User" is added with a  ajv.addSchema by getting the json schema from a json file created with ts-json-schema-generator from a typescript interface
json file:
{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "definitions": {
    [...]
    "Group": {
      "additionalProperties": false,
      "properties": {
        "fullname": {
          "type": "string"
        },
        "members": {
          "items": {
            "additionalProperties": false,
            "properties": {
              "email": {
                "format": "email",
                "type": "string"
              },
              "name": {
                "type": "string"
              },
              "surname": {
                "type": "string"
              },
              "uuid": {
                "type": "string"
              }
            },
            "required": [
              "email",
              "name",
              "surname",
              "uuid"
            ],
            "type": "object"
          },
          "type": "array"
        },
        "shortname": {
          "type": "string"
        },
        "uuid": {
          "type": "string"
        }
      },
      "required": [
        "fullname",
        "shortname",
        "uuid"
      ],
      "type": "object"
    },
    [...]
    "User": {
      "additionalProperties": false,
      "properties": {
        "email": {
          "format": "email",
          "type": "string"
        },
        "first_login": {
          "format": "date-time",
          "type": [
            "string",
            "null"
          ]
        },
        "group": {
          "anyOf": [
            {
              "$ref": "#/definitions/Group"
            },
            {
              "type": "null"
            }
          ]
        },
        "isAdmin": {
          "type": "boolean"
        },
        "isCoordinator": {
          "type": "boolean"
        },
        "isStaff": {
          "type": "boolean"
        },
        "is_active": {
          "type": "boolean"
        },
        "last_login": {
          "format": "date-time",
          "type": [
            "string",
            "null"
          ]
        },
        "last_password_change": {
          "format": "date-time",
          "type": [
            "string",
            "null"
          ]
        },
        "name": {
          "type": "string"
        },
        "privacy_accepted": {
          "type": "boolean"
        },
        "roles": {
          "additionalProperties": {
            "type": "string"
          },
          "type": "object"
        },
        "surname": {
          "type": "string"
        },
        "uuid": {
          "type": "string"
        }
      },
      "required": [
        "email",
        "first_login",
        "group",
        "isAdmin",
        "isCoordinator",
        "isStaff",
        "is_active",
        "last_login",
        "last_password_change",
        "name",
        "privacy_accepted",
        "roles",
        "surname",
        "uuid"
      ],
      "type": "object"
    }
  }
}

Options in v6:
const ajv = Ajv({
  allErrors: true,
  strictDefaults: true,
  strictKeywords: true,
  strictNumbers: true,
});

options in v7:
const ajv = new Ajv({
  allErrors: true
});

Btw, the exports are now targeting ESM modules and typescript - see Getting started. Yet to look at the issue.

Going to close it. Please review the release notes - there are some changes in v7 that in many cases would make quick upgrade not possible. Please resubmit a specific failing case if it is still there.

Hello @epoberezkin
I'm just testing the upgrade and I confirm the resolution of this issue, thank you for the effort
I fixed (or better, bypassed) the runtime error reported above by enabling the standalone generation of validation function. This works well with v7.0.1 except for the execution of tests with karma that fail due to duplicated validation functions as reported in #1361.
v7.0.2 fixed this problem but it seems that introduces a new problem with ajv-formats, I will open a new issues for that


Close/quit webDriver in future thread

Hi Andrey,
Sometimes i have problems with calling driver.close () or driver.quit () method, and test always is suspended too long timeout. I think we should close the driver in hockable thread (in com.codeborne.selenide.impl.WebDriverThreadLocalContainer#closeWebDriver(java.lang.Thread))
private synchronized static void closeWithTimer(WebDriver driver) {
        THREAD_DRIVER_STATE_STORAGE.markDriverIsClose();
        Future<?> future = Executors.newSingleThreadExecutor().submit((Runnable) driver::close);
        try {
            future.get(100, SECONDS);
        } catch (InterruptedException | java.util.concurrent.TimeoutException e) {
            throw new SkipException("Long wait for killing browser", e);
        } catch (ExecutionException e) {
            throw new SkipException("Error on browser kill " + e.getMessage(), e);
        }
    }

Nice! Let's do it.
Actually we also experience problems with closing webdrivers, so it will help us too.

Done in Selenide 2.20
Timeout is 5 seconds by default. Can be configured via system property selenide.closeBrowserTimeout (default value is 5000).


Respect SWIFTPM_MACOS_DEPLOYMENT_TARGET if it is set in the environment

Respect SWIFTPM_MACOS_DEPLOYMENT_TARGET if it is set in the environment, otherwise stay with the default minimum deployment target of 10.10 on macOS.
ToolsSupportCore has availability annotations to represent requirements on various versions of macOS.  Yet ToolsSupportCore itself doesn't specify a minimum deployment target for macOS, and so it always builds for macOS 10.10.
This aligns with both swift-package-manager and with swift-driver, and is important now that ToolSupportCore has started containing some code that is annotated with availability for Darwin platforms (Netrc support required 10.13 or later).  One can discuss whether the environment variable should be called SWIFTTSC_MACOS_DEPLOYMENT_TARGET, but since swift-driver uses the SwiftPM setting (and this is what the Swift toolchain sets), it seems reasonable to stick with that.  Using the same environment variable also avoids having to change the toolchain build configuration.

@swift-ci please test

@swift-ci please test linux

Looks like the same issue as in https://github.com/apple/swift-tools-support-core/pull/218, which appears to be an infrastructure issue.  @shahmishal

@swift-ci please test linux

leaking SWIFTPM_MACOS_DEPLOYMENT_TARGET seems wrong to me, if TSC and SwiftPM must share a setting it should be named something more generic

leaking SWIFTPM_MACOS_DEPLOYMENT_TARGET seems wrong to me, if TSC and SwiftPM must share a setting it should be named something more generic

Yes, I agree, and did this because SwiftDriver is already using but — but perhaps that was a bit hasty.  We should change the one in SwiftDriver then as well, if we have a generic one, or set all three independently.  We basically want to build the components of the toolchain with the minimum deployment target the toolchain supports, and ideally keep them the same.

leaking SWIFTPM_MACOS_DEPLOYMENT_TARGET seems wrong to me, if TSC and SwiftPM must share a setting it should be named something more generic

Yes, I agree, and did this because SwiftDriver is already using but — but perhaps that was a bit hasty. We should change the one in SwiftDriver then as well, if we have a generic one, or set all three independently. We basically want to build the components of the toolchain with the minimum deployment target the toolchain supports, and ideally keep them the same.

What would you think about having separate names and then having the toolchain build script set them all the same?

What would you think about having separate names and then having the toolchain build script set them all the same?

wither that or use a generic one like SWIFT_TOOLCHAIN_MACOS_DEPLOYMENT_TARGET

What would you think about having separate names and then having the toolchain build script set them all the same?

wither that or use a generic one like SWIFT_TOOLCHAIN_MACOS_DEPLOYMENT_TARGET

That would probably be better, especially as we extend this to other platforms that have deployment targets.  @artemcm any preference?

@swift-ci please test


Fix crash when click on rotate camera icon + take pic fast

App crashes because session is null but user tries to take pic ==> NPE
java.lang.NullPointerException: Attempt to invoke virtual method 'int android.hardware.camera2.CameraCaptureSession.capture(android.hardware.camera2.CaptureRequest, android.hardware.camera2.CameraCaptureSession$CaptureCallback, android.os.Handler)' on a null object reference
at com.google.android.cameraview.Camera2.lockFocus(Camera2.java:587)
at com.google.android.cameraview.Camera2.takePicture(Camera2.java:338)
at com.google.android.cameraview.CameraView.takePicture(CameraView.java:407)

I signed it!

I signed it!

I signed it!

@dilix, do I understand that the way this fix works is to only set the onClickListener of the FAB once the camera session is actually ready?

@drd in my case - yes.
The problem was that user could try to make a call before the cam is really ready.
So the fix is to provide a callback to notify when cam is ready to receive input.

@dilix How do you make the picture taking faster?

I don't make it faster I just restrict taking photo if camera is not ready
because this case crashes for me.
On Sat, Nov 18, 2017, 07:53 吴迪<EMAIL_ADDRESS>wrote:

@dilix https://github.com/dilix How do you make the picture taking
faster?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/google/cameraview/pull/175#issuecomment-345423186,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABVJ6EbZgDRV-5B9W-5UibQwa_WJ7OeQks5s3n7wgaJpZM4OJHYk
.

--
Best regards,
Dmitry.

I'm using: compile 'com.github.google:cameraview:9bbe07ba4474'
How can i update the build including this fix?
Thanks.

Repo is deprecated, close PR.

Hmmm, people have a Samsung Galaxy S7 phone and the camera does not autofocus. There is no rotate theme going on, at least not from the photographer's perspective. It just doesn't autofocus.  My guess would be to go to a store and download new camera software. Tried safe mode after reboot, deleting all cache and other memory. Autofocus just seemed to have gone bad. Manual focus also does not seem to work. Sometimes yes - when I am in the focus zone ! ;).  But no, the mechanism just does not focus. Don't understand the rotation story although appreciate the research you put into this! - John


INFO WORK

Hello Steven, congratulations on your extraordinary work. I've always wondered but how do you compile the host file and keep it updated? do you have special computers or software?
Best regards
Edward ( from italy)

Thank you for the kind words Edward @eglabitalia.
It's mostly manual processes.  Here is my main workflow:

Run a git pull locally.
Execute makeHosts here locally to refresh data and compile new hosts files.  This refreshes data from 27 different sources, and produces the 16 different amalgamated hosts files we offer here.
I use Git Tower to display the diffs for each data source.  I look at every change from all sources, paying more attention to domains added than domains removed. I sometimes catch mistakes at this stage.  When I do, i abort the release, contact the source curator, ask questions.  This can delay the release by a day or two.
Once everything looks good, I commit changes and push to GitHub.
Next I use @webpro's most excellent release-it to create a release tag, commit, and push the release on GitHub.

Other than that, I watch our Issues page as often as I can, and try to stay abreast of what's happening in the security space, every day.
That's how I work.  Thank you for asking!

Thanks for the reply Steven.  Congratulations again for the great job you do for all of us.
Best regards
Edward

Your welcome Edward @eglabitalia.  This repo is the collective work of many contributors and, I'm happy to say, they are all awesome people.


A few questions regarding the development of this repo

Hey first of all tysm for making this!
I have a few questions regarding your project. I want to extend its functionality to be able delete or compress some junk data of apps like WhatsApp. At least on Android you can safely remove some files that are otherwise not easily deletable via the app natively but on iOS (since there is no file access) this is not possible which is why I want to attempt doing just that via backups.

During development, did you use any kind of profiler? I'm asking because as soon as I open some folders in the WhatsApp container it will start to freeze with the CPU usage going close to 400% for half an hour until I close the process. I'd be happy to investigate but I'd like to know whether that's something you've already done before

Commit 985906ea0fe99b456ad898b3efb3cdb5c0563987 that adds the possibility to replace files also comes with the reEncryptDatabase() function call. Does that work in a way that I can straight up "edit" any file in a backup, then play back that backup (that then contains the edited file) onto the device, essentially editing the otherwise closed-down iOS file system?

As I have previously mentioned, I'd like to be able to manage some common storage-hungry data. That would be something along the lines of "we don't need full resolution images in there, let's compress them" for example - would you be willing to merge such additions or should I run it as an independent project, only merging your upstream changes each time? I'd be fine with both ways, I'd just like to know your preference on that.

Hey @Reiszecke!
I'm sorry that it took so long. I don't know why the GH mobile app doesn't send notifications.
That functionality is a great idea. A while ago, I actually had a problem where iMessage used up a large amount of storage on an iPad and there was no way to find or delete anything on the device. The backup explorer totally makes it possible. For a specific use case like that manual deletion is enough, but for WhatsApp and freeing space up in general an automatic tool would be awesome.

Regarding your first question: I think the algorithm that reassembles the tree folder structure in the Files tab is quite inefficient. I profiled it once, but I haven't focused much on performance yet. Especially because I haven't had problems anywhere near what you described. Could you share your system specs and about how much space WhatsApp takes up on your phone? Does it only happen in the WhatsApp container or is it slow everywhere?
Yes, exactly. I have to admit that I haven't gotten around to testing restoring the backup to the phone yet. I'm pretty sure it works though. I have an older iPhone lying around and will try this out soon.
Yeah, definitely. There is still plenty of room left for additional tabs in the program. :)

Hey man don't worry about it :)
Yeah actually I had the issue on iMessage first until I moved everything to WhatsApp which then showed the same bloaty behavior hahahaha

Does it only happen in the WhatsApp container or is it slow everywhere

I've only noticed it in the WhatsApp directory, but I haven't tried out many others since they probably are all smaller than this stupid WA folder hahaha

system specs

Early 2015 MBA (2.2 Dual Core i7), Big Sur 11.5
Backup is from an iPhone 7 32GB

I have an older iPhone lying around and will try this out soon

I may also soon have an extra iPhone but none that I can risk bricking yet, so looking forward to your info 😅

There is still plenty of room left for additional tabs in the program

That sounds great, happy to hear you are open for additions. Currently a lot more busy again than I was a few weeks ago but let's see :)
To the performance: The backup (32GB phone) is 11 GB in size
As soon as I hit the "Files" tab, after entering the password in the prompt it takes about 15 seconds to load the browser's list on the left tab.
Performance in the java files browser in the right tab:
Hitting Application Groups > AppDomainGroup...WhatsApp.private (almost no data in there) opens somewhat immediately
Hitting Application Groups > AppDomainGroup...WhatsApp.shared at first looks like it's the same contents but if you switch back and forth you notice that it's actually loading something up, the fans start reving.

how much space WhatsApp takes up on your phone

I have closed the jar, opened it again but this time let it load on first tap (so only 100% CPU usage then), and it took exactly 1:40 min to open the Application Groups > AppDomainGroup...WhatsApp.shared folder. To figure out its size I have exported it which took 7 minutes for the WhatsApp folder that is 4.8gb in size.
What's interesting is that with my WhatsApp Business installation, the .shared folder opens almost immediately. The folder structure is fairly similar so it's not necessarily the amount of files and folders in the root dir of the WhatsApp Application Group but rather about the literal size (or the amount of files that come along with that). Is there a chance that it recursively iterates to each subdir too? For example because instead of processing each sub-dir's contents on-demand, it may build the tree for the selected folder upfront? That would explain why a similar folder with less contents nested in subfolders is so much faster.


TextFormField forces whole screen rebuild method on focus change.

I tried to implement a simple form with 2 text inputs, but I encountered a weird problem. When the form is displayed on the next page opened via
 Navigator.of(context).pushReplacement(...)

every time when inputs change focus the whole screen rebuilds. Below I added simple example to reproduce this issue.
Code to reproduce issue

import 'package:flutter/material.dart';

void main() => runApp(MyApp());

class MyApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Form Validation Demo',
      home: MyCustomForm(),
    );
  }
}

class MyCustomForm extends StatefulWidget {
  MyCustomForm({Key key}) : super(key: key) {
    print('New widget!');
  }

  @override
  MyCustomFormState createState() {
    return MyCustomFormState();
  }
}

class MyCustomFormState extends State<MyCustomForm> {
  final _formKey = GlobalKey<FormState>();

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text('Title'),
      ),
      floatingActionButton: FloatingActionButton(
        child: Icon(Icons.arrow_forward),
        onPressed: () {
          Navigator.of(context).pushReplacement(
              MaterialPageRoute(builder: (BuildContext context) => MyCustomForm()));
        },
      ),
      body: Form(
        key: _formKey,
        child: Column(
          crossAxisAlignment: CrossAxisAlignment.start,
          children: <Widget>[
            Padding(
              padding: const EdgeInsets.all(8.0),
              child: TextFormField(),
            ),
            Padding(
              padding: const EdgeInsets.all(8.0),
              child: TextFormField(),
            ),
          ],
        ),
      ),
    );
  }
}

Steps to Reproduce issue

Open app
Change focus from first to second input

New widget! is printed once on app start

Click on FAB
Change focus from first to second input

New widget! is printed everytime focus is changed

Enviroment
iOS simulator -> iPhone XR - 12.1

Although it is closed, still the root cause of it is something else which I think should be fixed.
Take a look at this issue: https://github.com/flutter/flutter/issues/37878

@zoechi thanks for your responses :) For people encountering the same issue. Just put your initial state in initState. I know, it makes sense, right? Even though the widget is re-created, initState is fired only once, so you will not encounter problems with a re-created state. :)
Should be:
class ExampleWidget extends StatefulWidget {
  final initialData;

  ExampleWidget({this.initialData, Key key}) : super(key: key);

  @override
  ExampleWidgetState createState() {
    return ExampleWidgetState();
  }
}

class ExampleWidgetState extends State<ExampleWidget> {
  ViewModel _viewModel;

  @override
  void initState() {
    super.initState();
    _viewModel = ViewModel(
      initialData = widget.initialData,
    );
  }

  @override
  Widget build(BuildContext context) {
    return Text(_viewModel.initialData.toString());
  }
}

instead of:
class ExampleWidget extends StatefulWidget {
  final ViewModel viewModel;

  ExampleWidget(this.initialData, {Key key})
      : viewModel = ViewModel(
          initialData = initialData,
        ),
        super(key: key);

  @override
  ExampleWidgetState createState() {
    return ExampleWidgetState();
  }
}

class ExampleWidgetState extends State<ExampleWidget> {
  @override
  Widget build(BuildContext context) {
    return Text(widget.viewModel.initialData.toString());
  }
}

a lot of thanks ! this solve my problem !! :) 👍

@zoechi thanks for your responses :) For people encountering the same issue. Just put your initial state in initState. I know, it makes sense, right? Even though the widget is re-created, initState is fired only once, so you will not encounter problems with a re-created state. :)
Should be:
class ExampleWidget extends StatefulWidget {
  final initialData;

  ExampleWidget({this.initialData, Key key}) : super(key: key);

  @override
  ExampleWidgetState createState() {
    return ExampleWidgetState();
  }
}

class ExampleWidgetState extends State<ExampleWidget> {
  ViewModel _viewModel;

  @override
  void initState() {
    super.initState();
    _viewModel = ViewModel(
      initialData = widget.initialData,
    );
  }

  @override
  Widget build(BuildContext context) {
    return Text(_viewModel.initialData.toString());
  }
}

instead of:
class ExampleWidget extends StatefulWidget {
  final ViewModel viewModel;

  ExampleWidget(this.initialData, {Key key})
      : viewModel = ViewModel(
          initialData = initialData,
        ),
        super(key: key);

  @override
  ExampleWidgetState createState() {
    return ExampleWidgetState();
  }
}

class ExampleWidgetState extends State<ExampleWidget> {
  @override
  Widget build(BuildContext context) {
    return Text(widget.viewModel.initialData.toString());
  }
}

It is for the case of StatefulWidget but what about stateless widget.
Whenever the onscreen keyboard pops up, my whole screen is refreshed.

@zoechi thanks for your responses :) For people encountering the same issue. Just put your initial state in initState. I know, it makes sense, right? Even though the widget is re-created, initState is fired only once, so you will not encounter problems with a re-created state. :)
Should be:
class ExampleWidget extends StatefulWidget {
  final initialData;

  ExampleWidget({this.initialData, Key key}) : super(key: key);

  @override
  ExampleWidgetState createState() {
    return ExampleWidgetState();
  }
}

class ExampleWidgetState extends State<ExampleWidget> {
  ViewModel _viewModel;

  @override
  void initState() {
    super.initState();
    _viewModel = ViewModel(
      initialData = widget.initialData,
    );
  }

  @override
  Widget build(BuildContext context) {
    return Text(_viewModel.initialData.toString());
  }
}

instead of:
class ExampleWidget extends StatefulWidget {
  final ViewModel viewModel;

  ExampleWidget(this.initialData, {Key key})
      : viewModel = ViewModel(
          initialData = initialData,
        ),
        super(key: key);

  @override
  ExampleWidgetState createState() {
    return ExampleWidgetState();
  }
}

class ExampleWidgetState extends State<ExampleWidget> {
  @override
  Widget build(BuildContext context) {
    return Text(widget.viewModel.initialData.toString());
  }
}

It is for the case of StatefulWidget but what about stateless widget.
Whenever the onscreen keyboard pops up, my whole screen is refreshed.

You have to put it as Stateful Widget. Otherwise, Stateless will create new instance each time TextField is focused


1.7 REGRESSOIN: Prometheus cannot scrape pods when enablePrometheusMerge disabled

Bug description
When meshConfig.enablePrometheusMerge is disabled Prometheus fails to scrape pods with injected sidecars with server returned HTTP status 503 Service Unavailable error.
[ ] Docs
[ ] Installation
[x] Networking
[ ] Performance and Scalability
[x] Extensions and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
Expected behavior
Prometheus should be able to scrape the pods over mTLS when enablePrometheusMerge is disabled and provisionPrometheusCert is enabled.
Steps to reproduce the bug
Deploy Istio 1.7.x with the following config:
---
apiVersion: install.istio.io/v1alpha2
kind: IstioOperator
spec:
  profile: default
  namespace: istio-system
  components:
  values:
    prometheus:
      enabled: true
    grafana:
      enabled: true
    meshConfig:
      enablePrometheusMerge: false
      enableAutoMtls: true
   global:
      controlPlaneSecurityEnabled: true

And then deploy an app with prometheus metrics in a sidecar injectable namespace, e.g.:
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: prometheus-example-app
  name: prometheus-example-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: prometheus-example-app
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
        prometheus.io/path: /metrics
      labels:
        app.kubernetes.io/name: prometheus-example-app
    spec:
      containers:
      - name: prometheus-example-app
        image: quay.io/brancz/prometheus-example-app:v0.3.0
        ports:
        - name: web
          containerPort: 8080

Version (include the output of istioctl version --remote and kubectl version --short and helm version if you used Helm)
$ istioctl version --remote
client version: 1.6.8
control plane version: 1.7.3
data plane version: 1.7.1 (1 proxies), 1.7.3 (2 proxies)
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T13:41:02Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.12-gke.500", GitCommit:"98a221da900b81cfd64fa0e00bfad2a084ea918b", GitTreeState:"clean", BuildDate:"2020-09-21T09:19:38Z", GoVersion:"go1.13.15b4", Compiler:"gc", Platform:"linux/amd64"}

How was Istio installed?
istioctl manifest generate -f values.yml | kubectl apply -f -

Environment where bug was observed (cloud vendor, OS, etc)
GKE 1.19
Notes:
I'm trying to scrape metrics over mTLS so I disabled enablePrometheusMerge which will make istioctl to manually inject istio-proxy and save the cert in tmpfs and configures Prometheus to use those certs. After some investigation turns out that if you try to curl the metrics port using HTTP2 it will fail with: upstream connect error or disconnect/reset before headers. reset reason: connection termination, but if you curl with HTTP1.1 it works. And turns out that Prometheus uses HTTP2.
E.g.:
# this works
kubectl exec -n istio-system prometheus-6556574667-ggxm8 -c istio-proxy -- curl https://<IP_ADDRESS>:8080/metrics --cacert /etc/istio-certs/root-cert.pem --cert /etc/istio-certs/cert-chain.pem --key /etc/istio-certs/key.pem -k --http1.1 -sS

# this doesn't
kubectl exec -n istio-system prometheus-6556574667-ggxm8 -c istio-proxy -- curl https://<IP_ADDRESS>:8080/metrics --cacert /etc/istio-certs/root-cert.pem --cert /etc/istio-certs/cert-chain.pem --key /etc/istio-certs/key.pem -k --http2 -sS

@howardjohn might be interested in this.

I don't see provisionPrometheusCert in your IstioOperator?

@howardjohn sorry, didn't include it, but the default value is true when merge is disabled.


Feature/ted 1177

Codecov Report
Base: 98.27% // Head: 98.19% // Decreases project coverage by -0.09% :warning:

Coverage data is based on head (5e417cf) compared to base (d6a12c8).
Patch coverage: 91.83% of modified lines in pull request are covered.

:exclamation: Current head 5e417cf differs from pull request most recent head 8c10a52. Consider uploading reports for the commit 8c10a52 to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #425      +/-   ##
==========================================
- Coverage   98.27%   98.19%   -0.09%
==========================================
  Files         128      128
  Lines        6604     6631      +27
==========================================
+ Hits         6490     6511      +21
- Misses        114      120       +6

Impacted Files
Coverage Δ

ted_sws/data_manager/adapters/sparql_endpoint.py
92.47% <80.00%> (-7.53%)
:arrow_down:

...services/check_availability_of_notice_in_cellar.py
73.21% <90.47%> (+9.57%)
:arrow_up:

...s/data_manager/adapters/supra_notice_repository.py
100.00% <100.00%> (ø)

ted_sws/master_data_registry/resources/__init__.py
100.00% <100.00%> (ø)

ted_sws/notice_validator/resources/__init__.py
100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


ListLogEntries(Async) and Quota exceeded Read requests per minute

Hi experts,
I have the following error
Status(StatusCode="ResourceExhausted", Detail="Quota exceeded for quota metric 'Read requests' and limit 'Read requests per minute' of service 'logging.googleapis.com' for consumer 'project_number:masked'.")

when using the PagedEnumerable from ListLogEntries method as a plain enumerable like:
    public static IEnumerable<Google.Cloud.Logging.V2.LogEntry> ReadCloudLogs(string filter)
    {
        var client = LoggingServiceV2Client.Create();

        var projectName = new ProjectName(Environment.GetEnvironmentVariable("PROJECT_NAME"));

        return client.ListLogEntries(new [] { projectName }, filter, "timestamp", pageSize: 1000);
    }

Seems, ListLogEntries makes requests too fast. The possible (maybe a bit ugly) workaround is to call manually ReadPage / ReadPageAsync and add an artificial Task.Delay:
    public static async Task<IEnumerable<Google.Cloud.Logging.V2.LogEntry>> ReadCloudLogs(string filter)
    {
        const int pageSize = 1000;

        var client = await LoggingServiceV2Client.CreateAsync();

        var projectName = new ProjectName(Environment.GetEnvironmentVariable("PROJECT_NAME"));

        var logEntries = new List<Google.Cloud.Logging.V2.LogEntry>();

        var logEntriesResponse = client.ListLogEntriesAsync(new [] { projectName }, filter, "timestamp", pageSize: pageSize);

        var hasData = true;

        while (hasData)
        {
            var currentSize = logEntries.Count;

            var currentPage = await logEntriesResponse.ReadPageAsync(pageSize);

            logEntries.AddRange(currentPage);

            hasData = currentSize < logEntries.Count;

            if (hasData)
                await Task.Delay(TimeSpan.FromSeconds(1.5)); // no more than 40 request per minute
        }

        return logEntries;
    }

I believe this should be somewhere in the options, I probably missed something and you already have it, if so, point me out to this, please. Thanks!
Environment details

OS: Docker container mcr.microsoft.com/dotnet/runtime-deps:8.0
.NET version: 8.0
Package name and version: Google.Cloud.Logging.V2, version 4.1.0

Steps to reproduce
See above

ListLogEntries is called lazily - as you iterate through the returned entries. The client library can't really know "how fast is too fast" - if you want to add a delay of a 1.5s for every thousand entries, I'd do so like this instead (not tested, so there may be syntax issues, but it should show the basic approach):
public IAsyncEnumerable<LogEntry> ReadCloudLogs()
{
    const int EntriesPerDelay = 1000;
    const int DelayMs = 1500;
    var client = await LoggingServiceV2Client.CreateAsync();
    var projectName = new ProjectName(Environment.GetEnvironmentVariable("PROJECT_NAME"));
    int count = 0;
    foreach (var entry in client.ListLogEntriesAsync(new[] { projectName}, pageSize: 1000))
    {
        yield return entry;
        count++;
        if (count % EntriesPerDelay == 0)
        {
            await Task.Delay(DelayMs).ConfigureAwait(false);
        }
    }
}

I don't think this is something we'd add into the client libraries - but you could easily write your own extension method to do the delay.


Bad header rive_canvas.js:2936 warning & Problem loading file; may be corrupt

I have a LoadingComponent with rive, the .riv file is placed in the root of the public folder.

This same component is called from two other components, the animation works correctly in the first component, but not in the second, the only clue I have are these two warnings.

I'm not sure if I'm doing something wrong or if it's a bug from the library itself.

Hey @Jackdaw16, it's hard to tell exactly what the issue could be from your sample code, but I don't think it's that your .riv file is actually corrupt if it's working in one component. Not sure if this is a library bug or not but I suspect there could be some issue with the path in the src section of the params and the runtime is having issues loading the file. If that is the case we should have a better error message!
How are you running/bundling your app? Would it be possible to create a small github repo that reproduces this issue with the two components? I've tried to reproduce the issue you've seen myself but can't seem to.

Hi, Here I attach some screenshots of how I have structured the application. The application is running with craco, for Tailwindcss. Between today or tomorrow I will do a simulation with a small project and create a Github repository.
what is indicated in red is where the library is used, the components and where the .riv is located.

I was try to reproduce the same error in a smaller project without getting the same result.
But, I tried such a simple thing, it seems silly, but it worked out.
I added two points to the route and it worked perfectly.

I don't really know why, because in the example I made with a single point, the animation worked just the same.
https://github.com/Jackdaw16/example-rive
but thank you very much and a really useful library, actually teach a little course on this library and Rive.


Evaluate database engines according to requirements and decide which one to optimize

Problem definition
Based on the requirements in #63 , we need to evaluate the different database engines in order to find one that satisfies most of them based on a criteria.
Some preliminary work has been done in RFC 001 and insights from the community can be found in the comments of tendermint/tendermint#6032.
DoD

[ ] An RFC containing:
- The different databases evaluated and the reasoning behind their choice.
- An explanation of the methodology of the evaluation.
- Their ranking based on the requirements.
- Recommendation on which database to choose.
[ ] Optimize the chosen database based on the requirements established in #67 and feedback from users (#68)

Original issue tendermint/tendermint#9944

DB experiments.pdf

relevant https://github.com/smallnest/kvbench


ulauncher spotify extension crashed

the extension crashed
what should i do

Hi! Could you please confirm that you have followed the Installation section from readme? If you didn't, please do that first. If you did, please provide more details, eg what is your OS, what version of Python are you running etc.


Fix clicking on a profile in azure connection dialog

This PR fixes #

Coverage decreased (-0.01%) to 40.352% when pulling a01b44eb6e30ace2470c09a82db029c3101f8ae1 on fix/connectionDialog into b2a1738836d250e54814fae556b9160674e7f87c on main.


better multicall

hi, I think here is a better multicall, because it doesn't need deploy https://github.com/indexed-finance/multicall
but it only supports js, if only you can support it in go :)

that contract actually looks pretty cool. You should be able to use abigen to create native go bindings


Fixed validations bugs

Added check: username can only start with letter
Closes: #1
Fixes #<issue_number_goes_here>

It's a good idea to open an issue first for discussion.

[ ] Tests pass
[ ] Appropriate changes to README are included in PR

@googlebot I fixed it


Concern about signature used in Cardano

Dear developers, I'm wondering about the signature used in Cardano. Is it a general version of Ed25519, or I mean does the message need to be hashed before signing? According to the NIST standard, there are two versions of Ed25519, there are EdDSA and HashEdDSA. Looking forward to hearing from you.

Hi, we use the cryptonite Crypto.PubKey.Ed25519 library for signing transactions. The transaction body is hashed before signing.

I see, thanks a lot.

I see, thanks a lot.

Correction @QiFengWHu - there are two payment signing schemes in Cardano Shelley - "normal" and "extended". cardano-wallet only uses "extended" Ed25519. See BIP32-Ed25519 for details and cardano-crypto for the implementation.

The cardano-crypto for the implementation is really helpful. You are so nice to answer me by this resources.
By the way, the Ed25519_BIP.pdf is somehow invalid now, it's kindly unfortunate.


[Bug] 断点二次训练merge lora失败

Checklist

[X] 1. I have searched related issues but cannot get the expected help.
[ ] 2. The bug has not been fixed in the latest version.
[ ] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

Describe the bug
merge lora报错，我是在使用lora微调8b模型的脚本训练了一个checkpoint后又将本次checkpoint作为第二次训练的输入模型。在对第二次微调后的模型进行lora merge
Reproduction
merge_lora.py
Environment
sys.platform: linux
Python: 3.12.3 | packaged by Anaconda, Inc. | (main, May  6 2024, 19:46:43) [GCC 11.2.0]
CUDA available: True
MUSA available: False
numpy_random_seed:<PHONE_NUMBER>
GPU 0: NVIDIA GeForce RTX 4090
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.1, V12.1.105
GCC: gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
PyTorch: 2.3.0+cu121
PyTorch compiling details: PyTorch built with:
  - GCC 9.3
  - C++ Version: 201703
  - Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v3.3.6 (Git Hash 86e6af5974177e513fd3fee58425e1063e7f1361)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX512
  - CUDA Runtime 12.1
  - NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90
  - CuDNN 8.9.2
  - Magma 2.6.1
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=8.9.2, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.3.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=1, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF,

TorchVision: 0.18.0+cu121
LMDeploy: 0.5.3+6a230b3
transformers: 4.40.0
gradio: 4.41.0
fastapi: 0.112.0
pydantic: 2.8.2
triton: 2.3.1
NVIDIA Topology:
	GPU0	NIC0	NIC1	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	NODE	NODE	32-63,96-127	1		N/A
NIC0	NODE	 X 	PIX
NIC1	NODE	PIX	 X

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1

Error traceback
- This IS expected if you are initializing InternVLChatModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing InternVLChatModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of InternVLChatModel were not initialized from the model checkpoint at /root/autodl-tmp/mytrain_2/ and are newly initialized: ['language_model.base_model.model.model.layers.0.attention.wo.lora_A.default.weight', 'language_model.base_model.model.model.layers.0.attention.wo.lora_B.default.weight'
......]
Loading tokenizer...
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Traceback (most recent call last):
  File "/root/InternVL/internvl_chat/tools/merge_lora.py", line 25, in <module>
    model.language_model.merge_and_unload()
  File "/root/miniconda3/lib/python3.12/site-packages/peft/tuners/lora/model.py", line 838, in merge_and_unload
    return self._unload_and_optionally_merge(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/peft/tuners/lora/model.py", line 457, in _unload_and_optionally_merge
    target.merge(safe_merge=safe_merge, adapter_names=adapter_names)
  File "/root/miniconda3/lib/python3.12/site-packages/peft/tuners/lora/layer.py", line 472, in merge
    base_layer.weight.data = base_layer.weight.data + delta_weight
                             ~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~
  File "/root/miniconda3/lib/python3.12/site-packages/torch/_prims_common/wrappers.py", line 252, in _fn
    result = fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/torch/_prims_common/wrappers.py", line 137, in _fn
    result = fn(**bound.arguments)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/torch/_refs/__init__.py", line 1091, in add
    output = prims.add(a, b)
             ^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/torch/_ops.py", line 594, in __call__
    return self_._op(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/torch/_prims/__init__.py", line 359, in _prim_elementwise_meta
    utils.check_same_device(*args_, allow_cpu_scalar_tensors=True)
  File "/root/miniconda3/lib/python3.12/site-packages/torch/_prims_common/__init__.py", line 740, in check_same_device
    raise RuntimeError(msg)
RuntimeError: Tensor on device cpu is not on the expected device meta!

大佬解决了吗？

没有欸，目前还是调整数据后重启训练
一叶之秋 @.***> 于2024年9月28日周六 13:37写道：

大佬解决了吗？
—
Reply to this email directly, view it on GitHub
https://github.com/OpenGVLab/InternVL/issues/527#issuecomment-2380417249,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BETZR3RGU22ZSAAT4DMG3QLZYY6ATAVCNFSM6AAAAABM3YDAT2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOBQGQYTOMRUHE
.
You are receiving this because you authored the thread.Message ID:
@.***>

这个bug看着好像可以通过调整一下权重的device来解决


AllYAll updated to use github

No need for this, Spacedock got updated

No need for this, Spacedock got updated


FR: Matrix - Option to only show real relationships

(would want that feature in rl as well lol)
currently, we can only turn off implied siblings across different parent-hierarchy types. However, I would like to have a feature to turn off implied relationships entirely in the Matrix View.

Yeah, I can do that

Yeah, I can do that

I've added this in 2.20.0 :)

Thanks 🥳


ResourceEvent should be optional in query filters and providers

As a concrete example, let's say an entity has to be created from an RPC controller. A DoctrineResource::create method is then called. Thus, when calling it, data is filtered through a ZF\Apigility\Doctrine\Server\Query\CreateFilter\DefaultCreateFilter (by default, should no extra configuration provided) as shown here

In this concrete class (DefaultCreateFilter), the event is not used at all.
Although filtering is mandatory through the DoctrineResource::create implementation, the event may not be available.
That should not stopped the process of creating an entity through its dedicated Resource.

Originally posted by @jguittard at https://github.com/zfcampus/zf-apigility-doctrine/issues/280

@jguittard DoctrineResource extends AbstractResourceListener and method create is not called directly but via dispatch method so the event is there always available.
Can you show example when the event is not available in create method?

Originally posted by @michalbundyra at https://github.com/zfcampus/zf-apigility-doctrine/issues/280#issuecomment-256191169

I think he meant if you call DoctrineResource::create() directly since it's a public function. Maybe the API for the class can be simplified to avoid confusion by making the methods of that type protected in the future (but it's a BC break)

Originally posted by @gsomoza at https://github.com/zfcampus/zf-apigility-doctrine/issues/280#issuecomment-277415550

Oh, yeah, this is what he meant probably :) Thanks @gsomoza. I misunderstood. Sorry @jguittard :)
I have a lot of concerns about PR #281 and it seems to be also BC Break.
We can't change these methods to protected - as these are from ZF\Rest\AbstractResourceListener.
I'm afraid that the library doesn't support RPC endpoints at all right now. There is something in codebase already (DoctrineRpcServiceModel, ...), so before somebody has it in mind. Maybe we can add it.
Of course it will be nice to resolve also this issue somehow, to allow usage create/fetch/patch/... methods without the event. Hm... what if we override getEvent method to returns always ResourceEvent?
@jguittard @gsomoza thoughts?

Originally posted by @michalbundyra at https://github.com/zfcampus/zf-apigility-doctrine/issues/280#issuecomment-283003993


Generate XRechnung compliant XML invoice

Pull Request Summary
This commits handle generation of a XRechnung 3.0.1 invoice from a GOBL document. XRechnung supports two syntaxes that can be used in communication UBL and UN/CEFACT CII. Generation software needs to support at least one of those syntaxes while software that accepts XRechnung must accept both. The syntax of the document we decided to implement is the UN/CEFACT CII syntax since it is compatible not only with XRechnung but also with ZUGFeRD and FacturX. This PR does not check or validate if the generated document is 100% compatible with ZUGFeRD or FacturX since it focuses on compatibility with XRechnung.
Using the code from this PR we can generate a basic valid XRechnung 3.0.1 invoice or credit-note by providing a GOBL json file and running the CLI command:
go run ./cmd/gobl.xinvoice convert ./test/data/invoice-de-de.json > ./test/data/out/invoice-de-de.xml

Feedback
This PR handles only a basic invoice and the mechanism could be extended to handle correction invoices, self billed invoices, discounts and many more. We just need to specify what would do we need/want to support next.

@samlown Could you take a look at it, please?


Usability

Hi,
I am missing the part what data would be needed to apply this algorithm to a non-model organism.
Could you please indicate which data sets are needed and how they should be pre-processed to use the software, my feeling is  that otherwise it will be hard to use it even for a trained bioinformatician.

I appreciate your interest in our work, we've already trained the classification model based on multiple datasets. If you wish to conduct experiments on one of the datasets mentioned in the paper, you can simply download the data and the trained model and run  motif_extractor_example.ipynb.
However, if you wish to train your own model on your customed dataset, a simple way to use the software is by the following steps：

Transform your sequence data into the one-hot matrixes, example data can be found here

Modify the configuration file in experiments, and train your model via 5-fold cross-validation.

Conduct motif extractor by following the tutorials in motif_extractor_example.ipynb.

Thank you for your response.
Still, for me it is missing:

Clear explanation, how to create such one-hot matrixes (without needing to investigate individual exampla data sets myself, you are the experts and should explain how this needs to be done).

Clear explanation what are the best practise for configuration parameter, to be able to run a "valid" model training.

Example, how to apply the already trained model on an arbitray "new" species showing what input data is really necessary to get reasonable results.

Thank you in anticipation

Thank you for your interest in our work

For one-hot encoding for the input DNA sequences, each base is encoded as a vector of all zeros except one in a specific position, A is encoded as (1,0,0,0), T as (0,0,0,1), C as (0,1,0,0), and G as (0,0,1,0). The preprocessed hotspot sequences can be derived from  here.

You can also train on your customized data. You are able to crop the recombination hotspot sequences from the genome assembly GRCh38 (hg38) according to the recombination map(or so-called genetic map). Such recombination map can be different in resolution, available in Google Drive
The example recombination map can be found here:
Genetic map computed from the nature 2020 crossovers.
The data columns are as follows:
Chr (chromosome).
Begin (start point position of interval in GRCh38 coordinates).
End (end point position of interval in GRCh38 coordinates).
cMperMb (recombination rate in interval).
cM (centiMorgan location of end point of interval)
Chr	Begin	End	cMperMb	cM
num hot : 17300 num cold 1103150
hot_rate avg : 14.06 cold_rate 0.2243
chr1	829059	879059	3.68742022408169	0.2694653240675081
chr1	879059	929059	2.907389022833642	0.4148347752091902

For model training, you can just follow the optimal parameters illustrated in the experiment , example configuration is listed as follows:

{
"dataset":        "human_science_2019",
"num_data":       40000,
"max_length":     1000,
"min_length":     500,
"model":          "SeqModel_Attention",
"log_dir":        "logs",
"output_dir":     "output",
"output_prefix":  "SeqModel_Attention",
"input_length":1000,
"input_height":4,
"data_augmentation":1,
"hidden_dim":64,
"attention":1,
"equivariant":0,
"get_distribution":0,
"get_line_comparison":1,
"dp_rate":0.1,
"trails":4,
"folds":5,
"epochs":100,
"batch_size":64,
"lr":0.001,
"init_weight":0,
"reinforce_train":0,
"loss":"binary_crossentropy"
}

We have not provided a direct testing script in this repo so that you can directly run the model on a completely new dataset. Indeed, it would be more useful for bioinformaticians if a direct testing script is provided. However, implementation is straightforward. You can simply load the trained model and the transformed one-hot matrix and conduct the testing.

Pseudo code can be as simple as follows:
$ load your data in one-hot matrix
model = SeqModel(epochs=args["epochs"], args=args, loss=loss)
predictions = model.predict_mc( $your data, n_preds=4)

Ok,  thank you very much. I'll try. If you have any questions, please come back

I still don't know how to predict recombination hotspot by this proggram with my data


Delay forcing catchup till end of block time

To help prevent the blockchain flat lining a check was added to
nominate function to resend latest envelopes and kick off catchup task
after 10 msecs. This is better left till the time has been reached for
the next block start rather than the current. Otherwise catchup will
keep the node busy whilst others are nominating.

Re-running as I'm waiting for it. :)


I can't remap a key to "underscore"

I use a TypeMatrix keyboard with a french «bépo» mapping. On this keyboard the underscore "_" is Alt+Space.
On some environments (a VM under Parallels for example) it doesn't work. Therefore I'd like to remap an unused key to "_".
I can't find a way to use simple mapping to do this as "underscore" does appear in the drop-down list. How can I do this? Is this a feature not implemented yet?

I cannot find how to that either. In my case, I just want Right Command to be mapped to Underscore. I tried to map Right Command to Shift and to Minus at the same time (as Shift+Minus = Underscore), but that didn't work.

I am trying to do something similar.  I want to map caps lock directly to underscore.  Since an underscore is really shift+hyphen, a complex modification is required.  I can't figure out any way to add my own complex modification in the app, or submit a file to the website...
But I was able to create a complex modification in ~/.config/karabiner/assets/complex_modifications/underscore_mapping.json to do my mapping.  I've included the file, and @kurko I also included your right command mapping.
Download, unzip the file, and move to ~/.config/karabiner/assets/complex_modifications/.  Open the app, click on the Complex Modifications tab, click on Add rule, select the rule in the list (it should be there now) and click Enable.
underscore_mapping.json.zip

Apple arabic input source has a bug. It types kashida (ـ) instead of underscore (_) when pressing on shift+hyphen keys!
I'm trying to solve this bug using KE but without any success. I hope KE can add "underscore" key code to the codes list. Alternatively, it may allow entering text or unicode code in the "to" modifier.


Odd PPUs evident at all-England level

"Backwards GBG"
A few products (e.g. Isosorbide mononitrate) show generics consistently at lower prices. Diltiazem, of course, should only be prescribed by brand

Two MRs for one branded generic
For Metformin, Actavis has a variable price depending on pack size and Metabet has a consistent price:

We have emailed them about it
GBG at odd levels
This might just be a standard GBG issue, but it is striking that there are roughly equal amounts of PPU for Mesalazine at every price point between the cheapest and most expensive brands.

Diltiazem: A small pedant point but the BNF point around brand prescribing only pertains to >60mg. There are a lot of diltiazem generics. However the dmd does state Caution - AMP level prescribing advised at the VMP level (see here)
Mesalazine: I think we have to assume it is GBG but the price spread is odd. As regards quantity, the circles don't look proportionate to me.

Metabet: The Actavis product has apparently not been available since 2014. So the deeper puzzle there is what is actually being dispensed when the Actavis product is prescribed? And then, why does that have such variable pricing?

Metabet: I would assume the available Morningside preparation is being dispensed. The majority of people will not even realise the technical variation in a different MA holder. Also with only one available on their shelf, it is just the obvious to dispense.
Regarding the pricing I have had a response from NHS BSA to some confirmatory questions (in italics) I asked.
I have a query about  Metabet SR_Tab 1000mg (Actavis) . The most recent prescribing is attached and I was wondering what explains the variation in the net ingredient cost? Only two prescriptions (highlighted in green) correspond to price in dm+d. I have highlighted other groups of clustering on a NIC to try and spot a pattern.

No price is held on dm+d for the listing of Metabet (Actavis) discontinued product, therefore, the basis of reimbursement will be made as per the endorsed invoice price. This should explain the variations in the NIC.

Note on dm+d there is a price listed but this is the last price. There is another field Price basis which states there is "no price available"
Separately, the dm+d indicates that this product has been unavailable since 2014. Is there any point at which the BSA will not reimburse unavailable products?

The BSA will reimburse unavailable products provided sufficient information is supplied by the contractor.

Despite appearing illogical there may be occasions where an "unavailable" product is available for example a product is discontinued but pharmacy actually has a couple of months on their shelf. Allowing this in a blanket fashion though does not make sense for products discontinued 5 years ago, we will investigate further.
On a related note there is another identical product Metabet SR_Tab 1000mg (Morningside) with a different MA and BNF code. It does appear to be available and I’m assuming this is the physical product a patient gets.

You are correct there is a current listing of Metabet SR_Tab 1000mg (Morningside) on dm+d. However, should the contractor be dispensing the Morningside product then this should be made clear on the prescription submitted for reimbursement.

This gets to price variation. So we assume Morningside product is being dispensed, that means we would expect the Morningside dm+d price to be paid. Why might this be happening? My theory is the pharmacy contractor is endorsing the price on the invoice. This may or may not relate to dm+d, for example

they may have received a discount and endorsed the net cost less discount
they may have mixed up price of product unintentionally
the price may have been illegible and keyed in sightly wrong at BSA

Let me get this right: a pharmacy gets a prescription for Actavis; dispenses Morningside (and endorses with the brand and the price actually paid); and records an apparently random price, rather than the tariff price for Morningside (often quite a lot less than the that price?)
It is interesting that the Actavis one had all prescribing stopped when it stopped being available, and then went up again a year ago:

Regardless of what is getting dispensed, and how it's being priced, this looks like another SystmOne bug!

notebook here

Regarding Metabet, it's also interesting to note that the packaging for Sukkarto (the cheapest presentation) is identical to the Morningside Generic

https://twitter.com/blmerriman/status/1167180801202642960

On this, SQL to find all VMPs with an AMP from more than one supplier:
SELECT
  descr,
  dmd2_amp.vpid,
  bnf_code
FROM
  dmd2_amp
INNER JOIN (
  SELECT
    vpid,
    suppcd,
    COUNT(*) AS num
  FROM
    dmd2_amp
  WHERE
    flavourcd IS NULL
    AND vpid !=<PHONE_NUMBER>001105
    AND bnf_code LIKE '0%'
    AND bnf_code NOT LIKE '09%'AND nm NOT LIKE '%caplets%'
    AND avail_restrictcd = 1
  GROUP BY
    vpid,
    suppcd
  HAVING
    COUNT(*) > 1) dupes
ON
  dupes.vpid = dmd2_amp.vpid
  AND dupes.suppcd = dmd2_amp.suppcd
ORDER BY
  dmd2_amp.suppcd,
  dmd2_amp.vpid,
  bnf_code;

Output here


Byte Buddy version update

Continuing the investigation.
Caused by: java.lang.NegativeArraySizeException: -1
	at tech.httptoolkit.relocated.net.bytebuddy.utility.dispatcher.JavaDispatcher$DynamicClassLoader.invoker(SourceFile:34)
	at tech.httptoolkit.relocated.net.bytebuddy.utility.dispatcher.JavaDispatcher$Invoker$CreationAction.run(SourceFile:1)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at tech.httptoolkit.relocated.net.bytebuddy.utility.dispatcher.JavaDispatcher.<clinit>(SourceFile:12)

This exception starts from version 1.11.6. By looking at the generated source mapping I see that the try to create array with negative size is this line:
https://github.com/raphw/byte-buddy/blob/c51396ba504c0a7397d912f8e621d060bc360483/byte-buddy-dep/src/main/java/net/bytebuddy/utility/dispatcher/JavaDispatcher.java#L1270
Which fails as in the byte code Invoker interface was replaced with concrete class (in this commit it's Unsafe which is a class and has hashCode which has zero arguments... so my guess it's that it fails on this.
-- fast forward to latest byte buddy
Return value of JavaDispatcher.invoker() is net.bytebuddy.utility.Invoker interface but in the byte code (after r8) it is substituted with DirectInvoker  class which again has one method with 0 arguments.
The exception is thrown at this line:
https://github.com/raphw/byte-buddy/blob/master/byte-buddy-dep/src/main/java/net/bytebuddy/utility/dispatcher/JavaDispatcher.java#L1287
This can be fixed by leaving the Invoker interface untouched by adding
-keep class tech.httptoolkit.relocated.net.bytebuddy.utility.Invoker { *; }

to r8-rules.txt. This way the interface is not replaced and byte buddy is able to create DynamicClassloader.
Afterwards the agent is able to attach to processes like  before and we reach the second (or actually it is the first issue).
Since version 1.11.3 we are greeted by multiple SSLHandshakeExceptions during tests:
Unexpected failure for new http-url-conn: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

I currently have no idea what caused this... yet...

Looking at the standalone test the exception is also there (but it manages to succeed?)
https://github.com/httptoolkit/jvm-http-proxy-agent/runs/4601280285?check_suite_focus=true

Thanks for continuing the investigation @kamko, this is really interesting! I haven't found time to properly look into the details yet myself, but I should have some time next week to dig into this and see if I can work out what's up with those SSL handshake exceptions... Watch this space. Do let me know if you manage to make any more progress yourself.


FIX Correct invalid reference to register endpoint

I updated the tests that had timing issues to try to get rid of that issue.


Adapt smoke test expectations.

Codecov Report
Base: 92.86% // Head: 92.86% // No change to project coverage :thumbsup:

Coverage data is based on head (b5046a7) compared to base (0b03c75).
Patch has no changes to coverable lines.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #1611   +/-   ##
=======================================
  Coverage   92.86%   92.86%
=======================================
  Files         348      348
  Lines       24693    24693
=======================================
  Hits        22931    22931
  Misses       1762     1762

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.