[FEATURES] add additional information for a package

Hi guys. I've got requests for some new additions to the npm-search index. Let's discuss them here:

[ ] open issues on gihtub: How many open issues are there.
[ ] number of gihtub stars: How many people stared this projet.
[ ] number of gihtub forks: How many people stared this projet.
[ ] open PR on gihtub: How many open PRs are there.
[ ] last commit on gihtub: When was the last commit on master.
[ ] add any other usefull information (here are all the properties returned by Github's API)

Does this make sense?

It would make sense to also query GitHub for those for which we have the data. In the Yarn website we do this frontend once the detail page has been requested. For requesting GitHub data we'll need to have some API key rotation like npms does: source.
It would also make the replication slower, but that would be fine IMO, it's just the API usage limitations that I wasn't able to overcome for now

One thing that I would also love to add is jsDelivr hits in the last month. Right now the search results are sorted by npm downloads, which is great for nodejs/backend packages, but doesn't work that well for browser/frontend packages, especially those, which recommend CDN as the primary installation option.
It would be nice if we could either combine those numbers somehow, or simply had an option for sorting by cdn hits rather than npm downloads. Implementation should be as easy as using either this or this endpoint of our API.

Is it possible to get the monthly downloads batched, like this but with a specified 100/200 packages to look this up from at the same time, so we can look them all up?

There's ?page, so you can just do ?page=1, ?page=2, ... until you get no results.

I won't have time for now to add this, but feel free to contribute or contact me if I can help. It indeed could be possible to augment the index like that, although the flow we currently have is

get all packages in batches
loop over a batch of packages to get more info or get more info from the whole batch

So we'd need to control which packages would be in the batch (maybe something that can be added to your API first?

Those are valid concerns, adding the jsDelivr downloads seems "easy". As for the GitHub api requests that's a bit more tricky because of what @Haroenv said. Then you also have the freshness issue, currently we do not rebuild the full index periodically while if we provide things like number of opened issues, you might want the data to be as fresh as possible.
Still, if you already have ideas on how to do it well, please do contribute, make it faster, anything :)

Since the data is rather compact (just package name and one number), I think we could get everything at once and have it stored in memory during the indexing process. Our API currently computes stats for all packages every time (which is why it is that slow), even though it only gives you 100 results at once, so it would make more sense if we removed that limit and you'd be able to get all numbers in one request (we're talking about 2.5 MB of data per 100k packages and currently we have ~20k packages).

Seems possible, feel free to do a PR @MartinKolarik :)

@vvo @Haroenv as per the opened PRs and issues, I think it makes more sense to do it client side, indeed. I'll figure out a way to do it on https://ngx.tools.
When it comes to augmenting the npm-search index with additional information from github, I'll throw some "random" ideas here (I might be wrong in some aspects):

use authenticated REST calls in order to increase the rating limit from 60 to 5k req/hour (as Haroen mentioned).
partition the dataset (NPM packages) and query periodically github for a subset of packages.
we could query github info for popular packages first.
store the result in an intermediate database
during the (re)indexation process, use that intermediate (cache) database to read and augment the index whith github's info, without slowing the indexation process.
NOTE: depending on how the indexation process is done, we can skip the intermediate database and write directly the github info to the index database.

NPM has ≈ 700k packages. So, 700k / 5k = 140 hours ≈ 6 days. It'd take 6 days to process the 700k packages using one gitub API key, with 5k packages per hour. We could enhance this by using 6 github API keys and do it in 1 day. Right? We can even throw these calls inside Cloud Function and don't pay for the infrastruction (lots of Cloud Providers have free tier—GCP offers 2 million free call / month).
Alternatively, we could use GraphQL to query github, since one GraphQL call can replace multiple REST calls. A single complex GraphQL call could be the equivalent of thousands of REST requests.
I'm sure we'll figure it out ^_^

Thanks for the nice architecture thoughts!

Alternatively, we could use GraphQL to query github, since one GraphQL call can replace multiple REST calls. A single complex GraphQL call could be the equivalent of thousands of REST requests.

I did try out the GitHub GraphQL API, a bit strange at the beginning (I had to understand the actual GraphQL language), but afterwards feels super nice.
Example: https://github.com/vvo/zorgs/blob/master/src/zorgs/src/queries/repositoriesWithCommits.js

Very nice. In the same fashion, we could get the number of stars and issues like so:
{
  search(type: REPOSITORY, query: "user:angular", first: 3) {
    edges {
      node {
        ... on Repository {
          name
          stargazers {
            totalCount
          }
          forkCount
          issues {
            totalCount
          }
        }
      }
    }
  }
}

Which would give us:
{
  "data": {
    "search": {
      "edges": [
        {
          "node": {
            "name": "angular.js",
            "stargazers": {
              "totalCount": 58360
            },
            "forkCount": 28909,
            "issues": {
              "totalCount": 8785
            }
          }
        },
        {
          "node": {
            "name": "angular",
            "stargazers": {
              "totalCount": 35458
            },
            "forkCount": 8634,
            "issues": {
              "totalCount": 14507
            }
          }
        },
        {
          "node": {
            "name": "angular-cli",
            "stargazers": {
              "totalCount": 17163
            },
            "forkCount": 3753,
            "issues": {
              "totalCount": 7696
            }
          }
        }
      ]
    }
  }
}

Seems useful! Would be nice to try out if we can get this merged, can be added in github.js or another file. Ideally we would be able to get the issues, forks and stargazers of multiple (100/200) repositories at the same time


Do not include drafts in user post count. fixes #488

Notes for testing:

Creating a new post should increment the post count by 1
Creating a new draft should not increment the post count
Publishing a draft should increment the post count by 1
Converting a published post to a draft should decrease the post count by 1

We can either overwrite the method I modified, or add an additional
callback which decrements by 1 when a new draft post is made
On Fri, Apr 27, 2018, 4:20 PM Raemon<EMAIL_ADDRESS>wrote:

Our guidelines for editing example-forum are a little vague. Approximate
instructions are:

See if you can fix the problem within the lesswrong package. (Often
this involves overwriting whatever function was getting called in
example-forum)
2.
If fixing it in lesswrong is impractical, you can do it in
example-forum, but make sure to leave a comment immediately above your
changes that includes the word "LESSWRONG" in capital letters, so that we
can easily search example-forum for changes we've made to it.
I will take a look at this and see if option 1 or 2 seems better.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Discordius/Lesswrong2/pull/553#issuecomment-385083587,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA0sE-G6sgvwz0yaTdMkqPpbNJkHLeeVks5ts32ogaJpZM4Tq6aC
.

Looks like this is roughly correct (will give a more thorough review later today after dealing with some other things).
Specifically, good call on doing the minor tweak to the function in example-forum (although I'd add in a comment like "// LESSWRONG – fixing post count bug" or something), and then doing the rest in the lesswrong package.
I haven't yet doublechecked that it all works but looks good at first glance.
Upon reflection we're not really using the frontpage post count, although it might be good to go ahead and fix that while you have all the context loaded in your head. Up to you.

I've updated the pull request to include a // LESSWRONG comment. Let me know if there's anything else that needs to be done.
What are our guidelines around when to add tests?


When the value is changed the component does not render properly

const [starRating, setRating] = useState(4);
<ReactStars
onChange={(val) => {
setRating(val);
}}
value={starRating}
classNames='space-x-1'
isEdit={true}
activeColors={["red", "orange", "#FFCE00", "#FFCE00", "#FFE70F"]}
size={18}
/>
<button onClick={() => {setRating(5)}}>change rating to 0
The value is changed but the stars continue to show 4.
Am i doing something wrong?

you can't change value from outside like this <button onClick={() => {setRating(5)}}>.
you have to change value by click on star

fixed. now enjoy!!


FINERACT-2055: Internal Server Error Encountered When Accessing Individual Collection Sheet Option

FINERACT-2055
Issue: Internal Server Error Encountered When Accessing Individual Collection Sheet Option
This issue is caused by the usage of the 'of' keyword as an alias for the 'm_office' table. When capturing rows from the 'm_office' table with the alias 'of', it triggers an error.
Description
Describe the changes made and why they were made.
Ignore if these details are present on the associated Apache Fineract JIRA ticket.
Checklist
Please make sure these boxes are checked before submitting your pull request - thanks!

[x] Write the commit message as per https://github.com/apache/fineract/#pull-requests

[x] Acknowledge that we will not review PRs that are not passing the build ("green") - it is your responsibility to get a proposed PR to pass the build, not primarily the project's maintainers.

[x] Create/update unit or integration tests for verifying the changes made.

[x] Follow coding conventions at https://cwiki.apache.org/confluence/display/FINERACT/Coding+Conventions.

[x] Add required Swagger annotation and update API documentation at fineract-provider/src/main/resources/static/legacy-docs/apiLive.htm with details of any API changes

[x] Submission is not a "code dump".  (Large changes can be made "in repository" via a branch.  Ask on the developer mailing list for guidance, if required.)

FYI our guidelines for code reviews are at https://cwiki.apache.org/confluence/display/FINERACT/Code+Review+Guide.

@Mariappan-Sivakumar-ZS0528 Please help me understand why renaming the alias would sort this issue?

Hi @adamsaghy
While using 'of' as an alias for the 'm_office' table in MySQL, it's not recognized as an alias but rather as a reserved keyword in SQL. Therefore, employing 'of' in that context triggers an error. By renaming the alias to 'off', we resolve this issue as 'off' is a valid alias and doesn't conflict with SQL keywords.


Fix TalkApiClient random test failure

Normally you'd scratch your head about why things randomly fail. In this case, I'm actually not sure why it sometimes passes...
This will fix it in any case.

LGTM


Proposal: add context to the standard library

@Sajmani and I are interested in adding golang.org/x/net/context to the standard library, probably as the top-level package context.
(Why not net/context? It has little to do with networking, except that people often use it with networking; it doesn't even cross network boundaries on its own without help.)
Some places in the standard library which could use it:

https://golang.org/pkg/net/http/#Request -- add a new Context field, alongside like the existing (Context.Done-compatible) Request.Cancel field. It will take precedence over Cancel, and we can make it work for both outbound and inbound requests. For server requests it would use CloseNotifier's mechanism for c* ancelation. (which for http2 includes the http2-level explicit stream cancelation frame too, also used by gRPC)

https://golang.org/pkg/net/#Dialer -- same as http.Request. net.Dialer also has a Done-compatible Cancel field, which could be a Context.

https://golang.org/pkg/os/exec/#Cmd could have a new Context field to automatically kill+wait the process when a context expires, if the context expires before the child?

https://golang.org/pkg/database/sql/#DB.Query already takes ...interface{} which can be special-cased to say that if the first parameter is a context.Context, it's used for deadlines. Trashy or clever? Maybe both. We could also add parallel DB.QueryContext which is a bit of a mouthful.

.... (where else?) ....

Open questions:

do we do something with the io package in the same release, or later? i.e. do we add *os.File.IO(context.Context) io.ReadWriteCloser sort of methods, to curry a context into io interfaces? Or wait on that. If it'd be nice to push down cancelation into even OS-level file I/O, but we don't even really do that inside Google with IO-like methods and pervasive Context usage, since we don't really use *os.File. So I'm inclined to wait on that. Too many operating systems to deal with.

Concerns:

we can't add it to as many places in the standard library we'd like to since APIs are frozen. Basically we can only add it to structs. While we've told people not to add contexts to structs, I think that guidance is over-aggressive. The real advice is not to store contexts. They should be passed along like parameters. But if the struct is essentially just a parameter, it's okay. I think this concern can be addressed with package-level documentation and examples.

what field name in structs to use? We use "ctx" for variables, but struct names are typically spelled out. But does that cause too much stutter? e.g.

package http
type Request struct {
    Context context.Context
}

For others following, there was a substantial discussion on golang-dev about this as it related to net/http: https://groups.google.com/forum/#!topic/golang-dev/cQs1z9LrJDU
As for naming: the only stutter is in the field name in the source. Package users are going to call r.Context.WithValue or r.Context.Done, which has the benefit of being explicit and (say you point out, Brad) retains the ctx` naming convention.
Looking forward to this happening!

Thinking about http.Handler, people can alternatively rewrite a different kind of handlers that have a different signature for ServeHttp. The advantage is that one can easily add a Context parameter.
But also, add return values. For instance a boolean sentinel value to control whether the handler should be the last one to be executed.
The top level mux object that encapsulates the whole request dispatching logic can still be a regular http.Handler
There are also a few things that I am not very fond of. context.TODO is one of them. But I don't know if it is too late to change things or not.
Is this the right place to discuss about API design ?

I think naming it Context in structs is good. There's a stutter, but only for the author of the API. Couldn't be clearer for the users of the API.
Would it be possible to write a go vet check for obvious misuse of a Context in a struct? If so, it would be great to roll that out at the same time. Posting a sign next to the pool is good, but a lifeguard is better.
As much as I'd love Context to be deeply baked into the standard library, I'd be happy with it simply being added to the standard library to say "this is blessed, find good ways to use it" and later adding the best ideas to the stdlib where they can be and kept in third party packages where they cannot. However, I'm not opposed to any of the suggested integrations.
Having the first interface{} parameter to Query be a Context is clever, though, so I'd much rather have the explicit QueryContext.

This is maybe a bit out of place for this particular issue, but I think it would be worth considering a different Context implementation (strawman benchmark) before it gets included in the stdlib.
In that benchmark, I compare a bogus map+lock "caching" context implementation to the default implementation. Even with as few as 10 items, there's a noticeable improvement (%-wise, not in absolute terms. I realize we're talking nanoseconds here) on lookups into the cached version. For contexts deeper than 10 items, there's a noticeable improvement. The construction overhead is similar, but I think with a non-bogus implementation the map version could be even faster for overhead than the linked list one.
I think there's an opportunity to have Context be a *struct-with-hidden-fields instead of an interface, which would afford the opportunity to build a faster implementation of it (e.g. by having .WithValue be a struct method instead of a package method that can be smart about chaining, or a .WithValues to avoid the linked-list-building overhead when adding multiple items to the context at the same time) and by possibly adding a .Compact method to compactify an existing Context to make subsequent lookups fast. It's not possible to build these things in a clean fashion with the current interface-based implementation (since the only way for the context to see up the chain is to actually do lookups, and the only way to know the lookup keys is to wait for some user code to provide them).
I'm mostly thinking about the pattern of:
func MyHTTPHandler(rw ResponseWriter, r *Request) {
  c := context.Background()
  // WithValue 10-30 services/contextful things here... this number will likely grow
  // as context becomes even more prevalently used.

  // 100's of lookups into c here... if it were compacted here then all the lookups
  // would be O(1).
  myActualHandler(c, rw, r)
}

Which, I think, is a pretty common pattern (I don't have data on hand for this assumption, however).
In general, for most stdlib-type libraries, there would simply be competition for the best implementation. But I think that the community is rallying around around a single Context implementation (because it's THE way to build composable/stackable libraries which are loosely coupled). There can't really be an alternate implementation that doesn't stem from the stdlib (or x/) that has any hope for survival (since everything needs to adopt the stdlib one to actually be used by anything). To be clear: I think it's GOOD for the community to standardize here, I just think it might be worth considering a slightly speedier implementation before cooking it into the 1.x API compatibility guarantee.
Note that the WithTimeout and WithDeadline package functions are well-served by the interface implementation right now: it allows e.g. mocking of the clock. Some similarly mockable interface would need to be built for that functionality, but AFAIK that mocking requirement doesn't really apply to WithValue.

How do you expect putting context in http.Request (or any other parameter-ish struct) will interact with adding values (or timeouts or whatever) to the context?
I can think of two basic approaches, caller-save and callee-save:
func f(w http.ResponseWriter, req *http.Request) {
	ctx := req.Context
	g(w, req)
	// ... something with ctx ...
}
func g(w http.ResponseWriter, req *http.Request) {
	req.Context = context.WithValue(req.Context, foo, bar)
	h(w, req)
}

or
func f(w http.ResponseWriter, req *http.Request) {
	g(w, req)
	// ... something with req.Context ...
}
func g(w http.ResponseWriter, req *http.Request) {
	req2 := new(http.Request)
	*req2 = *req
	req2.Context = context.WithValue(req.Context, foo, bar)
	h(w, req2)
}

Of course, it's fine if both caller and callee save the old value, but it's bad if neither of them does it, each expecting the other to do so. So, should we recommend one and document and promote it up front?

Traditionally, context objects should carry configuration. The x/net/context package extends context objects' responsibilities by providing constructs for timeout and cancellation signals. In the scope of stamp coupling of request handlers, it is a valuable addition. But I am not sure how it will scale to the rest of the standard library and furthermore whether the context may end up being abused for signaling purposes. The two cases you mention above (exec.Cmd and sql.DB.Query) does this by solely using a context object for handling signals even though there is no necessity for storage. Are you trying to solve the issue of how data is passed among functions or also trying to have primitives around how the entire execution could be signaled from outside?
One of the concerns I would add to the list is that the context should be scoped to a function call.
"Do not store Contexts inside a struct type; instead, pass a Context explicitly to each function that needs it. The Context should be the first parameter, typically named ctx" (link)
If the standard library won't be agreeing with this due to compatibility reasons, don't you think it will create confusion?

@rakyll, I addressed that final concern in my first post. ("While we've told people...")

If there's any way to add context to the standard library without making http or net aware of context, this would be the way to not overwhelm newcomers.

The plan @bradfitz is proposing makes http and net aware of context, but not in a way that requires their use for basic cases. The http package is already very large, but you only need to use a very small fraction of it to achieve "hello world"-esque examples. Adding context to these packages won't harm newcomers, and should make life easier in more complicated/large codebases.

嗨各位
在 https://golang.org/pkg/net/http/#Request 增加像 golang.org/x/net/context 这样的 Context 字段仍旧给开发者造成很多约束啊.
那么增加 context 的目的是什么呢? 是要推荐一种具体的用法还是要解决绑定自由控制对象?
我认为开发者更需要自由控制绑定对象. 还不如直接增加
type Request struct{
//...
Context interface{}
}
来的简单通用.

I mailed out https://golang.org/cl/20346 &https://golang.org/cl/20347
The first copies x/net/context to the standard library. The second reworks x/net/context to just use the standard library's version.

@kr, making a new request (req2 := new(Request) ...) is probably the safer option. You don't know whether one of your callees is still using the original Request.

@riannucci, I think redesigning Context is too big and out of scope. We have tons of code already using the existing Context and we want to make migration easy.

@bradfitz yeah, I kinda figured as much, but I was hoping that since it's only in x now and not in the stdlib, we still had some API flexibility (e.g. introduce new one to stdlib, deprecate x version, remove x version in 1.8-era or something like that)

I would love to see support for Context in text/template, such that:
ctx := context.WithValue(context.TODO, "Title", "Through the Looking Glass")
template.Must(template.Parse(`Title: {{.Title}}`)).Execute(os.Stdout, ctx)

would output something like:
Title: Through the Looking Glass

If we are worried about accidentally leaking stuff from the Context into a template (because you used a string as a key elsewhere), then I propose adding a special type to distinguish values you want exposed:
// in the text/template package
type Expose string

// in your program
ctx := context.WithValue(context.TODO, template.Expose("Title"), "Through the Looking Glass")
template.Must(template.Parse(`Title: {{.Title}}`)).Execute(os.Stdout, ctx)

This way, you are deliberate about exposing values to the template.

@chowey: what problem you are trying to solve? If keys have to be dynamic, why not to use a map?

@bradfitz @riannucci yes, that's my sentiment too. I understand though that x/context is widely used but since it's about to be frozen into the stdlib, I'd rather have the simplest interface possible. Not only for the current users but the ones that have yet to come. I will still understand if it's difficult from the POV of logistics but...

What do you find 'not simple' about the current net/context API? The most
basic use-case revolves around WithValue / Value, which are (I think!)
fairly straightforward.
As Brad says though, it's probably a little too late to change anything:
although it's an "x" package, it's been in the wild for a long while.
On Mon, Mar 14, 2016 at 6:36 AM atdiar<EMAIL_ADDRESS>wrote:

@bradfitz https://github.com/bradfitz @riannucci
https://github.com/riannucci yes, that's my sentiment too. I understand
though that x/context is widely used but since it's about to be frozen into
the stdlib, I'd rather have the simplest interface possible. Not only for
the current users but the ones that have yet to come. I will still
understand if it's difficult from the POV of logistics but...
—
Reply to this email directly or view it on GitHub
https://github.com/golang/go/issues/14660#issuecomment-196311945.

@elithrar well, the most interesting use is the propagation/monitoring of cancellation signals, not really WithValue (even though it helps for handler chaining).
I don't find WithValue very ergonomic compared to passing a mutex protected map.
If we have to have a canonical Context, we don't really need it to be an interface either. A struct with hidden fields suffices.
Also, WithTimeout and WithDeadline are somewhat doing the same. You could remove one of those. That would make things already a bit simpler.
I understand what Brad said. Still I'd rather the decision to be clearly examined for the long term. It was in x/context and not the stdlib for a reason.

I share @atdiar 's concerns.  While I understand how golang.org/x/net/context works, it took me multiple reads through the godocs and blog post for it to click.  I find the current API kind of confusing.
I threw together an alternative implementation (albeit in a rush) with what I believe is a simpler API and easier learning curve: https://github.com/bobziuchkovski/context, godocs at https://godoc.org/github.com/bobziuchkovski/context
I realize it's a bit late in the game, but I feel it's at least worth discussing the current x/net/context API before it hits stdlib.

@bobziuchkovski For posterity, can you update your comment (or add a new one) to clarify the pros/cons of your lib vs. the existing x/net/context? I can see that Terminated() and TimeRemaining() are new interface methods (with clear uses), but what are the downsides here?

Sure.  Terminated() and TimeRemaining() are mostly synonymous with the existing Done() and Deadline().  In terms of the changes there, I think Terminated() better conveys that the context was actually canceled.  I believe a x/net/context could finish it's work and be "done" but the Done() channel wouldn't close unless an expiration or CancelFunc were fired.
For TimeRemaining() vs Deadline(), there's more value (for me anyway) in knowing how much time is left vs an expiration date.  I can't personally think of a case where I'd want the Deadline() date vs simply knowing how much time (if any) is left.  This isn't a major concern, but just something to discuss.
Adding the WithValue and WithTimeout methods to the interface, like I've done, definitely makes it "fatter", so that could be a downside.  I believe it makes the parent/child relationships clearer and makes the context cleaner to construct/work with than all of the different x/net/context.With* functions, but that might just be me.
I think my biggest concern with the current x/net/context is the CancelFunc stuff and the Background() and TODO() funcs.  I "get it" now, but those were the biggest barrier to my understanding of the package.  It's far easier for me to comprehend Context.Cancel() and implicit cancelation via timeout with the implementation I provided than all of the CancelFunc return values from x/net/context.  Similarly, it's easier for me to comprehend creating new contexts via New and Context.WithTimeout() and Context.WithValue than using TODO(), Background() and the different WithValue(), WithDeadline(), WithTimeout(), and WithCancel() constructor funcs from x/net/context.  On the one hand, I get that using those keeps the interface slimmer, but on the other hand, it makes x/net/context far more confusing (to me).
I'm not necessarily putting forth this alternative implementation as the alternative, but rather an alternative to hopefully spark some discussion around the current x/net/context API.

Thanks Bob.
I think that—given the recurring theme of 'net/context not being clear
about usage'—some of this could be addressed through documentation, if API
changes aren't palatable. It's not the first time I've seen confusion about
how it works/how to use it across the mailing list/SO/forums/etc.
(to address another comment: net/context exists to replace mutex-protected
maps)
On Mon, Mar 14, 2016 at 9:48 AM Bob Ziuchkovski<EMAIL_ADDRESS>wrote:

Sure. Terminated() and TimeRemaining() are mostly synonymous with the
existing Done() and Deadline(). In terms of the changes there, I think
Terminated() better conveys that the context was actually canceled. I
believe a x/net/context could finish it's work and be "done" but the
Done() channel wouldn't close unless an expiration or CancelFunc were
fired.
For TimeRemaining() vs Deadline(), there's more value (for me anyway) in
knowing how much time is left vs an expiration date. I can't personally
think of a case where I'd want the Deadline() date vs simply knowing how
much time (if any) is left. This isn't a major concern, but just something
to discuss.
Adding the WithValue and WithTimeout methods to the interface, like I've
done, definitely makes it "fatter", so that could be a downside. I believe
it makes the parent/child relationships clearer and makes the context
cleaner to construct/work with than all of the different
x/net/context.With* functions, but that might just be me.
I think my biggest concern with the current x/net/context is the
CancelFunc stuff and the Background() and TODO() funcs. I "get it" now, but
those were the biggest barrier to my understanding of the package. It's far
easier for me to comprehend Context.Cancel() and implicit cancelation via
timeout with the implementation I provided than all of the CancelFunc
return values from x/net/context. Similarly, it's easier for me to
comprehend creating new contexts via New and Context.WithTimeout() and
Context.WithValue than using TODO(), Background() and the different
WithValue(), WithDeadline(), WithTimeout(), and WithCancel() constructor
funcs from x/net/context. On the one hand, I get that using those keeps the
interface slimmer, but on the other hand, it makes x/net/context far more
confusing (to me).
I'm not necessarily putting forth this alternative implementation as the
alternative, but rather an alternative to hopefully spark some
discussion around the current x/net/context API.
—
Reply to this email directly or view it on GitHub
https://github.com/golang/go/issues/14660#issuecomment-196405772.

@elithrar I definitely agree there.  Even without API changes, a lot of my concerns could definitely be addressed via documentation.

@bobziuchkovski Cancel being a method on context lets functions the context is passed to cancel the context of their caller, which sounds like a bad thing. In fact, doing the right thing requires explicit effort with that API. WithCancel returning the cancel function separately means you get to (have to) choose who has that power.

@tv42 Yeah, that occurred to me, but if that's the reason for the current API, that seems like a lot of engineering effort to solve a problem where "don't do that" might work.  :smile:   More specifically, "don't cancel a context you didn't create".  I don't know.
I can understand the argument that it's hard to change the API after it's been in the wild for some time.  But I can also honestly say that it took me genuine head scratching and re-reading of the godocs and blog post multiple times before it clicked.
It might be as Matt suggested and just require some additional documentation.  Or it might be something where some API tweaks would honestly help.  I feel a bit bad for being a nay-sayer, but that's also why I tried to contribute some example code as a means of providing constructive feedback.

https://godoc.org/golang.org/x/net/context hints at the existence of a static analysis tool, yet I see no mention of one here or in linked discussions or CLs.
Is a static analysis tool considered out of scope for the initial stdlib inclusion of context? That seems reasonable but given concerns over type safety and the doc references, I kind of expected a tool would at least get mentioned in the proposal.

My main concern here is presenting clear advice about structs. The standard library should demonstrate the idiomatic best practices for using context, or else no one else will use context idiomatically either. That said, I do agree with Brad that - if done clearly - we can relax the struct prohibition a bit.
Brad wrote:

While we've told people not to add contexts to structs, I think that guidance is over-aggressive. The real advice is not to store contexts. They should be passed along like parameters. But if the struct is essentially just a parameter, it's okay. I think this concern can be addressed with package-level documentation and examples.

I think this is correct. net.Dialer and exec.Cmd are clearly "parameter structs" in that from the start they contained things that only made sense for a single operation (Deadline for net.Dialer, results like Process for exec.Cmd).
For sql.DB, overloading the args ...interface{} list seems pretty hacky, but the alternative is a whole new parallel API, which is almost as bad. If we're going to do that, though, it seems like maybe it should be the last arg instead of the first. At least then any references to positional arguments continue to use the same indexes, and it avoids the context being a middle argument in calls to functions that take 'query string, args ...interface{}'.
Let's leave os.File out of it for now. Most operating systems don't support timeouts on file reads anyway, and os.File.IO seems very clearly storage of a context.
A field 'Context context.Context' is fine. I do wonder if maybe Context should be in package io. I've suggested this before but I don't remember if there was a reason it can't be done. It's not about reader/writer io but it's certainly almost always some kind of input or output operation, and io.Context is not so stuttery.
The biggest problem is going to be http.Request. Once you know that Contexts must only be passed in parameter-like structs, that implies http.Request must be a parameter-like struct and so the answer to @kr's excellent question has to be that you make a shallow copy of the entire request to pass along a different context, as Brad said. I think this will surprise people. It's actually already the case that http.Request is parameter-like and that programs should only be modifying copies of the http.Request, not the original, but Context will make that even more important, and it will be a surprise. It will need very clear documentation.
But it seems OK to go forward with. I had been hoping to ask @Sajmani what his thoughts were in person, but I have not gotten a chance yet. So I will ask here: Sameer, how does this all look to you?
Thanks.
P.S. I wonder if we should employ the race detector here as well. When run under the race detector, the http server could kick off two goroutines for each request instead of one. Specifically, serverHandler.ServeHTTP could replace:
    handler.ServeHTTP(rw, req)

with
    var c chan http.Request
    if raceEnabled {
            c = make(chan http.Request)
            go func() {
                    c <- *req
            }()
    }
    handler.ServeHTTP(rw, req)
    if raceEnabled {
            <-c
    }

It's a little heavy but so is the race detector generally. The coordination on the channel makes sure that when serverHandler.ServeHTTP returns there are no pending uses of the request, but the copying of the request into the channel will conflict with any write to a field of req during handler.ServeHTTP, which should report races in handles that overwrite the original req's Context field (or others). In practice some fields are OK to write (for example fields lazily initialized, like FormValues), so this exact code won't work but the general idea of reading from unmodifiable fields in a separate goroutine under the race detector seems like it should be able to serve as a mechanical check that people are doing the right thing. Of course any common middleware layers that already make their own copies of an http.Request and then pass that further down the chain would need to add this pattern too. I don't know of any instances of that in the standard library.

One area seems haven't been discussed is the "detach" functionality of a context. We've found legitimate use cases (at least inside Google) that sometimes detach a context to pass into an async routine within the ServeHTTP path seems useful (if not the only correct way of handling it). For example, some server logging routine will need to hold references of {req,resp} but generally have nothing to do with the end user, so the post-response logging will happen in a goroutine that potentially lasts longer than ServeHTTP.
I wonder if it is worth considering to merge the detach functionality into the context library?

@i3d if the context is not tightly coupled with a specific request as is being proposed, creating another new top context object (with the intended detached timeout/deadline value) should be sufficient and no additions would have to be made to the context library. Which would be the generic pattern to handle any task than needs to outlast the execution context it stems from.
If any field had to be added to a http.Request, that would be a deadline, I would tend to think.
I am still concerned that if we choose the easy without rethinking execution contexts, unnecessary complexity will be introduced in the stdlib. (Sorry for being persistent about this)

Maybe I haven't fully grasp the principle proposed here but I thought the whole "parameter" concept is to clarify that a context (at least under http.Request topic) is tightly coupled with the Request (and in general, a context should be coupled with its intended operation), that's it, it is a parameter along with the Request, and it will be invalid to use aftermath. The detach functionality is to make sure anything that still valid for the Request (e.g. user, creds, etc) are transferred into the new context but the time sensitive values (e.g. deadline, timeout duration, cancellation, etc) are not. "Creating" a brand new context for the async operation, not sure if it can work on all cases.

In my understanding, the context should be tightly coupled with its goroutine ideally, i.e. passed as a separate argument.
It will be clearer since that's more likely to be the idiomatic use elsewhere. ( A function that does part of its job asynchronously should accept a context.Context as argument).
The idea of creating a new top level context is to detach the cancellation signals. Indeed, any request specific values can be copied to the new context as values. That should not be a problem.
In fact, your example typically makes me realize that if the context is kept as a proper argument, things will be clearer.
Since the http APIs are already frozen, the idea was to have a field on http.Request holding a context. If it is because a cancellation signal can be received from upstream then I guess yes, it is necessary. If it is just in order to pass a context.Context with or without a deadline/timeout, better use a proper context-aware wrapper around the API, imho.

What if the context on Request is unexported and has a quasi-getter that only allows you to obtain the context once, like
func (r *Request) Context() context.Context {
  if r.ctx == nil {
    panic("a message explaining you've done wrong")
  }
  c := r.ctx
  r.ctx = nil
  return c
}

and a quasi-setter that returns a new copy of the Request with the ctx set, like
func (r *Request) WithContext(ctx context.Context) *Request {
   rc := r.copy() //or whatever needs to be done
   rc.ctx = ctx
   return rc
}

It would only catch some misuse but it would at least make it clear that there's something going on that you should pay attention to, whether you're looking at the documentation or code in the wild.

Why not make Request itself implement Context? After all, Context is just and interface, and adding methods will not violate the Go 1 compatibility guarantee.

@rsc Overall the plan sounds OK. In the specific case of http.Request, I suggest we provide an API that makes it difficult to do the wrong thing. In particular, (*http.Request).Context could be a method that returns the Context, so the caller cannot change it. We provide the additional method (*http.Request).WithContext(Context) *Request to create a copy of the request with a new Context (which may be derived from the original request Context).

@rsc https://golang.org/pkg/net/http/#StripPrefix effectively "modifies the request in-place"

@chowey unless you're talking about embedding, making Request implement interface directly (via internal functionality) would greatly limit flexibility of what you can do with contexts. I want to be able to define my own context types with special functionality -- since http.Handler and http.HandlerFunc take Request pointers (not interfaces), I wouldn't be able to do anything all that interesting or useful with Request-based context functionality if I couldn't create a request with a custom context.

@Sajmani, I like that. I'll submit the context CLs (as package context for now, can be moved later) and proceed with the http and rest of the standard library plumbing. I have a bunch of other bugs which would be much easier with contexts available.

@Sajmani @bradfitz does this mean that Request will land its first unexported field? That sounds like a major shift? Granted, in this case it has getter/setters.

@extemporalgenome, yes, I guess so. I'm not that worried about it. I don't see how it prevents anything testing-wise, for instance.

It might affect assignability, in case someone else has some sort of clone of the Request struct (I don't have anything in mind re: this, or any idea why anyone would do this, but it would be a change).

@riannucci, I don't think it would. It would still be assignable, both as pointer and value.

(we won't add a mutex or sync.once to the http.Request)

I suspect @riannucci means in case someone tries to say: var req http.Request = struct { Method string; ... }{ ... } . In any case, adding struct fields is listed as not breaking compatibility.
Re unexported, I wasn't voicing a dissenting opinion, just a point of interest. I don't see how it'd affect testing either. With this proposal, https://godoc.org/golang.org/x/net/context/ctxhttp looks like it could go away altogether.

@bradfitz I think I was thinking of this: http://play.golang.org/p/gLXfzLvJlA (which doesn't compile, I picked pe.File as the first stdlib struct I could find with mixed eported/unxeported fields).

@extemporalgenome yeah, you're right, it's a moot point because this sort of change is explicitly blessed (and again, I don't know why anyone would have relied on this behavior before, but I've seen it happen across third-party libraries).

I'm a bit late to the discussion, but would it make more sense to add a Context to sql.Tx instead of the Query args? Automatic rollback on cancellation would be desirable, no? Alternative, maybe less desirable version would be a method like BeginWithContext(ctx)

@pbnjay, let's move database/sql-specific discussion to the newly-created #15123.

@bradfitz btw, I noticed your note about potentially changing the baseCtx from the server for each request in cabb1402568ae7d05d9d5adf569
Perhaps the same or related, what do you think about having a root context for the server that is used as the parent for each request context? this way, as the server is shutting down, it can signal to every context it should finish or cancel?

@pkieltyka, "it should finish" sounds like graceful shutdown (#4674) but semantically, canceling a context is much more abrupt than that.

@bradfitz yea its true that cancelling every in-flight context would be pretty abrupt and what I'm referring to is for purposes of a graceful shutdown.
The context tree is the one thing that could actually make it graceful though. Depends how a listener on the ctx.Done() channel considers the signal - should it cancel abruptly or just stop on the next tick? I'm pretty sure in the wild we'll see both cases.  hmm.. should there be a ctx.Canceled() channel?

FWIW, our team has used the cancelation of contexts for graceful shutdown (e.g. wiring SIGTERM -> cancel()). It would be useful to have degrees of cancellation (e.g. STOP EVERYTHING NOW v. Stop when you can). That said, we tend to write crashsafe software, so when we want stop-everything-now, we usually don't want to rely on the application logic to implement that (and we just kill -9 it).

@pkieltyka, @riannucci, changing the context API and/or semantics aren't really in scope for this bug. They've been frozen for some time. I suggest we move any discussion of graceful shutdown to #4674. The answer may involve contexts in some way, but not by canceling some root context.

There's a typo @ https://github.com/golang/go/commit/2cc27a7de9e7d14cb6702153688d02746c6a49ea#diff-2f929a7660c606fd0c5b268dad1a9bd4R415
waitDone is redefined inside the if block.
@bradfitz

@OneOfOne, https://go-review.googlesource.com/22581. In the future, you can reply to code in Gerrit which we use for our code reviews.

This is done, so closing. Context lives in package context for now (maybe subject to change yet, but unlikely), and is used by net, net/http, net/http/httptrace, and os/exec for now. The database/sql changes to use Context didn't happen for 1.7.

Context lives in package context for now (maybe subject to change yet, but unlikely)

Are there some contending alternative names that have been suggested? I am just curious, because I could not think of any. If there was a discussion about this elsewhere, can you point to that; I'd rather not start bikeshedding here.

Should we update blog.golang.org/context to point users to the
standard context package in Go 1.7+?

@minux, no rush. Eventually. Maybe at Go 1.8 time.

@riannucci
I share you concern about the WithValue linked-list performance.
Now I use below pattern in my codes,
func MyHTTPHandler(rw ResponseWriter, r *Request) {
  c := context.Background()
  c = c.WithValue(c, "placeholder", struct{Member1 string, Member2 string, Member3 int}{"foo", "bar", 42})
  myActualHandler(c, rw, r)
}

Do you think it could help the looking up performance or not?

@phuslu instead of guessing, why not write a benchmark?

@Thomasdezeeuw Thank you.
But the more important things(I think) is, Does this pattern is the RIGHT practice of context.Context ?
As you see, the original context.Context is used as a nested structure and seems that each node is immutable(Please correct me if I misunderstand). the original common pattern is,
ctx = context.WithValue(ctx, "placeholder", ...)

And the pattern which I used is treat context.Context as a mutable object -- although it may has potential performance improvement.
It will encourage people use below code against context.Context
ctx.Value("placeholder").(struct {}).Member1 = ...

It smells bad. Hopefully could get comments from you and others.

@phuslu @riannucci @Thomasdezeeuw, please move discussion of Context.WithValue performance to a new bug.

Hello!!
I have a question about sharing values between middlewares.
I think we'll be able to share values  between middlewares in the following code by go 1.7.
func (a *authHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
        user := GetUserFromRequestCookie(r)
        ctx := r.Context()
        ctx = context.WithValue(ctx, "user", user)

        r = r.WithContext(ctx)

        a.next.ServeHTTP(w, r) // next handler is ArticleHandle
}

func (a *ArticleHandle) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context()
    user := ctx.Value("user").(*User)
}

But r.WithContext(ctx) exec shallow copy for goroutine. I need not shallow copy for only sharing values.  It is slow. right? I want to set context to Request. But now it cannot.
func (a *authHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
        user := UserFromSession(r)
        ctx := r.Context()
        ctx = context.WithValue(ctx, "user", user)

        // r = r.WithContext(ctx)
        r.SetContext(ctx)

        a.next.ServeHTTP(w, r)
}

func (a *ArticleHandle) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context()
    user := ctx.Value("user").(*User)
}

Please tell me best way sharing values between middlewares using only net/http and context. I think we should not implement context wrapper like 3rd party framework impl for only sharing values.

@sona-tar We don't the issue tracker to ask questions. Please see https://golang.org/wiki/Questions for good places to ask. Thanks.

@davecheney Thx. I moved here. https://forum.golangbridge.org/t/share-values-between-middlewares-on-net-http/2952


Implement a package registry

Currently,  golang.org/x/tools/imports manages to import proper go packages in generated files, and it works in most cases. However, it has certain limitations. For example, if users specify civil.Date golang.org/x/tools/imports  cannot tell if they try to use cloud.google.com/go/civil or github.com/golang-sql/civil for sure.
Instead of leaving it to golang.org/x/tools/imports, this PR implements a package registry that imports an appropriate package based on fields of Spanner tables.  I'll implement the following features once this PR gets merged.

Make the default type module packages configurable
Make the default global module packages configurable
Allow users to configure imports by type (table)
Implement custom serialization/deserialization

You should regenerate test files. I think there will be the unexpected differences. (I'm not sure why CI didn't warn it)

@kazegusuri I've updated the PR to import packages on demand, which means packages are imported if they are actually used in the file. I'll leave some follow-up comments below. Would you review the PR again? 🙏

I'm closing this issue for now since we must reconsider the interface.


Fix out of range bug in date histograms

Fix for the bug reported at https://github.com/CartoDB/support/issues/962
The problem was that the total range between max and min timestamps is above the range of integer but both operands are within integer limits. PostgreSQL try to accommodate the result and it doesn't fit so an out of range error is triggered.
Making both values as double precision (float) fixes the bug.
This is consistent with the basics query with no overrides, since it gets the values from date using datepart function, that returns double precision.

Nice catch @ivanmalagon
Thank you so much for tackling this issue.
Just a minor detail, could you update NEWS with info relevant to this PR.
After that, please feel free to merge it.


⚠ Bofrost.de has degraded performance

In 5cb858d, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 2430 ms

Resolved: Bofrost.de performance has improved in 95c4b8e after 7 minutes.


Problems running nanocl on Ubuntu wth podman

Describe the bug
When playing with nanocl on Ubuntu 24.04 I've encountered couple of issues:

It's related to existing Ubuntu bug:
https://bugs.launchpad.net/ubuntu/+source/libpod/+bug/2040483 /
This prevents podman and thus nanocl from stopping containers.
Disabling apparmor heled.

ndns exists with this error:

dnsmasq: failed to create listening socket for <IP_ADDRESS>: Address in use
It's because it is used by podman's own dns service:
$ sudo lsof -nP -iUDP
COMMAND    PID            USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
aardvark- 9305            root    9u  IPv4  84026      0t0  UDP <IP_ADDRESS>:53

As described here https://www.redhat.com/sysadmin/podman-new-network-stack

After reboot of host majority of containers were gone - nanocld, ....
podman does not have daemon by default - so restart of containers should be managed by systemd -  not exactly sure why some containers restarted and some not . Did not look into details

While first issue is purely Canonical problem, second and third  one I think will be more generic - I assume this will affect nanocl on other platforms too.
To Reproduce
Clean install of Ubuntu 24.04 and sudo apt install podman
Install nanocl as per it's documentation (must do sudo nanocl install because of right on podman socket in /var/run/podman/podman.sock")
failing ndns cn be seen via sudo podman ps -a
reboot system, and see what containers are still running
Expected behavior
podman should works same as docker
Desktop (please complete the following information):

OS: Linux
Distribution: Ubuntu
Version 24.04

Hey thanks to have took time to open a very detailed issue, i'll investigate and provide a fix!

So i investigated why on system reboot container aren't restarting, and it's seems there is a problem with conmon this is the error i get:
Error: unable to start container "e4404aaae8a9fec8047942b6b7d9617edd6ff2b69b8220402b84df2e9d068d5d": container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...

I was able to fix it on a fresh ubuntu 24 install using this script:
It came from this issue: https://github.com/containers/podman/issues/21024
wget https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_22.04/amd64/conmon_2.1.2~0_amd64.deb -O /tmp/conmon_2.1.2.deb
sudo apt install /tmp/conmon_2.1.2.deb

After i got another error:
Error: unable to start container "614b6b451087c3966f93b7e25b6fbceeb7d27bbf630e96bc010f474418330d5c": crun: cannot stat `/run/nanocl`: No such file or directory: OCI runtime attempted to invoke a command that was not found

Looks like podman don't create mounted directory on restart.
The only workarround i see right now is to create a systemd profile that run before podman-restart that create the directory..

I'll add this later in the documentation but this is what the steps to add the systemd profile:
sudo vim /etc/systemd/system/nanocl-podman-restart.service

The profile:
[Unit]
Description=Create /run/nanocl directory before podman-restart
Before=podman-restart.service

[Service]
Type=oneshot
ExecStart=/bin/mkdir -p /run/nanocl

[Install]
WantedBy=multi-user.target

Then we enable the profile:
sudo systemctl daemon-reload
sudo systemctl enable nanocl-podman-restart

And it's look like there is deeper problem with podman.
Some of the data cannot be serialized correctly because they didn't correctly followed the docker structures.
Failed to deserialize JSON: unknown variant stopped, expected one of ``, created, running, paused, restarting, removing, exited, dead at line 1 column 192
I'll see if it come from the library i use but it generate the structure from the openapi spec from docker.

I'd recommend to try on distro with first class support for podman - like Fedora - behavior there might be different. Ubuntu support of podman seems to be bit dubious ...

Has this podman situation improved since?


Render contact points.

Add toggle to render contact points with DebugRenderManager or use Gizmos.

Implemented: 6aa389a.


Use AnimatedSwitcher's _childNumber as Key in layoutBuilder's Stack

Previously KeyedSubtree.wrap gave children of the AnimatedSwitcher their own key as a key, if available, and their _childNumber as a key, if not. This could lead to multiple children having the same key in the defaultLayoutBuilder's Stack and thus a duplicate key exception being thrown, though (see #121336 for a detailed description of how this bug occurs). This is fixed by always using the uniquely identifiable _childNumber as a key for the children of the AnimatedSwitcher.
This does not impair the functionality of having children of the same type be differentiated and transitioned between by giving them different keys, as the children's original key is still used to determine if a transition should take place or not.
This fixes #121336.
Pre-launch Checklist

[x] I read the Contributor Guide and followed the process outlined there for submitting PRs.
[ ] I read the Tree Hygiene wiki page, which explains my responsibilities.
[ ] I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
[ ] I signed the CLA.
[x] I listed at least one issue that this PR fixes in the description above.
[x] I updated/added relevant documentation (doc comments with ///).
[ ] I added new tests to check the change I am making, or this PR is test-exempt.
[ ] All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

@JimGerth Can you please sign the CLA (see previous message) and add a test for your change? Also, it looks like some other checks are failing as well. Please take a look at those as well. Thanks.

I've written with @Hixie on discord and they told me to change (or replace in this case) the old test.
I eventually decided to entirely revert the previous PR #107476. See my previous comment, the PR description and the commit message for my reasoning.

auto label is removed for flutter/flutter, pr: 121408, due to Validations Fail.

In cl/512895929 there's a slight goldens diff that could possibly be due to this change, WDYT @goderbauer?

In cl/512895929 there's a slight goldens diff that could possibly be due to this change

I triaged them as acceptable.


Replay attack for signed messages

What is this issue about?
The LSP6-KeyManager uses a signed message for executing a relay call through executeRelayCall(...) function.
According to the LSP6 Specs:

The message to sign MUST be of the following format: KeyManager address + signer nonce + data payload . These 3 parameters MUST be:

packed encoded (not zero padded, leading 0s are removed)
hashed with keccak256

The final message MUST be signed using ethereum specific signature, based on EIP-191.

This way of signing messages, introduces a security risk, allowing users to replay these messages across different chains if:

The KeyManager is deployed on the same address    ||  (Could be done  ✅ )
The nonce of the signer is the same                             ||  (Could be done ✅ )
The data (Payload) is the same                                     ||  (Could be done ✅ )

Suggested Solutions
Looking at the EIP-155, they included the chainId in the transaction to ensure that a transaction could be signed for only one chain.
Message signing utilizes the same mechanisms as transaction signing, but message signing doesn't include this kind of extra information. It only includes what you give it to, so in our case, it will include just the KeyManager address, the nonce, and the data (Payload).
We could have 2 solutions:

Drop EIP-191 support for signing messages and adopt EIP-712 which includes the chainId in the signing message, and is also used to structure the signed message so it is more readable.
Including the chainId in the signed message without using the EIP-712.

To watch out for:

chainId should be used and not networkId, as some chains have the same networkId but different chainId.
ETH and ETC have the same networkId but different chainId.
chainId shouldn't be referenced with a variable in the constructor while deploying: ❌ ❌ ❌ ❌

    constructor(address _target) {
        target = _target;
        chainId = block.chainid;
    }

Instead it should be mentioned explicitly in the smart contract while forming the signed message:  ✅ ✅ ✅ ✅
        bytes memory signedMessage = abi.encodePacked(
            block.chainid,
            address(this), // needs to be signed for this keyManager
            _nonce,
            _data
        );

Because, in the event of a post-deployment hard fork of the chain, the chainId cannot be updated, and signatures may be replayed across both versions of the chain. As a result, an attacker could reuse signatures to execute the relay call twice.

We decide against this as it increases the complexity on the signing side a lot, especially for non web libraries like web3swift and others. While the only benefit is the increased visibility in tools like MetaMask, which are not the preferred ways to handle, manage and use a Universal profile.
In the current method we will also sign chain id, which will make the signature safe from replay attacks.
getting the call data from a call to the executeRelayCall(signature, none, data) function, is as simple as cutting of the first bytes. So visibility and decodability is given for the current way of signing, while making signing as simple, as hashing and signing a simple message sign(keccak256(chainid, keyManagerAddress, nonce, data))


modify keyvault certificates starter readme

Modify README of azure-spring-boot-starter-keyvault-certificates to follow the specification of sdk repo.

After this PR merged, maybe we should create another PR to change tenantId to tenant-id.
Refs: https://github.com/Azure/azure-sdk-for-java/pull/17741


Filter iso code to include

Proposed changes
Possibility to show a subset of original countries (a sort of filtered list). Subset must be passed to builder as list of ISO CODES.
Types of changes
What types of changes does your code introduce?
Put an x in the boxes that apply

[ ] Bugfix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist
Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

[x] I have read the CONTRIBUTING doc
[ ] I have added necessary documentation (if appropriate)
[x] Any dependent changes have been merged and published in downstream modules

Further comments
If this is a relatively large or complex change, kick off the discussion by explaining why you chose the solution you did and what alternatives you considered, etc...

This does not serve a generic use case

why not?
I was interested in a library to let user chose the language to use in the UI and I've translation for a subset of languages, so this kind of feature was useful in my use case (or if you can apply a filter automatically based on the available resource string would be fine too)
The library as it is doesn't cover this use case


Is Value.forAll working properly for empty values?

if (Option.none().forAll("aaa"::equals)) {
    System.out.println("YES");
} else {
    System.out.println("NO");
}

if (List.empty().forAll("aaa"::equals)) {
    System.out.println("YES");
} else {
    System.out.println("NO");
}

prints:
YES
YES

I expected:
NO
NO

Which is correct?

Vavr is correct, as the conjunctive identity is true, and forAll is essentially a conjunction. See also Universal quantification for the empty set


Error: could not connect to ollama app, is it running? Windows

What is the issue?
I am facing this error:-
Error: could not connect to ollama app, is it running?
can someone please help me out?
OS
Windows
GPU
Nvidia, Intel
CPU
Intel
Ollama version
0.1.48

Hi @ArshSharan - is Ollama running in the tray bar? You may need to start it first. If you're still seeing this issue let me know and I can reopen 😊

Yes, it runs in the tray bar, but it still gives me the same error. When I write the command a few times on the terminal, it opens multiple ollama (icons) in my tray bar.

I also encountered this issue, and Ollama was NOT in the tray bar.  Running ollama start resolved the issue for me, but Ollama is still not in the tray bar (although it usually is).
My OS is Windows and my computer crashed a few times earlier this week, which I blame on Windows.
ollama version is 0.3.0


"dot" with args ['-Tpng', '/tmp/tmp20yg4hj7'] returned code: 1

Have this problem python 3.7
`Connecting to zabbix
Getting zabbix hosts from group MES3124F
Getting data from hosts
Getting data from hosts
"dot" with args ['-Tpng', '/tmp/tmp20yg4hj7'] returned code: 1
stdout, stderr:
b''
b"Error: /tmp/tmp20yg4hj7: syntax error in line 5 near ','\n"
Traceback (most recent call last):
File "./zabbix_lldp_map.py", line 233, in
graph.write_png(get_config('graphviz.imagefile'))
File "/usr/lib/python3/dist-packages/pydot.py", line 1734, in new_method
encoding=encoding)
File "/usr/lib/python3/dist-packages/pydot.py", line 1817, in write
s = self.create(prog, format, encoding=encoding)
File "/usr/lib/python3/dist-packages/pydot.py", line 1945, in create
assert process.returncode == 0, process.returncode
AssertionError: 1
`

Hi. I have this problem too.
Python 3.9.5
`
Connecting to zabbix
Getting zabbix hosts from group autodiscovered_test
Getting data from hosts
Getting data from hosts
"dot" with args ['-Tpng', '/tmp/tmpx5hbk_44'] returned code: 1
stdout, stderr:
b''
b"Error: /tmp/tmpx5hbk_44: syntax error in line 5 near ','\n"
Traceback (most recent call last):
File "/etc/zabbix/zabbix-scripts/zabbix_lldp_map-master/zabbix_lldp_map.py", line 232, in
graph.write_png(get_config('graphviz.imagefile'))
File "/etc/zabbix/zabbix-scripts/env/lib/python3.9/site-packages/pydot.py", line 1743, in new_method
self.write(
File "/etc/zabbix/zabbix-scripts/env/lib/python3.9/site-packages/pydot.py", line 1828, in write
s = self.create(prog, format, encoding=encoding)
File "/etc/zabbix/zabbix-scripts/env/lib/python3.9/site-packages/pydot.py", line 1956, in create
assert process.returncode == 0, (
AssertionError: "dot" with args ['-Tpng', '/tmp/tmpx5hbk_44'] returned code: 1
`

I also had the problem, and after analyzing the code, the error was caused by incorrect formatting when generating the dot code.
You can print out the value of the variable 'graph' and paste it into this URL(http://dreampuf.github.io/GraphvizOnline) to verify that the generated dot code is correct, If the format is wrong, you'll see that the error output is the same as your stderr output.

The location of the code causing the error is on line 127, the program will take the host's type inventory field and look for the icon file from the current folder based on the mapping relationship in the configuration file, so you need to check whether the type field of the host is filled in.


Adapt Import to allow railtoolkit/schema format

done for Path in https://github.com/railtoolkit/TrainRuns.jl/commit/00eda496f5535e4007055cef406b243b2ddf2857

done for Trains in https://github.com/railtoolkit/TrainRuns.jl/pull/12/commits/a5868af2c5633972cb8e666e5cd9c61b67c5992a


Fix RP prod disk

Extending /, /var, /home in cases where libs explode in size. Noticed this in the Canary region.
du -hs /usr/lib/* | sort -rh | head -n 10
512M    /usr/lib/firmware
du -hs /usr/lib64/* | sort -rh | head -n 10
406M    /usr/lib64/az
#some other nits

and 2Gb for / is gone...

and 2Gb for / is gone...

I saw this being the case with core files related to some auoms bug - emergency fix to remove abrt in #1167.

closing for #1172


Will it consider implement NIP23

I noticed that nostr-rs-relay does not support long text at present, but there have been long text proposals a few days ago, such as NIP23. Will you consider implementing this NIP?

Why do you think we do not support "long text"?  NIP-23 does not require anything special from relays, except for NIP-33, which we already support.
The default maximum length of events is 128KB; however this is fully configurable, and could be extended to much larger values if desired.

Why do you think we do not support "long text"? NIP-23 does not require anything special from relays, except for NIP-33, which we already support.
The default maximum length of events is 128KB; however this is fully configurable, and could be extended to much larger values if desired.

oh, thanks! Greate

ok, thanks!


TypeError: set on proxy: trap returned falish for property 'removeListener'

Describe the bug
When I have both MetaMask and Exodus extensions enabled, i get this error:

When I disable Exodus - everything is ok.
This is my config:
import { createWeb3Modal } from "@web3modal/wagmi/react";
import { useEffect, useState } from "react";
import { bsc, polygon, mainnet } from "wagmi/chains";
import { WagmiProvider } from "wagmi";
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
import { createConfig, http } from "wagmi";
import { injected, safe, walletConnect } from "wagmi/connectors";

const projectId = "xxx";

const queryClient = new QueryClient();

type W3MProviderProps = {
  children: React.ReactNode;
};

const metadata = {
  name: "xxx",
  description:
    "xxx",
  url: "https://xxx/",
  icons: ["xxx"],
};

const config = createConfig({
  chains: [mainnet, bsc, polygon],
  connectors: [
    walletConnect({ projectId, metadata, showQrModal: false }),
    injected({
      target: "metaMask",
      shimDisconnect: true,
    }),
    safe({ allowedDomains: [/app.safe.global$/], shimDisconnect: true }),
  ],
  transports: {
    [mainnet.id]: http(),
    [bsc.id]: http(),
    [polygon.id]: http(),
  },
});

const safeWalletId =
  "xxx";

createWeb3Modal({
  wagmiConfig: { ...config, connectors: [config.connectors[0]] },
  projectId,
  includeWalletIds: [safeWalletId],
  allWallets: "HIDE",
  themeMode: "light",
  themeVariables: {
    "--w3m-z-index": 2000,
  },
});

export function WagmiWeb3ModalProvider({ children }: W3MProviderProps) {
  const [ready, setReady] = useState(false);

  useEffect(() => {
    setReady(true);
  }, []);

  if (!ready) {
    return null;
  }

  return (
    <WagmiProvider config={config} reconnectOnMount={false}>
      <QueryClientProvider client={queryClient}>{children}</QueryClientProvider>
    </WagmiProvider>
  );
}

And I'm not able to connect to MetaMask when Exodus is enabled and it also not appears as one of the connectors.
Why is it happening and how to fix it?
Link to Minimal Reproducible Example
No response
Steps To Reproduce
Enable both MetaMask and Exodus extensions.
Config should look like this:
const config = createConfig({
  chains: [mainnet, bsc, polygon],
  connectors: [
    injected({
      target: "metaMask",
      shimDisconnect: true,
    }),
  ],
  transports: {
    [mainnet.id]: http(),
    [bsc.id]: http(),
    [polygon.id]: http(),
  },
});

Wagmi Version
2.5.6
Viem Version
2.7.15
TypeScript Version
^5.3.3
Check existing issues

[X] I checked there isn't already an issue for the bug I encountered.

Anything else?
No response

Looks like a Web3Modal issue


transfer file and folders from repository big-data-analysis

transferred the contents of the defunct repository big-data-analysis into this repository
deleted the repository big-data-analysis

ok to merge with the base branch.


Improving visual QA + System-initiated notifications

Adding MRTK voice commands for simple system commands (next, previous step)

Adding visual and task context to QA

Overall UX improvements for the QA

Merging angel_system with the latest version of the unity ARUI plugin

ARUI shows notification and provides audio feedback when system detects a skip error

@ovenmitt For any binary files included here, do you think we could add those as git-LFS tracked items? I know there are some complexities around quotas and bandwidth limitations, so I'm not sure if you are aware of those for public github projects. It is ultimately the "right thing to do" with git to keep binaries files out of the git history.

@ovenmitt For any binary files included here, do you think we could add those as git-LFS tracked items? I know there are some complexities around quotas and bandwidth limitations, so I'm not sure if you are aware of those for public github projects. It is ultimately the "right thing to do" with git to keep binaries files out of the git history. If we go this route, we will have to filter through commits to replace file contributions with GitLFS usage, or we could possible squash all of these changes into a single commit for simplification.

I will remove the files, since they are not necessary for the ARUI to work at this point.

Hi @ovenmitt! I have a rebase of this branch onto latest master as well as a fix for the code style of the files causing that CI failure. Do you mind if I force-push that to here?

I have a fix for what is causing the code-style CI to fail, but I can't seem to push it here, despite the Maintainers are allowed to edit this pull request. seeming to be enabled. The following diff patch file may be applied with git apply style-fix.txt
style-fix.txt

Can you run ./angel-workspace-shell.sh -- black . in your terminal, observe the changed and then commit those? That should be what is necessary to address the style issues. The items that I keep seeing crop up in this CI job are consistent with what that call fixes.


[RESOLVED] Interface eth0(): High error rate (>2 for 5m)

Problem started at 10:39:29 on 2023.05.05
Problem name: Interface eth0(): High error rate (>2 for 5m)
Host: S100-Primario
Severity: Warning
Operational data: errors in: 3, errors out: 0
Original problem ID: 20343
Event details in Zabbix: http://<IP_ADDRESS>/tr_events.php?triggerid=23483&eventid=20343

Problem has been resolved in 6m 0s at 10:45:29 on 2023.05.05
Problem name: Interface eth0(): High error rate (>2 for 5m)
Host: S100-Primario
Severity: Warning
Original problem ID: 20343


cli tool still outputs ember to the console

Currently ember-cli does not make use of the cli.name parameter we pass it, and so it always outputs ember in its help commands for example.
I already started work in my ember-cli fork to fix that.
I should have a PR for ember-cli soon.
I'll update this issue with my progress in the coming days.

:+1:

So is this still happening? I was confused when I loaded this up. :D

This has been resolved in PR #157


Replication Controller Documentation is not Present.

I haven't been able to find any comprehensive documentation on Replication Controllers within the project's repository.

@Phantom-Intruder Would you be interested to write on this topic???

You're most welcome @gauravrattan


cannot build rdkakfa-sys with M1 Mac according to developer guide

We've added the support for ssl/sasl in #5693 and I found not able to build rdkafka with M1 Mac according to the developer guide. After some attempts, here's a possible solution:

brew install openssl@3.
No need to brew install cyrus-sasl.
Before building, export OPENSSL_ROOT_DIR="/opt/homebrew/opt/openssl@3".

Any ideas? cc @xx01cyx @skyzh @tabVersion
Related: https://github.com/fede1024/rust-rdkafka/issues/491

brew link --force openssl works for me

brew link --force openssl works for me

agreed. I followed 2nd solution and fixed it https://stackoverflow.com/questions/29792009/cmake-osx-mac-openssl-brew

openssl@3 is a keg-only formulae, I'm afraid force-link it may break some other applications. 🤔
openssl@3 is keg-only, which means it was not symlinked into /opt/homebrew,
because macOS provides LibreSSL.

For me

brew link --force openssl
export OPENSSL_INCLUDE_DIR=/path/to/openssl/include

I don't think force-link is a good idea. What about changing to static link for all platforms? 🤣 We came back to the old path.

... hopefully it won't hurt compile time too much, and won't burst target directory size. Considering we already have so many crates?

... or make rdkafka's SSL an optional feature for dev profile?

Or how about forking the rdkafka-sys to edit the cmake configuration until the upstream fixes that? 🤣 Note that we're already using the patch madsim-kafka.

Looks good.

Wait a minute. If we enable dynamic-linking feature of rdkafka, then users only need to install librdkafka?

Doesn't work. Need some adjustments on openssl.

I don't think force-link is a good idea. What about changing to static link for all platforms? 🤣 We came back to the old path.

I have the openssl in my PATH but cmake still cannot find the dep

In case someone might need it, you can add them to .vscode/settings.json 🥸:
    "terminal.integrated.env.osx": {
        "OPENSSL_ROOT_DIR": "/opt/homebrew/opt/openssl@3"
    },
    "rust-analyzer.cargo.extraEnv": {
        "OPENSSL_ROOT_DIR": "/opt/homebrew/opt/openssl@3"
    },


Item not select

After open a select if scroll the values with right scroll bar  and try to select a  option  this not selected.

Strange, it's work with mouse wheel but not with bar... I try debug this issue this week, I have big release next week, so probably fix that before if I have same on business application.


RucioInjector fails to create rules against MSS endpoints

Impact of the bug
WMAgent
Describe the bug
I'm actually not sure whether it's a bug, or whether there is no such RSE. However, while running some tests, I noticed that the RucioInjector component fails to create a container rule against an MSS endpoint.
How to reproduce it
Create a rule against T1_US_FNAL_MSS
Expected behavior
There are actually two issues to be tackled here.

is T1_US_FNAL_MSS a valid RSE? Is our WMAgent rucio accounts allowed to create such rules?
ReqMgr2 supports workflow parameters for custodial and non-custodial sites (in addition to priority, phedex group, etc). Rucio does not such concept, so once we start migrating to RucioInjector, it could be that the component will try to make 2 container rules for the same dataset (1 custodial, 1 non-custodial). I think we will have to drop/skip the custodial data placement and only proceed with the non-custodial.

Additional context and error message
From RucioInjector in vocms0193:
2020-04-15 10:08:27,589:140658649351936:INFO:RucioInjectorPoller:Creating container rule for /RelValH125GGgluonfusion_14/Integ_Test-SC_LumiMask_PhEDEx_Agent133_Val_Alanv11-v11/NANOAODSIM against RSE T1_US_FNAL_Disk
2020-04-15 10:08:29,295:140658649351936:INFO:RucioInjectorPoller:Container rule created for /RelValH125GGgluonfusion_14/Integ_Test-SC_LumiMask_PhEDEx_Agent133_Val_Alanv11-v11/NANOAODSIM under rule id: [u'814097ee9ee740eba048c15ac1a92c55']
2020-04-15 10:08:29,295:140658649351936:INFO:RucioInjectorPoller:Creating container rule for /RelValH125GGgluonfusion_14/Integ_Test-SC_LumiMask_PhEDEx_Agent133_Val_Alanv11-v11/NANOAODSIM against RSE T1_US_FNAL_MSS
2020-04-15 10:08:30,719:140658649351936:ERROR:Rucio:Exception creating rule replica for data: ['/RelValH125GGgluonfusion_14/Integ_Test-SC_LumiMask_PhEDEx_Agent133_Val_Alanv11-v11/NANOAODSIM']. Error: Provided RSE expression is considered invalid.
Details: RSE Expression resulted in an empty set.

@ericvaandering @nsmith- can you please help me understand this MSS failure?

It's *_Tape these days (as all mass storage systems in use by CMS are tape)
You can see RSEs via rucio list-rses.
Worth keeping in mind, some tape RSEs will only allow rules created with ask_approval=True (I am not sure if that is automatically set for those or not, Eric might)

Really, though, we need to abandon CustodialSites and NonCustodialSites in favor of a structure like
rules: [{'rse_expression': '...', 'copies': 1}, ...]

for defining the container rules placed on workflow outputs

In the interim the same can be accomplished entirely within rucio via subscription mechanism, which matches DIDs to regular expression on injection and creates rules. We can even use this for now and figure out what to do about final output placement later.

AFAICS, Unified does not use the CustodialSites configuration, so it's very unlikely that WMAgent will have to create data placement against _Tape RSEs (maybe T0 only). Which means, at least during this migration, we could perhaps skip the CustodialSites data placement and only perform the non-custodial one.
As for those workflow parameters, I have made another GH issue:
https://github.com/dmwm/WMCore/issues/9639
such that we don't forget to make those changes when PhEDEx gets out of the picture.

_MSS is not a valid RSE. They are called _Tape in Rucio. However, we've also got agreement that under Rucio we will hold off on writing NanoAOD to tape for a bit.
I thought these subscriptions would be put in place by Unified, not the agent.
For the short term no one will make the tape request. For catchup, Ops will have to do it, maybe with subscriptions. Long term, do we want to do it with subscriptions still or have the agent start doing it?

I see Nick has largely said the same thing while I was being interrupted with something else. :-)

Sounds good. It might be worth understanding why unified does the custodial copy itself rather than setting CustodialSites. (e.g. https://cmsweb.cern.ch/phedex/prod/Request::View?request=2177988 )

Another requirement from the Rucio migration discussion we had today.
Besides the mapping from _MSS to _Tape, we should implement the following logic for tape (custodial) requests:

first we create a standard rule request, thus leaving ask_approval=False. Some sites might forbid it, which means the rule creation will fail. If it does, then
create the same rule request, but this time explicitly set ask_approval=True. This time it should go through.

@nsmith- can you please confirm whether the comment/description above is still correct?

Yes that description is correct. I looked into the type of exception and one would expect an rucio.common.exception.AccessDenied exception for a site that requires approval. That exception could be due to other access denial reasons, but at least it is much less likely to stem from some validation error or otherwise.


chore: migrate to react 18

What:
Migrate to react 18

Why:
We want to stop supporting react<18 in Wave 2.0 (#239)

How:

Bump react and react-dom versions to ^18.2.0 (not sure if we should target ^18.0.0 instead)
Use react's useId instead of the internal generateId in useGeneratedId

Checklist:

[ ] Fix type errors
[ ] Use react's useId in Month.tsx instead of the internal generateId
[ ] Remove our internal util generateId once it's no longer used
[ ] Remove nanoid dependency
[ ] Remove IE11 polyfill required for nanoid once it's no longer used (basically remove polyfills.ts)
[ ] Fix dependency errors (docz is not compatible with React 18)
[ ] Ready to be merged

Since we already refactored useGeneratedId to not use nanoid I think it makes sense to finish removing nanoid in this PR, that would mean that #240 is a duplicate, I'm keeping it alive for now to hear other's opinions

I'm closing this since I was using a fork and now I've opened the PR directly from the branch of this repo in #311


cigien: Watch martha[\W_]+stewart[\W_]+cbd[\W_]+oil(?:[\W_]+(?:\d++|[\da-f]{5,}+)s?)*

cigien requests the watch of the watch_keyword martha[\W_]*+stewart[\W_]*+cbd[\W_]*+oil(?:[\W_]*+(?:\d++|[\da-f]{5,}+)s?)*. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

martha[\W_]*+stewart[\W_]*+cbd[\W_]*+oil(?:[\W_]*+(?:\d++|[\da-f]{5,}+)s?)* has been seen in 1 true positive, 0 false positives, and 0 NAAs.

martha[\W_]*+stewart[\W_]*+cbd[\W_]*+oil(?:[\W_]*+(?:\d++|[\da-f]{5,}+)s?)* has been seen in 1 true positive, 0 false positives, and 0 NAAs.

Approved by tripleee in Charcoal HQ

Approved by tripleee in Charcoal HQ


Reconfiguring built-in Prometheus listeners (TCP ports 20201, 20202)

Describe the bug
Prometheus exporter TCP ports (20201, 20202) are enabled by default on Ops Agent, which causes problems for users wanting to bind to those ports for other purposes, or reduce network exposure.
While the Prometheus listeners are fairly minimal (a fairly simple handler for /metrics), since the daemons run as root, users wanting to run Ops Agent in a security–sensitive environment will want to eliminate inbound requests.
Additionally, users wanting to run their own service binding to TCP ports 20201 or 20202 will run into conflicts.
There's no obvious way to reconfigure these ports, whether changing the binding address or port numbers. At the very least, it should be possible to bind these ports to localhost instead (::1 or <IP_ADDRESS>).
I recognise that these ports are used by Ops Agent for self monitoring, so avoiding listening on the ports entirely is likely infeasible.
To Reproduce
Steps to reproduce the behavior:

Environment: RHEL 7, Ops Agent google-cloud-ops-agent-2.22.0-1.el7.x86_64
Use default config
Run netstat -anp | grep :2020:tcp6       0      0 :::20201                :::*                    LISTEN      15072/otelopscol
tcp        0      0 <IP_ADDRESS>:20202           <IP_ADDRESS>:*               LISTEN      15123/fluent-bit
tcp        0      0 <IP_ADDRESS>:20202         <IP_ADDRESS>:45340         ESTABLISHED 15123/fluent-bit
tcp        0      0 <IP_ADDRESS>:45340         <IP_ADDRESS>:20202         ESTABLISHED 15072/otelopscol
tcp        0      0 <IP_ADDRESS>:51388         <IP_ADDRESS>:20201         ESTABLISHED 15072/otelopscol
tcp6       0      0 <IP_ADDRESS>:20201         <IP_ADDRESS>:51388         ESTABLISHED 15072/otelopscol

Observe that ports 20201 and 20202 are bound to the zero address.
Observe this config in /run/google-cloud-ops-agent-opentelemetry-collector/otel.yaml:  telemetry:
    metrics:
      address: <IP_ADDRESS>:20201

Observe this config in /run/google-cloud-ops-agent-fluent-bit/fluent_bit_main.conf:[OUTPUT]
    Match *
    Name  prometheus_exporter
    host  <IP_ADDRESS>
    port  20202

Expected behavior
I would expect to be able to reconfigure the bind address to force the ports to bind to localhost only (::1 and <IP_ADDRESS>), as well as change the port numbers.
While iptables rules are additionally useful as a defense-in-depth method, avoiding binding unnecessarily in the first place may be preferable.
Environment (please complete the following information):

VM distro / OS: RHEL 7
Ops Agent version: google-cloud-ops-agent-2.22.0-1.el7.x86_64
Ops Agent configuration: default config

Additional context
https://issuetracker.google.com/251023934

I'm having the same issue.
@jeremyvisser When I try that link to the Google Issuer Tracker, I get Access Denied. Was there any information there beyond what you wrote here?
Did you try changing the host in those config files to <IP_ADDRESS>?

The link is just context about how the issue affected internal usage of the tool, but otherwise uninteresting.
I didn’t try altering those /run/google-cloud-ops-agent-*/* files, as they’re dynamically generated every time Ops Agent launches. Even if it “worked”, it wouldn’t be the correct fix.


change to enable laptop investigation of disp net

adds modified chunk sizes for 8gb laptop work (for balls/mutant configs)
adds L2 dispnet regularization options
adds dispnet visualization to render notebook

gah got wrong origin :/


Clustered and generalised VRP

Is your feature request related to a problem? Please describe
This issue tracks clustered VRP and generalised VRP. The two are somewhat related, since they both specify clusters (groups) of clients. The difference is in what's required for these clusters:

[ ] Clustered VRP requires all clients in the same group to be visited in sequence, one after another, before the route maybe continue with clients outside the cluster. We can already do this by modifying the distance matrix, but that does not work well with e.g. a max distance constraint.
[x] Generalised VRP requires only a single client in the group is visited, rather than all of them. We already support optional clients, so we do not have to worry about not all locations being visited. But we do still need to build in some sort of support for this.

We do not have any facility in place to deal with such higher-level constraints currently.
Describe the solution you'd like
Support for clustered and generalised VRP.
Additional context
Came up in #441.

Let's:

[ ] Do a mini literature review on applications (relevance) and algorithmics (solution approaches);
[ ] Think about an interface for this;
[ ] Think about how this may be supported algorithmically.

Interface could be something like:
from pyvrp import Model

m = Model()

cluster = m.add_cluster(only_one=True/False, name="group")

client1 = m.add_client(..., cluster=cluster)  # now part of the cluster
client2 = m.add_client(..., cluster=None)  # default, not part of any cluster

One cannot add depots to clusters. Additionally, what should the default be for only_one? Should we provide a default? And is only_one a good name?

OK, so some more thinking about the interface:

I like client_group more than cluster. This also immediately makes clear that depots should not be passed - those are not clients!
only_one could be renamed to mutually_exclusive, somewhat similar to how argparse handles mutual exclusion. I am not sure about a default value.

The above definition also covers VRP with multiple time windows as a special case:

For each of a client's time windows, duplicate the client and assign it one of the time windows.
Mark the duplicated client as optional.
Add all the duplicated clients to a mutually exclusive client group.
Remove the original client with multiple time windows from the problem.
???
Profit

Depending on the speed at which we implement multi-trip VRP (and the speed at which I can finish my last paper 😆), this may or may not make the 0.8.0 release. Tentatively 0.9.0, but you never know!

This description of the VRP with lunch breaks given in Vidal et al. (2014) is simply a mutually exclusive client group that needs to be visited only when a route is non-empty:

This will be really easy to add in if we support client groups.

Returning to this, I have realised a few things:

Clustered VRP should be possible already: simply make all edges into, and out of, the cluster sufficiently expensive. Then the solver should realise that entering and leaving the cluster only once (i.e., visit every client in the cluster consecutively) is optimal.
Clustered VRP with weak cluster links - all clustered clients must be on the same route - is equivalent to a precedence constraint that requires all clustered clients to be visited before some dummy node is visited. So this is essentially a pickup and delivery problem, where all clustered clients want to route a zero load to the dummy node. When we tackle pickup and delivery in #331, we get this variant essentially for free. That seems to be a recurring theme.
Generalised VRP is the only variant here that is non-trivial. So we should probably tailor this issue to focus only on that problem.

Returning to this, I have realised a few things:

Clustered VRP should be possible already: simply make all edges into, and out of, the cluster sufficiently expensive. Then the solver should realise that entering and leaving the cluster only once (i.e., visit every client in the cluster consecutively) is optimal.
Clustered VRP with weak cluster links - all clustered clients must be on the same route - is equivalent to a precedence constraint that requires all clustered clients to be visited before some dummy node is visited. So this is essentially a pickup and delivery problem, where all clustered clients want to route a zero load to the dummy node. When we tackle pickup and delivery in Pickup and delivery #331, we get this variant essentially for free. That seems to be a recurring theme.
Generalised VRP is the only variant here that is non-trivial to express in the form of some other VRP. So we should probably tailor this issue to focus only on that problem.

Let's then add an argparse like interface to pyvrp.Model, since argparse solved this problem so well:
m = pyvrp.Model()

group = m.add_mutually_exclusive_group()
client1 = group.add_client(...)
client2 = group.add_client(...)

Moving this up to v0.8.0.

We should:

[ ] Find some benchmark instances;
[ ] Think about the granular nieghbourhoods for mutually exclusive groups. They probably shouldn't be in each other's neighbourhood because only one of them can ever be in the solution at the same time.
[ ] More?

I'm waiting for a reply to a request for benchmark instances I emailed about yesterday. Once I have them I'll run a benchmark on literature instances.

I have some benchmark instances for the GVRP now. With #514 nearly finished we can thus start benchmarking to compare.

GVRP is now implemented. Let's get the benchmark instances underway and then we should be good to go for that variant. I'll keep the issue open for CluVRP.

I'm running a benchmark of current main (c50ed1d335d3dba89d856a9095468a4467a6207e; job ID 7685011; 1h GVRP) on the benchmark instances of https://github.com/PyVRP/Instances/pull/10.

GVRP is done; renaming this to CluVRP.

The relevant part of this issue for 0.8.0 was GVRP, which is now done. CluVRP will probably happen at some point as well, but that's not currently planned. I suspect either in 0.9.0, or 0.10.0, but it's a bit of a stretch goal.

Could be fun to pick this up for 0.10.0, to flesh out the client groups a bit more.

Maybe combine with group-based service level (i.e., min_num_services for a group) based on what I gather from #441.


Maintained ?

Hi,
Just wondering if this is maintained ?
Best regards,

My teams moved from a multi repo to a mono repo so we aren't really using this any longer.  If folks want to fork and run with it I am cool with that.  I am going to update the README.md that we aren't using this any longer...


miller: add caveat message

In response to johnkerl/miller#343 (approved by @johnkerl), add a humorous message that's shown to the user when they install Miller.  (Or when they request info about the formula.)
Resolves johnkerl/miller#343.

[x] Have you followed the guidelines for contributing?
[X] Have you checked that there aren't other open pull requests for the same formula update/change?
[X] Have you built your formula locally with brew install --build-from-source <formula>, where <formula> is the name of the formula you're submitting?
[X] Is your test running fine brew test <formula>, where <formula> is the name of the formula you're submitting?
[x] Does your build pass brew audit --strict <formula> (after doing brew install <formula>)?

Opened new PR in correct repo: https://github.com/Homebrew/homebrew-core/pull/57327


SSL Error

I get ssh error from the mopidy-youtube extension.
This is the logfile:
2018-02-25 15:00:30,277 INFO [1639:YouTubeBackend-7] mopidy_youtube: Searching YouTube for query 'Ycg5oOSdpPQ'
2018-02-25 15:00:30,707 ERROR [1639:Core-9] mopidy.core.library: YouTubeBackend backend caused an exception.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/mopidy/core/library.py", line 19, in _backend_error_handling
yield
File "/usr/local/lib/python2.7/dist-packages/mopidy/core/library.py", line 347, in search
result = future.get()
File "/usr/lib/python2.7/dist-packages/pykka/threading.py", line 52, in get
compat.reraise(*self._data['exc_info'])
File "/usr/lib/python2.7/dist-packages/pykka/compat.py", line 12, in reraise
exec('raise tp, value, tb')
File "/usr/lib/python2.7/dist-packages/pykka/actor.py", line 201, in _actor_loop
response = self._handle_receive(message)
File "/usr/lib/python2.7/dist-packages/pykka/actor.py", line 295, in _handle_receive
return callee(*message['args'], **message['kwargs'])
File "/usr/local/lib/python2.7/dist-packages/mopidy_youtube/backend.py", line 192, in search
tracks=search_youtube(search_query)
File "/usr/local/lib/python2.7/dist-packages/mopidy_youtube/backend.py", line 98, in search_youtube
result = session.get(yt_api_endpoint+'search', params=query)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 521, in get
return self.request('GET', url, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 508, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 618, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 506, in send
raise SSLError(e, request=request)
SSLError: HTTPSConnectionPool(host='www.googleapis.com', port=443): Max retries exceeded with url: /youtube/v3/search?q=Ycg5oOSdpPQ&part=id&type=video&key=AIzaSyAl1Xq9DwdE_KD4AtPaE4EJl3WZe2zCqg4&maxResults=15 (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)'),))

Have you tried updating certifi?

No... Can you please explain me how?
I also noticed that the api method called is not the same as in the googleapis description here (the required parameter 'part' is missing)

Pip install --upgrade certifi. The code should be fine.

Thank you for your reply. I solved the ssl issue installing requests[security], certifi was already up-to-date.


Big integers (Uint*) from byte slice array with from_be_bytes

Could you please implement this without heap allocations?
also, you can add a slice_to_u64_be here and use that: https://github.com/rust-bitcoin/rust-bitcoin/blob/f4e26caa9457348c62add2263407552efe9a7e21/src/util/endian.rs#L54

also, you can add a slice_to_u64_be here and use that:

@elichai I did, but it fails to compile since this function is not being used from anywhere (i.e. triggers dead_code error):
error: function is never used: `slice_to_u64_be`
  --> src/util/endian.rs:4:9
   |
4  |         pub fn $name(slice: &[u8]) -> $type {
   |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...
55 | define_slice_to_be!(slice_to_u64_be, u64);
   | ------------------------------------------ in this macro invocation
   |
note: the lint level is defined here
  --> src/lib.rs:43:9
   |
43 | #![deny(dead_code)]
   |         ^^^^^^^^^
   = note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)

I meant that you'll use that instead of:

                let data: Vec<u64> = (0..$n_words)
                    .map(|n| {
                        let mut a = [0u8; 8];
                        a.copy_from_slice(&bytes[(n * 8)..(n * 8 + 8)]);
                        u64::from_be_bytes(a)
                    })
                    .collect();

Oh yes, true

done

Now, with https://github.com/rust-bitcoin/rust-bitcoin/pull/435 merged, it will be nice to merge this one as well before the next version release

Btw, trying to understand how these all is different from https://github.com/rust-bitcoin/bitcoin_hashes/blob/master/src/util.rs#L149-L210 and why not to move there all these functions (old and new ones)?


Add storage read syscall

Codecov Report

:exclamation: No coverage uploaded for pull request base (0.11@7611b51). Click here to learn what that means.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             0.11     #340   +/-   ##
=======================================
  Coverage        ?   94.90%
=======================================
  Files           ?       49
  Lines           ?     8496
  Branches        ?        0
=======================================
  Hits            ?     8063
  Misses          ?      433
  Partials        ?        0

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more