[RFC] Split HTTP::Client::Response in two

Context
I think it's a shame that having a compiled and type-safe language we are using the same type for non-streaming (consumed) and streaming responses:
require "http/client"

response = HTTP::Client.get("...")
response.body # okay
response.body_io # not okay

HTTP::Client.get("...") do |response|
  response.body # not okay
  response.body_io # okay
end

The "okay" and "not okay" here mean that when it's "not okay" we get a runtime error. This is just a minor thing because you'll probably notice it right away and change it to use the other. However, it can be quite confusing to understand why you get that error. And also, we could catch this at compile-time.
Proposal
I propose we have this hierarchy:

abstract HTTP::Client::Response

HTTP::Client::Response::Consumed: returned in the non-block version and has a String body.
HTTP::Client::Response::Streaming: returned in the block version and has an IO body

Some decisions to make

Should we do this? I personally think so.
Do we use the same name for the body property in both cases? That means you can always use body (nice) but the type changes depending on whether it's in a block or not. The downsides are that it can be a bit confusing, and it's also a breaking change. Alternatively we can keep it as body and body_io so it's not a breaking change.

I was having the same thought process for it actually, so I'm all for it. body and body_io always felt weird and out of place. So also 👍 to calling it body on both.
If wanted Streaming could maintain body_io as deprecated alias to body for a while. Consumed will not need body_io as that was never working for its usecases in the first place and body keeps working as expected.
The biggest breakage here will be people typing stuff as Response which in this approach will loose its body and body_io properties, so people will have to update their type declarations to continue using it.

Another idea: let both have body and body_io:

for non-streaming we have body and we could expose body_io as an IO::Memory over body.
for streaming we could make a call to body consume the body_io into a String, then replace body_io with IO::Memory over body. The problem with this is if someone consumes a bit of the IO before calling body.

But maybe the downsides are not that bad. And we could keep a single Response type.
But I still feel having two types is better.

Yeah, that seems like it's equally brittle to use than the current interface. I would prefer forcing users to always call .body.gets_to_end if they want a string explicitly (building on the always  IO option outlined above).

Yeah, this part of the API is really shady and I'm all in for improving that. I'm not to sure about having two separate types, though. Feels a bit overhead and IMO might not be that user-friendly.
There's another solution I've already mentioned in https://github.com/crystal-lang/crystal/issues/8461#issuecomment-558258251: Yield the IO as a second argument to the block in the streaming scenario.
I'd like to inspect some real examples to see how this is used in the field. That should help us discuss an optimal improvement for this API.

Hm, yielding it as second argument still seems to have potential runtime errors or weird behavior when mixing using that IO with calling Response#body at some point.

According to skimming the code available on Github, by far the most common scenario is a quick and dirty call which doesn't really need a response object and just retrieves the body as a string:
HTTP::Client.get("http://example.com").body

(more elaborate versions also include a .success? check or - rarely - some crude redirect logic)
Maybe there should be a convenience method for this which also handles some HTTP protocol context (like status code).
If we had this, the majority of use cases (the simple ones) would be dealt with. Then there would actually be little reason to have a non-streaming variant of Reponse... All more complex use case which still only want the body as string would just have to call gets_to_end.
I think that might really be a great solution. Essentially, it just means removing Response#body and providing an easy-to-use alternative for the most common use case.

Why there is one that consumes the body to a string, and not the other way around? It's not obvious at a first glance, so I think having a single behavior for both will be less confusing.
We could have:
getter body_io : IO
# Consummes the body IO
def body!
  @body_io.gets_to_end
end

Why there is one that consumes the body to a string, and not the other way around? It's not obvious at a first glance, so I think having a single behavior for both will be less confusing.
We could have:
getter body_io : IO
# Consummes the body IO
def body!
  @body_io.gets_to_end
end

That's https://github.com/crystal-lang/crystal/issues/9901#issuecomment-725412160 with different names, so 👎

If we want to do something for 1.0 I am inclined to:

keep a single response type
allow the non-streaming to have a body_io with a IO::Memory over body
make the streaming response raise on runtime when body is called

That way:

body_io is always available if needed as lower level fetch of the body, but potentially only once.
There is no ambiguity if some data from the body_io is consumed in the streamed response

If we want to tweak further the design I think is after 1.0.

The one-liner is not an good enough arguments, because we can do:
p HTTP::Client.get("http://example.com").body
p HTTP::Client.get "http://example.com", &.body_io.gets_to_end
# Perhaps even p HTTP::Client.get "http://example.com", &.body!

@j8r I don't understand your last comment. What point are you trying to make? Is it hat the yielding variant is only marginally longer and the "simple" one isn't necessary?

Yes, I mean that all could be streaming, and as user-friendly as non-streaming. Both kind of users - those needing more low level/performance and those just wanting something easy -  will be happy then.

@bcardiff Sounds good to me. That would allow a later discussion about bigger changes, possibly in conjuction with #6011.

If we want to do something for 1.0 I am inclined to:

I'm not sure if that really improves the situation, it just seems to shift the inconsistency away from the particular case that triggered this discussion, so I think I would then rather prefer to leave things as is. (Personally I think 1.0 is the perfect opportunity for a breaking change like this and starting with a clean API into the stable phase).
To me this is about getting rid of having both body and body_io, which is where all the confusion, weirdness and inconsistency stems from.

@bcardiff

make the streaming response raise on runtime when body is called

With this RFC I'm trying to avoid runtime errors and move them to compile-time errors.
@straight-shoota
If the most common case is fetching a response without a block and getting the body as a string then it shouldn't matter if we introduce two response types, because most don't have to deal with a specific type name. The body property will continue working for them. The advantage is that when you use the streaming version you will get a compile-time error if you type body instead of body_io (same for the other way around).

@asterite

If the most common case is fetching a response without a block and getting the body as a string then it shouldn't matter if we introduce two response types

But what's the point in having two separate types when one of them is rarely used and can simply be emulated using the other one?

But what's the point in having two separate types when one of them is rarely used and can simply be emulated using the other one?

Type safety.

I know that's the goal for this RFC. My suggestion to remove Response#body would be completely type safe. A second type for type safety if only required if there's a real need for a non-streaming response type. And that's questionable.

A second type for type safety is only required if there's a real need for a non-streaming response type

I think in most cases you want a non-streaming response because responses are small. However, in a few cases you'd like a streaming response, for example when downloading a large file, or when the response is meant to be streaming (like sending lines of JSON that you parse individually, and the connection is kept open).
The way we design it, for the streaming way you have to do it in a block so you can't forget to close the IO. In Go you always get an "IO" and you have to remember to close it, and we didn't like that.
So, we have two use cases and the way you use them in the same except that in one case you deal with a String, in the other you deal with an IO. We could extract that outside of the Response object, but then it would be a bit weird:
response, body = HTTP::Client.get("...")

When you talk about an HTTP response body, the body really belongs to the response, it's not a separate entity. And it's just nice to be able to pass the response to other functions.
What I'm wondering is: if we split it into two types, in which way that produces friction?

I feel like the most simple breaking change we could do right now is removing body_io, making body return IO always and let that be IO::Memory for the non streaming case. This increases API usage verbosity for the moment, but would also provide a simple baseline for adding convenience API in a non-breaking manner, say for example HTTP.get_body(...) : String.

@asterite
I totally agree to your assessment of the typical use cases. I already mentioned that in https://github.com/crystal-lang/crystal/issues/9901#issuecomment-725430190
My point is, that when you want the response as a string, chances are high you don't really care about the response object. The vast majority of use cases are literally just HTTP::Client.get("http://example.com").body.
So why not make it easier for these use cases and just return the body (as String) and not the Response with cached body?
For example, we could end up with two API methods like this:
HTTP::Client.get("http://example.com") # returns String
HTTP::Client.get("http://example.com") do |response|
  # here we consume response body io
end

If you just want a simple response body, you use the non-yielding method and get a string result. The method would also provide basic status code handling (for example raise or return nil unless successful). That's more included features for robustness and simpler to use than the current API.
If you have more advanced use case, like responding to specific status codes, handling response headers etc. you use the yielding method and get a response object (which only provides a streaming IO). If you need the body as string, you have to consume the IO.

The vast majority of use cases are literally just HTTP::Client.get("http://example.com").body

Then the vast majority of use cases are wrong. Consuming the body without checking that the status is okay? It seems like a latent bug to me.

So why not make it easier for these use cases and just return the body (as String) and not the Response with cached body?

Because a body without an HTTP status code can't be trusted.

Totally agree and that's why I mentioned that the method should check the status code and fail if it's not a success.

Ouch, I should finish reading before replying 😊

Guess what someone just asked in gitter? https://gitter.im/crystal-lang/crystal?at=5fb01edab4283c208a656378
I can't believe there's so much resistance to make this trivial change.

And I say it's trivial because I already have this implemented in my machine, I'm just waiting for an "okay" so I can send a PR.

I don't see any resistance to improving this. We just need to agree on the solution.
IMO having to separate types is completely unnecessary. We really need only a single type with IO. I think the best improvement we can do right now is to add IO::Memory for non-streaming instance. So that's pretty much what @bcardiff and @jhass have been proposing as well. There's only different opinions on whether the IO should be named body_io or body (and subsequently what to do with the string-returning method).
Just adding a fallback for body_io is the easiest change. It doesn't break anything. It doesn't improve much either, but it would fix the issue discussed on gitter.
Renaming body_io to body OTOH is a breaking change. It's easy to fix though (just insert to_s). And when it's used in a place where either IO or string do, there's no need to change any code either. So maybe it's fine. However, it's not necessarily better than renaming to body. IMO it doesn't matter whether the property ends up being named body or body_io. Staying with the latter won't break code that is already using the streaming interface. And it would allow to properly deprecate body before removing/breaking its return type.

If you take a look at newer languages like Rust, Elm... even Haskell has this... you define all possible states a type can be (algebraic data type) and then there's no way to represent an invalid state. In my proposal, when you fetch a non streaming response, you can only read it as a string. If you are streaming it, you can only treat it as an IO. Of course you can explicitly go from one to the other (build an IO memory or read everything into a string) but as it's been avoided all the time here, implicit behavior is not good.
I sincerely can't see the downside of having two types other than hypothetical situations.

A very practical downside from the API design is that IMO a non-streaming type is simply not necessary at all. So we can eventually have just a single type with streaming IO.
It means we can't have a proper fix right now. That's a benefit of the two-type solution. But the final solution to have only one type sounds much better to me.

How can you make it so that users don't forget to close the IO? If you have a solution for that, I can concede.

I'd suggest to remove the overload that returns a Response and only keep the one that yields the instance.
That would also free up the non-yielding overload  for a convenience method that directly returns String as suggested in https://github.com/crystal-lang/crystal/issues/9901#issuecomment-725430190. Note that it doesn't have to be that way. The convenience method needs extra discussion. But it would be an option.

I think we are going in circles here so I'll just stop. Sorry :-)

You're probably right.
I'll still add this to my last response because I might have been thinking to far ahead: We can obviously also keep the non-yielding overload and have it return a Response instance. That one would just not provide access to the network IO, but the buffered body via IO::Memory. So the data situation and the responsibility to close a response are exactly the same as now. What changes is just that you can't access the body as a string directly. Instead you always have to consume an IO, either memory or network depending on how the response is obtained.

Which is what I suggested in https://github.com/crystal-lang/crystal/issues/9901#issuecomment-725410164 and https://github.com/crystal-lang/crystal/issues/9901#issuecomment-725453347 already :P

And @bcardiff in https://github.com/crystal-lang/crystal/issues/9901#issuecomment-725435327. It essentially just differs on whether to stay with body_io or move to body for the streaming IO.

Oh, I really like that. @jhass I think I missed that third option, sorry.
So, to see if I understand things correctly, the idea is:

have a single HTTP::Client::Response type whose body is always an IO
In the case of a streaming response, the IO is directly connected to the socket, and it will be closed once the block ends
In the case of a non-streaming response, we read the HTTP body, close the response/connection, and set body to an IO::Memory over the string body

The nice thing about this approach is that you always deal with a response in the same way: it's an IO. The not-so-nice thing about this is that for the non-streaming version you have to do response.body.gets_to_end to get a string, which has three "problems":

it's a bit more verbose than just body: but I'm fine with
IO::Memory#to_s and IO::Memory#gets_to_end will create a duplicate string: however, we can make IO::Memory know that it was initialized from a string, and in that case return the original string. There's still one more memory allocation for the IO::Memory, but it's fine (it's just 80 extra bytes).
if the body is always an IO one could pass it down to JSON.parse or X.from_json, or X.from_yaml, and parsing an IO is slower than parsing a String: but you can always read the IO into a string if you want to avoid that performance issue.

So I think I'm 👍  for this approach. It's a breaking change, but it's probably better to do it before 1.0 is released.

Yes, that's the idea.

As mentioned previously it's easy to build helpers on top of it to get rid of the verbosity, both first class in stdlib or second class as or inside a shard.
A "read only" mode/flag  sounds good, as a generally useful performance optimization too. This way we can use IO::Memory as an IOish view over a string, even a static one (think file embedding).
If the parsing slow down due to this turns out to be a real world issue, it seems easy to have JSON and YAML detect an IO::Memory in "read only" mode and access the underlying string instead.

Great :+1: Sounds like we're pretty much agreed then. It seems body vs. body_io is still to be settled.
body is the better alternative because the _io suffix is simply unnecessary. But the change to a bit more concise name has to match the cost of a breaking change.
Staying with body_io has the benefit of not breaking any existing code using the streaming response. body returning string (for non-streaming response) could be properly deprecated and continue to work for now. So effectively, we wouldn't need to break anything immediately.
IMO it's better to go into 1.0 with a deprecated method than a broken one.

I would strongly prefer body, making body_io a deprecated alias to it is fine for me.


Implementation of max-retries.

resolves #38.

Current coverage is 92.90% (diff: 100%)

Merging #50 into master will increase coverage by 0.05%

@@             master        #50   diff @@
==========================================
  Files             8          8
  Lines           951        958     +7
  Methods         129        129
  Messages          0          0
  Branches        208        210     +2
==========================================
+ Hits            883        890     +7
  Misses           68         68
  Partials          0          0

Powered by Codecov. Last update 2a4d253...c35013d

It may be good to make NodeJS samples on toolkit show how to use this setting since it is not included in the retry config. Do you think this is necessary @jmuk?

Creating new branch since there was a big refactor since this was first used.


Changed LiKafkaSchemaRegistry to use correct id from MD5Digest

Fixed LiKafkaSchemaRegistry to use MD5Digest asString method that provides correct hex digest of data for schema id instead of toString which is not overridden over default impl

Minor fix. Self approving.


python310Packages.plugwise: 0.27.7 -> 0.27.9

Diff: https://github.com/plugwise/python-plugwise/compare/refs/tags/v0.27.7...v0.27.9
Changelog: https://github.com/plugwise/python-plugwise/releases/tag/v0.27.9
Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 223513 run on x86_64-linux 1

5 packages built:

home-assistant-component-tests.plugwise
python310Packages.plugwise
python310Packages.plugwise.dist
python311Packages.plugwise
python311Packages.plugwise.dist


Add instruction how to migrate to v0.50.0

What this PR does / why we need it
Checklist

[ ] Update changelog in CHANGELOG.md.

Trigger e2e tests

/run cluster-test-suites

Thanks Nick!


A Chinese translation

Signed-off-by: koushiro<EMAIL_ADDRESS>Changes

Add Chinese translation (includes README/_sidebar/0/1/2/3/4)
Add .gitignore file, remove .DS_Store files.

Related issue: #52

If someone finds something wrong or inaccurate in the translation, please let me know, I will be very appreciated.
如果有人发现翻译中有错误或不准确的地方，请告诉我，我会非常感激。


Decisions page not rendering

Hello Simon,
I'm experiencing a strange behavior with Structurizr Lite. I have the following folder structure:
.
├── architecture
│   ├── decisions
│   │   └── 0001-record-architecture-decisions.md
│   ├── documentation
│   │   └── 01-introduction.md
│   ├── workspace.dsl

This is the content of the files:
WORKSPACE.DSL
workspace "VAS Platform 2.x" "Architecture of the VAS Platform" {
    # Disabling implied relationships
    !impliedRelationships false

    # Structurizr specific references
    !docs documentation
    !adrs decisions

    model {
        #####################
        # Internal Entities #
        #####################
        enterprise "VAS Ecosystem" {
            ss_vasPlatform = softwareSystem "VAS Platform"
        }
    }

    views {
        ##########################################
        # Common styles for common look and feel #
        ##########################################
        #LND-01
        systemLandscape "lnd01" {
            title "LND-01: VAS Platform 2.x High Level Architecture"
            description "This diagram presents the whole echosystem of the VAS Platform from a logical point of view"
            include *
            exclude "element.tag==hidden"
            autoLayout "lr"
        }
    }
}

0001-record-architecture-decisions.md (created with ADR tool)
# 1. Record architecture decisions

Date: 2021-01-01

## Status

Accepted

## Context

We need to record the architectural decisions made on this project.

## Decision

We will use Architecture Decision Records, as [described by Michael Nygard](http://thinkrelevance.com/blog/2011/11/15/documenting-architecture-decisions).

## Consequences

See Michael Nygard's article, linked above. For a lightweight ADR toolset, see Nat Pryce's [adr-tools](https://github.com/npryce/adr-tools).

01-introduction.md
## Overview and Goals
---

The high level architecture of the system, with regards to functionalities and logical components, is illustrated in the diagram below:

![](embed:lnd01)

I start Structurizr Lite from the root with the following command:
docker run -d -it --rm -p 8080:8080 -v $PWD/architecture:/usr/local/structurizr --name structurizr structurizr/lite:latest
Then I can see the diagrams, the documentation, but I cannot see the decisions.
This is a simplified version of my model, which is presented correctly on the Cloud.
I also have the v1.x of the model (same folder structure), which works fine locally up to version 2875, but refuses to load on any version newer than that.
docker run -d -it --rm -p 8080:8080 -v $PWD/architecture:/usr/local/structurizr --name structurizr structurizr/lite:2875 ==> GOOD
docker run -d -it --rm -p 8080:8080 -v $PWD/architecture:/usr/local/structurizr --name structurizr structurizr/lite:latest ==> BAD
Funny enough, the simplified version I present here does not work at all on version 2875.
Both models are fine on the Cloud.
Thanks for helping

This should now be fixed in 2919+.

Yes, I confirm the new version works on the v2.x model.
I still have some weird behavior with the v1.x model, which I can see only if I use the version 2875 connected to port 8081. I suspect it has something to do with the JSESSIONID cookie, but wasn't able to pinpoint. Will open a new ticket when I will have more time to investigate.
Thank you


JPG image file type could not be inferred even with extension

Describe the bug
Returning the following error when I try to use serverless image handler on a jpg image stored in s3 bucket:
{"status":500,"code":"RequestTypeError","message":"The file does not have an extension and the file type could not be inferred. Please ensure that your original image is of a supported file type (jpg, png, tiff, webp, svg). Refer to the documentation for additional guidance on forming image requests."}
Please note the original image DOES have an extension.
To Reproduce
original jpg image in s3 bucket:
https://s3.ap-southeast-2.amazonaws.com/na2-cloudinary/original/melbourne-female-adult-jobs-test-x4idd447w1slvz5l3jv83x.jpg
image handler url after transformation:
https://d12km5p1xfmt50.cloudfront.net/0x300/filters:watermark(na2-cloudinary,na-logo-text.png,35p,60p,30,30)/filters:format(webp)/original/melbourne-female-adult-jobs-test-x4idd447w1slvz5l3jv83x.jpg?signature=b4e56d64936a79599670da3c1aabf7096d5d3a94ff4c4cba2582217a1b8ce826)
Expected behavior
expect the transormed image to be returned.
Please complete the following information about the solution:

[ ] Version: v6.0.0
[ ] Region: ap-southeast-2
[ ] Was the solution modified from the version published on this repository? no

Additional context
all other JPG images, running from the same bucket are working as expected.
some metadata from the failing image:
TrueColor
sRGB
APP14
Adobe RGB (1998)
ProfileDateTime 1999-06-03 00:00:00
ProfileCMMType	Adobe Systems Inc.
ProfileVersion	2.1.0
ProfileClass	Display Device Profile
ColorSpaceData	RGB
ProfileConnectionSpace	XYZ
ProfileDateTime	1999-06-03 00:00:00
ProfileFileSignature	acsp
PrimaryPlatform	Apple Computer Inc.
CMMFlags	Not Embedded, Independent
DeviceManufacturer	none
DeviceModel
DeviceAttributes	Reflective, Glossy, Positive, Color
RenderingIntent	Perceptual
ConnectionSpaceIlluminant	0.9642 1 0.82491
ProfileCreator	Adobe Systems Inc.
ProfileID	0
ProfileCopyright	Copyright 1999 Adobe Systems Incorporated
ProfileDescription	Adobe RGB (1998)

@njtmead thank you for your feedback. It looks as a bug here https://github.com/aws-solutions/serverless-image-handler/blob/b6a4f88567ba7a1f2c64978f9a2edd0051c4e938/source/image-handler/image-request.ts#L384
Your image melbourne-female-adult-jobs-test-x4idd447w1slvz5l3jv83x.jpg is definitely JPEG, but has the signature ffd8 ffe2 we don't expected in the method inferImageType.
See your image signature below:
00000000: ffd8 ffe2 0240 4943 435f 5052 4f46 494c  .....@ICC_PROFIL
00000010: 4500 0101 0000 0230 4144 4245 0210 0000  E......0ADBE....
00000020: 6d6e 7472 5247 4220 5859 5a20 07cf 0006  mntrRGB XYZ ....
00000030: 0003 0000 0000 0000 6163 7370 4150 504c  ........acspAPPL

As a workaround you can do the following steps in the AWS S3 Console:

select melbourne-female-adult-jobs-test-x4idd447w1slvz5l3jv83x.jpg
select "Actions" -> "Edit metadata"
in the "Metadata" section set "Content-Type" to image/jpeg
save changes

After that you might see the same error, because it was cached in the CloudFront. To verify the changes you can either invalidate the CloudFront distribution(for example for /* path) or use different parameters(for example different resize values).
We will add it to our backlog.

This has been added to the backlog and will be addressed in a future release

Hello @fisenkodv @dougtoppin
Any update on this issue? I'm facing the same error, thank you!

Hi @mrb,
Could you create a new issue with some more information so we can try and reproduce it on our end?
Thank you,
Simon


[FLOC-4401] enable persistent connections in flocker client code

Two things going on here:

There persistent keyword only exists on treq.request and friends, not treq.HTTPClient.request and friends. Unfortunately treq.HTTPClient.request takes kwargs so it never tells you if you are passing it an argument it's never heard of, instead it just silently does nothing with the argument.
treq.HTTPClient.request by-default does not use persistent connections.

These two facts combine to show us that we had a lot of persistent=False lines in our code that could just be deleted.
It does not seem that treq today offers a convenient interface to use persistent connections (despite the fact that it appears to). Thus, we just remove all references to persistent connections, and hope the default stays such that persistent connections are not used.
After that bit of cleanup, I went about modifying our code that creates treq HTTPClients so that it could create ones that had connection pools with persistence, and the appropriate plumbing to (A) test that the object created was at least trying to be persistent and (B) actually use persistent connections from our benchmarking scripts.
This is mostly done so that we can spin up large clusters and launch many many containers without spending lots of time generating SSH keys and negotiating new connections.  If SwissCom is having troubles with the CPU usage of the control node, we can help them change their flocker client code to also use persistent connections.

Thanks @sarum90
I've commented (above) that I didn't understand the behaviour and the number of connections I was seeing when I tested the FlockerClient with persistent_connections=True.
Having read the code, it seems that HTTPConnectionPool can container a mixture of non-persistent connections and up to maxPersistentConnections persistent connections.
There seem to be a bunch of bugs and confusion around the HTTPConnectionPool and persistent connections:

https://twistedmatrix.com/trac/search?q=persistent&noquickjump=1&ticket=on

I also tried to see, by profiling on the client side, whether this does indeed reduce the time spent in the SSL handshake.
As far as I could tell it doesn't. Perhaps because the HTTPConnectionPool doesn't actually limit the number of connections. Instead what seems to happen is that you create as many connections as there are outstanding HTTP requests.
I also think that I may have got confused and mislead you about persistent connection being the answer to the SSL handshake CPU problems.
I think I was mixing up persistent connections with HTTP pipelining, which is where you can send multiple HTTP requests down the same connection without waiting for responses to previous requests. That feature seems not to be implemented in Twisted web so far and may never be. Instead I think people are moving to HTTP2 multiplexing.
Anyway, you should double check whether this does have the desired effect and merge if your convinced or test it first on a benchmark cluster to verify the CPU savings.

I'll probably.... just remove all of the fake persistent args and get rid of the rest of this PR?
no sense doing it if it didn't work. (I assumed it did... kind of sad at treq for not actually making this work). I'll send this out for re-review once I've reduced this PR.

Aha, I think this has the answer:

https://github.com/twisted/treq/issues/71#issuecomment-53835158

So the connection pool doesn't limit the total connections.
It'd be interesting to see whether the combination of persistent=True and increasing the pool size (and a larger number of idle connections) would lower the number of new connections.
Also possibly limit the parallelism in the client library.
But pipelining (if it were supported) would be even more effective I think and allow the benchmarks to run with even SSL handshakes.
I've merged forward to force some new builds...after upgrading treq and twisted recently.

Looks good @sarum90
So the upshot of this will be:

Places where we use treq directly will use persistent connections, by way of the global treq persistent connection pool,
...except where we use treq.get/post(...., persistent=False), as in the testtools, which start and then query a test HTTP server.
And the Flocker API client bypasses  the treq.api, instead instantiating a treq.client.HTTPClient and supplying it with a custom Agent without specifying an HTTPConnectionPool ...which will result in a new non-persistent connection pool for each request.

I guess I'm a bit confused why your original code (reverted in 64beddaf3d0514ff081387bb3de3c5b1d9bc13cd) didn't seem to work.
Having dug into the t.w.Agent code, it looks correct.
I wonder if I was using it correctly?
Anyway. The cleanups here make sense, but we should probably update the PR title before merging and this PR shouldn't close FLOC-4401.
Please merge

Oh, it looks like you updated the issue title. So we can close it.

Thanks for the review. Merging.


Can't pull student repos

Describe the bug
I cannot pull updates in students assignment using gh classroom pull student-repos -a ... command. It says:
failed to run gh: can't sync because there are diverging changes; use `--force` to overwrite the destination branch
. error: exit status 1

however, there are no diverging changes. It also does not work when there are no updates, as shown here:
$ gh classroom clone student-repos
? Select a classroom: XXXX
? Select an assignment: YYYY
Creating directory:  /ZZZZ-submissions
Cloning into: /ZZZZ-submissions/ZZZ-AAA
Cloning into: /ZZZZ-submissions/ZZZ-BBB
Cloned 2 repos.

$ gh classroom pull student-repos
? Select a classroom: XXXX
? Select an assignment: YYYY
Pulling repo: /ZZZZ-submissions/ZZZ-AAA
failed to run gh: can't sync because there are diverging changes; use `--force` to overwrite the destination branch
. error: exit status 1
Pulling repo: /ZZZZ-submissions/ZZZ-BBBB
failed to run gh: can't sync because there are diverging changes; use `--force` to overwrite the destination branch
. error: exit status 1

$ gh --version
gh version 2.52.0 (2024-07-15)
https://github.com/cli/cli/releases/tag/v2.52.0
$ gh extension list
NAME          REPO                 VERSION
gh act        nektos/gh-act        v0.2.55
gh classroom  github/gh-classroom  v0.1.14

Steps to reproduce the behavior

Clone students repositories using gh classroom clone ....
(Optional) Modify the reference repository and request Pull Request, accept it.
Try to pull repository again

Expected vs actual behaviour
I would expect all repositories to be updated. Now I need to update them manually, which works ok (no diverging changes).
Logs
Paste the activity from your command line. Redact if needed.
$GH_DEBUG=true gh classroom pull student-repos -a AAAAAA
* Request at 2024-07-23 13:35:52.177507471 +0200 CEST m=+0.028484741
* Request to https://api.github.com/assignments/AAAAAA
* Request took 414.42728ms
Pulling repo: XXXXXX
failed to run gh: [git remote -v]
[git config --get-regexp ^remote\..*\.gh-resolved$]
[git remote -v]
[git config --get-regexp ^remote\..*\.gh-resolved$]
* Request at 2024-07-23 13:35:52.616066256 +0200 CEST m=+0.020500993
* Request to https://api.github.com/graphql
* Request took 295.087327ms
[git -c credential.helper= -c credential.helper=!"/usr/bin/gh" auth git-credential fetch -q upstream refs/heads/master]
[git rev-parse --verify refs/heads/master]
[git merge-base --is-ancestor master FETCH_HEAD]
can't sync because there are diverging changes; use `--force` to overwrite the destination branch
. error: exit status 1

The command was executed in the same directory where gh classroom clone student-repos --all was executed before.

Thanks for opening this issue, rlalik. I'm also experiencing the same issues for Github Classroom.

same here, same error message ...Easy to reproduce, just clone a repo or an assignment and try to pull :
gh classroom clone student-repos
gh classroom pull student-repos

Ditto. I'm having the same issue.

I'm glad I still have my old Python scripts... they are compatible with gh classroom but not as convenient - I have to manually add projects to a config class.txt file where 'class' is the name of a class and download the corresponding roster CSVs.

The get_repos.txt file describes the usage.
I've included a sample class config file, 4u.txt, for a formatting example
The script expects all your class projects to be in the same parent directory (described in your class config file) and for your classroom.csv file to be in that directory as well.
Also works with group projects and the appropriate csv file for the groups.
If you don't use ~\Documents or ~/documents as your base directory, you can change it in the common_ui.py file.

I just dusted it off and it ran as expected on the previously gh classroom cloned repos.
These are personal scripts and I haven't added a licence. Use at your own risk and do whatever you like with them if you do.
get_repos.zip

Could it have something to do with fetching upstream (starter repo) as well as origin during pulls? For the students' work we only need the updates to origin unless I am missing something.


no more missing constants for LWRP class

http://manhattan.ci.chef.co/job/chef-trigger-ad_hoc/28/downstreambuildview/
➜  test-custom-resource-bug kitchen converge original-chef-12-9-41
-----> Starting Kitchen (v1.7.3)
-----> Creating <original-chef-12-9-41>...
       Bringing machine 'default' up with 'virtualbox' provider...
       ==> default: Importing base box 'bento/ubuntu-14.04'...
       ==> default: Matching MAC address for NAT networking...
       ==> default: Checking if box 'bento/ubuntu-14.04' is up to date...
       ==> default: A newer version of the box 'bento/ubuntu-14.04' is available! You currently
       ==> default: have version '2.2.5'. The latest is version '2.2.7'. Run
       ==> default: `vagrant box update` to update.
       ==> default: Setting the name of the VM: kitchen-test-custom-resource-bug-original-chef-12-9-41_default_1465241273190_44789
       ==> default: Fixed port collision for 22 => 2222. Now on port 2202.
       ==> default: Clearing any previously set network interfaces...
       ==> default: Preparing network interfaces based on configuration...
       ==> default: Clearing any previously set network interfaces...
       ==> default: Preparing network interfaces based on configuration...
           default: Adapter 1: nat
       ==> default: Forwarding ports...
           default: 22 => 2202 (adapter 1)
       ==> default: Booting VM...
       ==> default: Waiting for machine to boot. This may take a few minutes...
           default: SSH address: <IP_ADDRESS>:2202
           default: SSH username: vagrant
           default: SSH auth method: private key
           default: Warning: Remote connection disconnect. Retrying...
           default: Warning: Remote connection disconnect. Retrying...
           default:
           default: Vagrant insecure key detected. Vagrant will automatically replace
           default: this with a newly generated keypair for better security.
           default:
           default: Inserting generated public key within guest...
           default: Removing insecure key from the guest if it's present...
           default: Key inserted! Disconnecting and reconnecting using new SSH key...
       ==> default: Machine booted and ready!
       ==> default: Checking for guest additions in VM...
       ==> default: Setting hostname...
       ==> default: Machine not provisioned because `--no-provision` is specified.
       [SSH] Established
       Vagrant instance <original-chef-12-9-41> created.
       Finished creating <original-chef-12-9-41> (0m38.05s).
-----> Converging <original-chef-12-9-41>...
       Preparing files for transfer
       Preparing dna.json
       Resolving cookbook dependencies with Berkshelf 4.3.3...
       Removing non-cookbook files before transfer
       Preparing validation.pem
       Preparing client.rb
-----> Installing Chef Omnibus (install only if missing)
       Downloading https://omnitruck.chef.io/current/install.sh to file /tmp/install.sh
       Trying wget...
       Download complete.
       ubuntu 14.04 x86_64
       Getting information for chef current  for ubuntu...
       downloading https://omnitruck.chef.io/current/chef/metadata?v=&p=ubuntu&pv=14.04&m=x86_64
         to file /tmp/install.sh.1376/metadata.txt
       trying wget...
       sha1     bc4a6642d6093b54de856da31f0651a56b0fdc8e
       sha256   f1cf5d0f6dd12d2d2296ec6d8dbb16363f8541f5c15298cafa70e65ff2b5a22f
       url      https://packages.chef.io/current/ubuntu/14.04/chef_12.11.18-1_amd64.deb
       version  12.11.18
       downloaded metadata file looks valid...
       downloading https://packages.chef.io/current/ubuntu/14.04/chef_12.11.18-1_amd64.deb
         to file /tmp/install.sh.1376/chef_12.11.18-1_amd64.deb
       trying wget...
       Comparing checksum with sha256sum...

         If you are installing
       on production servers via an automated process this is DANGEROUS and you will
       be upgraded without warning on new releases, even to new major releases.
       Letting the version float is only appropriate in desktop, test, development or
       CI/CD environments.

       WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING

       Installing chef
       installing with dpkg...
       Selecting previously unselected package chef.
(Reading database ... 38789 files and directories currently installed.)
       Preparing to unpack .../chef_12.11.18-1_amd64.deb ...
       Unpacking chef (12.11.18-1) ...
       Setting up chef (12.11.18-1) ...
       Thank you for installing Chef!
       Transferring files to <original-chef-12-9-41>
       Successfully installed appbundler-0.9.0
       Parsing documentation for appbundler-0.9.0
       Installing ri documentation for appbundler-0.9.0
       Done installing documentation for appbundler after 0 seconds
Fetching: appbundle-updater-0.2.12.gem (100%)
       Successfully installed appbundle-updater-0.2.12
       Parsing documentation for appbundle-updater-0.2.12
       Installing ri documentation for appbundle-updater-0.2.12
       Done installing documentation for appbundle-updater after 0 seconds
       2 gems installed
-----> Cleaning chef checkout
-----> Creating /opt/chef/embedded/apps directory
-----> Installing Packages
           running: apt-get -y update
           running: apt-get -q -y install build-essential git
-----> Extracting chef from https://github.com/chef/chef/archive/praj/FLOW-927/s3cmd.tar.gz
       Unkown tar entry: pax_global_header type: g.
-----> Installing dependencies
           running: /opt/chef/embedded/bin/ruby /opt/chef/embedded/bin/bundle install --without server docgen test development
-----> Installing gem
           running: /opt/chef/embedded/bin/ruby /opt/chef/embedded/bin/bundle exec /opt/chef/embedded/bin/rake install
-----> Updating appbundler binstubs for chef
           running: /opt/chef/embedded/bin/ruby /opt/chef/embedded/bin/appbundler /opt/chef/embedded/apps/chef /opt/chef/bin
-----> Finished!
       Starting Chef Client, version 12.11.18
       Creating a new client identity for original-chef-12-9-41 using the validator key.
       resolving cookbooks for run list: ["s3"]
       Synchronizing Cookbooks:
         - s3 (0.1.0)
       Installing Cookbook Gems:
       Compiling Cookbooks...
       AptRepository
       AptUpdate
       Batch
       Breakpoint
       ConvergeActions
       CookbookFile
       Cron
       DEPRECATED_OPTIONS
       Deploy
       DeprecatedLWRPClass
       Directory
       DscResource
       DscScript
       Env
       ErlCall
       Execute
       File
       Git
       Group
       HttpRequest
       Ifconfig
       InlineResources
       LWRPBase
       Launchd
       Link
       Log
       Mdadm
       Mount
       Noop
       Ohai
       OsxProfile
       Package
       PlatformDependentValue
       PlatformFamilyDependentValue
       PowershellScript
       Reboot
       RegistryKey
       RemoteDirectory
       RemoteFile
       ResourceRequirements
       Route
       RubyBlock
       Script
       Service
       Subversion
       SystemdUnit
       Template
       TemplateFinder
       User
       WhyrunSafeRubyBlock
       WindowsScript
       Converging 1 resources
       Recipe: s3::default
         * deploy_revision[/root/lessons] action nothing (skipped due to action :nothing)

       Running handlers:
       Running handlers complete

       Deprecated features used!
         Using an LWRP provider by its name (S3Scm) directly is no longer supported in Chef 12 and will be removed.  Use Chef::ProviderResolver.new(node, resource, action) instead. at 1 location:
           - /tmp/kitchen/cache/cookbooks/s3/recipes/default.rb:4:in `block in from_file'

       Chef Client finished, 0/1 resources updated in 01 seconds
       Finished converging <original-chef-12-9-41> (1m44.31s).
-----> Kitchen is finished. (2m23.02s)
➜  test-custom-resource-bug

That was already a class method I think, so this is probably creating a global. If that's the goal (and you'll need a hell of a justification if thats the case) it should be $deprecated_constants to make that clear.

I think this is a viable fix @coderanger. This PR aims to fix a backwards compatibility issue when using custom providers written under chef ( < 12.4) and running them under version 12.4 and greater. Lets say you have a scm.rb in your mycookbook provider folder and you have a recipe using this:
deploy_revision "/blah/blah" do
  scm_provider "mycookbook_scm"
  action :nothing
end

The problem today is that the @deprecated_constants hash is an instance variable (defined in a module) created and fed from an instance of the base chef::Provider but it is accessed from a derrived provider class Chef::Provider::Deploy and therefore it is a completely different hash.
So using a class variable ensures that the variable fed the deprecated constants and later accessed is the same object. Using $deprecated_constants would be a true global and more evil than @@deprecated_constants

@PrajaktaPurohit I think if you rebase against master you will get the fixes to the dsc_script merged earlier today  and perhaps turn this green.

@mwrock My issue is that the module gets used via extend instead of include, I thought the semantics of that make it "class level" at Class.

gotcha @coderanger. I think technically @deprecated_constants is class level like you say but the variable is written to as a class variable of Chef::Provider and then accessed from the class Chef::Provider::Deploy and therefore separate classes.

Right, I think the change we want is to call Provider.deprecated_constants instead of self.deprecated_constants :)

perhaps. currently the @deprecated_constants variable is wrapped via a private deprecated_constants method. We'd have to make that public and have the const_missing reference Provider.deprecated_constants which seems odd to do inside of the module that defines deprecated_constants.
Note this is pushing the envelope of my familiarity with ruby internals and usage patterns which is a super awesome learning experience but excuse me as I spew forth words that come from shallow knowledge.

This (CoreClass.accessor) is a pretty common pattern, we do it in a lot of other places for registration stuffs. There is almost no time when you actually want a @@classvar, they are super error prone :)

Ping @lamont-granquist @jkeiser to check that I'm not totally mis-remembering everything, which is never impossible.

I think another rebase should green-up travis now @PrajaktaPurohit

So the thing that feels weird to me now is the Chef::Provider.deprecated_constants inside the module code. This may be largely aesthetic but having this code exist as a module and then coupling it to the class seems like a bad move. If the choice is between that and the use of @@deprecated_constants I lean toward @@deprecated_constants.
Perhaps another route would be to have the deprecated_constant related methods pulled out of the module alltogether and placed directly in the class. This mirrors what is done in resource.rb. I have not tried this or scrutinized the nuances but if that "works" I think it may be the most straight forward path.
Thoughts @coderanger ?

@mwrock We've gone pretty far in removing @@classvars from the codebase. Adding more seems like moving in the wrong direction given there is no actual need for it here and it's confusing enough that multiple Ruby devs are unsure of the precise semantics when combined with extend :)

:+1: on the recent change to whack the module which gets rid of the @@classvar and makes resource.rb and provider.rb follow a similar pattern which is less confusing (at least to me).

Those new methods need very clear comments on them so we know what to remove in Chef 13, that's why they were in a module in the first place. 👎

If we do put them in a module lets do it on both sides.
So maybe I'm just being overly anal on the use of the class from inside the module where is is used. Anyone else? I don't want to over think this.

The reason for the module is to make life easier in Chef 13, so we can easily remove the code. @jkeiser has used that pattern in a lot of places now.

that sounds reasonable and I'd assume we'd want to the removal on both sides.

@PrajaktaPurohit put the module back and replicated in resource.rb as well. I'm still :+1:

@mwrock already done!

oh sorry @PrajaktaPurohit , that wasn't intended to sound like a command(yuck) but simply restating what you had done. Thanks so much!

One minor note about exception types if we want to care about this, if we don't do it now it will probably get fixed later when we enable that cop :) 👍 🚢


Add "logout" key

Adding the "Logout" key for the menu translation

Needed for the pull request #116 of the frontend project


SQS Event Fails to trigger job with message: null exception

Hi,
I am trying to setup this plugin to trigger a build when messages are pushed to a AWS SQS queue.
Every time I push a message to the queue, i see the following in my jenkins log file and do not get the job triggered.

16-Jul-2018 22:31:48.145 INFO [Thread-15] io.relution.jenkins.scmsqs.logging.Log.write 000068 Got message with subject: null

16-Jul-2018 22:31:48.145 WARNING [Thread-15] io.relution.jenkins.scmsqs.logging.Log.write 000068 Message contains no text

16-Jul-2018 22:31:48.145 INFO [Thread-15] io.relution.jenkins.scmsqs.logging.Log.write 000068 Test if any event matches job test-sqs

16-Jul-2018 22:31:48.145 INFO [Thread-15] io.relution.jenkins.scmsqs.logging.Log.write 000068 Event(s) did not match job.

What should the queue message and the message attribute contain in order for the plugin to trigger a build ?

I had this issue since i was not using the latest version of the plugin. I downloaded the latest release, build it and deployed the plugin. Work fine now :)


Create manifests for OpenSUSE

We are currently processing in provision-dashboard.sh the manifests files installed in CaaSP. However, these manifests

are only installed in CaaSP (ie, the caasp-container-manifests.rpm is not available for OpenSUSE)
they use images based on SLE

We should support OpenSUSE, and that means that we should

get those manifests installed in /usr/share/caasp-container-manifests somehow

we want to use https://github.com/kubic-project/caasp-container-manifests as the only
source of truth for these manifests, so we don't want to keep a copy in this repo

replace paths (as we currently do) in the provision-dashboard.sh, but also some images names by the OpenSUSE equivalents.

as @inercia said, it might prove to be difficult. @mjura i didn't think this would be challenging enough for two persons to work on it, but if you have any issues and there's anything i can do to help, let me know, i'd be happy to.


Enable resizing comment window

Hi, thanks for the brilliant extension!
Would it be possible to enable resizing of the "add comment" window (as the web version allows) to be able to see it in full?

The comment input box will now grow, up to a max, if you add more lines.


Input table not capturing UserId

When user submit Input into prompr, data captured into Input table but missing the UserId foreign key.
Essensial issue for training the OpenAi

Generated JWT successfully contains the UserID in client local storage. Currently analyzing server JWT intake and destructuring.


Do not preload response content

The question
I would like to not load the response body, unless I get resp code == 200.
Explaination: I'm limited proxies, and I would like to know if I can only load reponse body if needed. Sometimes I face 403 (CloudFlare) and the response is quite big, which might lead to additional expenses.
My only idea so far is to use stream=True, but I want to make sure if it's gonna work the way I described above.
Documentation suggesion
That's a separate one, but I couldn't find wether I can use socks4/5 proxies or not in the docs.
Thanks in advance!

I don't think so, that is not how a proxy server works. Anyway, the only way to know is to experiment by yourself. But I highly doubt it.

I couldn't find wether I can use socks4/5 proxies or not in the docs.

The way you use proxies in requests, nothing special here.

Can you use OPTIONS requests?


Create logging infrastructure

Create some foundations for logging. Meaning some rudimentary log categories, some macros, and a standard for when/what to log.

Not sure this will be needed outside of log categories. Esp not this early.


[dart2js] Simplify output file name generation

OutputProvder.createOutputSink has a somewhat arcane interface
  /// Returns an [OutputSink] that will serve as compiler output for the given
  /// component.
  ///
  /// Components are identified by [name], [extension], and [type]. By
  /// convention, the empty string `""` will represent the main output of the
  /// provided [type]. [name] and [extension] are otherwise suggestive.
  // TODO(johnniwinther): Replace [name] and [extension] with something like
  // [id] and [uri].
  OutputSink createOutputSink(String name, String extension, OutputType type);

  /// Returns an [BinaryOutputSink] that will serve as compiler output for the
  /// given URI.
  BinaryOutputSink createBinarySink(Uri uri);

There are several things that are a bit awkward.

It is not possible to determine the actual name of an output when the OutputSink writes to a file (if the sink in in-memory, the three components can be a key).

Actual file-names don't seem to honour the extension. 'js' is passed for the main unit, but if the command-line has --out=dir/foo, the output files are dir/foo and dir/foo.map, not dir/foo.js and dir/foo.js.map.

The use of the name and extension is inconsistent between the main unit ("") and the part files:

createOutputSink("o.js_2", "part.js", OutputType.jsPart)
createOutputSink("o.js_2.part", "js.map", OutputType.sourceMap)
createOutputSink("", "js", OutputType.js)
createOutputSink("", "js.map", OutputType.sourceMap)

The fact that we have so many OutputType values is a smell - e.g. why do we need OutputType.js and OutputType.jsPart?

deferredPartFileName tries to guess the name. It is used from several places (js_emitter and dump-info) and if we changed the naming scheme, these would become inconsistent - the extension is written in several places. dump-info guesses the name of the main unit.

dart2js --out=foo/ will crash when it tries to write the main unit. It should probably try to compile to out.js in the given directory.

I is pretty weird that the part files have .js in the middle, e.g. out.js_3.part.js. I would have though something like out_part3.js or out.part_2.js would be more natural.

@johnniwinther You have added several TODOs in this area - did you have a vision for how this code might be more straight-forward?

Sound great to simplify here.
That said, it may not be worth doing a change like (7). Internal customers have subtle infra dependencies on the file name pattern (e.g. for filters, matchers, deobfsucation services), so it would require a careful rollout.


NIMBUS-712 :: Improve startup time

Description
After implementing caching for the EventHandlerConfigFactory, startup time was significantly reduced (in my case from ~120s to ~40s).
Overview of Changes

Added caching to EventHandlerConfigFactory

Type of Change

[X] New feature
[ ] Bug fix
[ ] Refactor/Technical Debt
[ ] Documentation Update
[ ] Test case change
[ ] This change requires a documentation update
[ ] Other

Test Details
N/A
Additional Notes
N/A

You did it @kropptrevor!
Thank you for signing the Anthem Opensource Software CLA.


:one::one::one: add 111 call out

@ajthomascouk Can you add/extend a test to check for the message on the various result page configurations please?

The spacing on the message needs a bit of a tweak for a results page with only nearby results. See the screenshot.

At the moment this URL will show the same as the image but that will change the next time it is restarted. https://connecting-to-services-pr-417.dev.beta.nhschoices.net/find-a-pharmacy/results?location=tr21

Tests added and spacing sorted

@ajthomascouk I can't see the changes for the spacing in https://github.com/nhsuk/connecting-to-services/pull/417/commits/11ffc67b96e9eeb73dc09633aa6a8a421df051b8


Set timeout

Setting the XHR timeout (in milliseconds) can only be done after the open() method in versions of Internet Explorer. Thus, the workaround approach to creating your own XHR to pass in for opts.xmlHttpRequest doesn't work since open is called afterwards.
This is also cleaner than creating a custom XHR for each time you'd like to xr.get with a more aggressive timeout.

If it's preferred, I can rebase my PR on top of Jared's changes in #26 - as it'll create a merge conflict otherwise since we'd end up in the same place.


How to smoothly interpolate between CPPN parameterizations?

This is less of an issue and more of a discussion, so please tag appropriately. I took the Lucid library and set some of the channel renderings to music:
video: https://www.youtube.com/watch?v=qPi5UPAlwl8
repo: https://github.com/thoppe/DeepMDMA
I'm really pleased with the result (and kudos for such an awesome library that's easy to hack on!). My question involves the transition between the parameterizations. In my video (and in the last video of the this notebook especially), the CPPN blurs out between each image. This happens even if you train a hundred different models all on the same channel, due to the different local minima found from the random initializations.
To mitigate this somewhat, I started with a source parameterization P(t=0), slowly moved the optimization, and retrained starting from the prior timestep initialization. This definitely helps, but it takes forever to train the model iteratively like this! Is there a way to keep the CPPN in a visual appealing part of the space when interpolating?

If you have a question about Lucid’s behavior and want to discuss it with other users, also consider the  #proj-lucid channel on the Distill Community Slack Workspace which you may join here.
Lucid is alpha-status research code, and we appreciate your understanding that this can sometimes mean we will not have the resources to resolve your specific issue—even if it may be an entirely reasonable request in a more mature codebase. Thank you! 🤗


Bootstrap 3 with Laravel

So I'm using laravel as a back end and I would like to use this plugin but I need my image/video to come from the backend instead of hard-coded url so what I did is the following
                       Launch
my controller returns the url properly but the problem is that instead of displaying an image/video, it just displays the url string.
Am I missing something that I should be doin ?

For support you need to post your full example.


Added KinematicCollision

This was done so that the output of KinematicBody's move() and move_and_slide() would be the same. In addition, by eliminating the use of Dictionary, new users can more easily discover the available output parameters in the documentation.

I'm not really sure about this, instancing objects is not that cheap in
Godot and you might want to use a lot of these in a lot of objects, so it's
probably better to not go this way for performance.. even though these
functions are pretty heavy performance wise already..
On Fri, Jul 21, 2017 at 1:29 PM, Lee Zher Huei<EMAIL_ADDRESS>wrote:

This was done so that the output of KinematicBody's move and
move_and_slide would be the same. In addition, by eliminating the use of
Dictionary, new users can more easily discover the available output
parameters in the documentation.

You can view, comment on, or merge this pull request online at:
https://github.com/godotengine/godot/pull/9751
Commit Summary

Added KinematicCollision

File Changes

M scene/3d/physics_body.cpp
https://github.com/godotengine/godot/pull/9751/files#diff-0 (246)
M scene/3d/physics_body.h
https://github.com/godotengine/godot/pull/9751/files#diff-1 (68)
M scene/register_scene_types.cpp
https://github.com/godotengine/godot/pull/9751/files#diff-2 (1)

Patch Links:

https://github.com/godotengine/godot/pull/9751.patch
https://github.com/godotengine/godot/pull/9751.diff

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/pull/9751, or mute the thread
https://github.com/notifications/unsubscribe-auth/AF-Z2zESJvQJ3gi40wOvXMfVlcwDhyqjks5sQNHUgaJpZM4Ofn2T
.

Yeah, agreed. I realised there was a simpler (and possibly more elegant) solution. I've created a separate pull request for it at #9752
I'll close this one if you think that one is better.


:Gbrowse command has been removed

Vim bootstrap version(number in first line): 2022-12-07 02:54:26
Operating system: Arch
Vim/Nvim version: NVIM v0.8.0
Languages: vimscript

Description
In the generated script there is a mapping in order to open current line on GitHub:
"" Open current line on GitHub
nnoremap <Leader>o :.Gbrowse<CR>

But when using it there is an error message:
":Gbrowse has been removed in favor of :GBrowse"

Closed by #432


When a cluster component is disabled, the component image is not pulled

Signed-off-by: Fish-pro<EMAIL_ADDRESS>
What type of PR is this?
/kind cleanup
What this PR does / why we need it:
When a component is disabled, the component image is not pulled
Which issue(s) this PR fixes:

Fixes #
Special notes for your reviewer:
Does this PR introduce a user-facing change?

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

/ok-to-test


Fix etcd health check so that all results are reported and multiple

etcd instances running on the same node will all be checked.
Fixes #702.

This is now necessary due to #928.

So this passed the first time and then failed twice.

[aws] failed: [localhost] (item=1) => {"attempts": 30, "changed": true, "cmd": ["ssh", "-o", "StrictHostKeyChecking=no", "-o", "UserKnownHostsFile=/dev/null", "-F", "/home/jenkins/workspace/K2_PR-931-LK7VVOZCYFVJFS5IIQCVH77NCGSRTN47QGG3RXV2JS27WSVU4QBQ/cluster/aws//pr-931-3/ssh_config", "etcd-1", "etcdctl", "--ca-file=/etc/etcd/ssl/client-ca.pem", "--cert-file=/etc/etcd/ssl/client.pem", "--key-file=/etc/etcd/ssl/client-key.pem", "--endpoints=https://<IP_ADDRESS>:2379,https://<IP_ADDRESS>:4001", "cluster-health"], "delta": "0:00:02.562528", "end": "2017-11-05 02:38:09.994119", "failed": true, "idx": "1", "rc": 4, "start": "2017-11-05 02:38:07.431591", "stderr": "Warning: Permanently added 'ec2-18-221-97-241.us-east-2.compute.amazonaws.com,<IP_ADDRESS>' (ECDSA) to the list of known hosts.\r\nError:  client: etcd cluster is unavailable or misconfigured\nerror #0: client: endpoint https://<IP_ADDRESS>:4001 exceeded header timeout\nerror #1: client: endpoint https://<IP_ADDRESS>:2379 exceeded header timeout", "stderr_lines": ["Warning: Permanently added 'ec2-18-221-97-241.us-east-2.compute.amazonaws.com,<IP_ADDRESS>' (ECDSA) to the list of known hosts.", "Error:  client: etcd cluster is unavailable or misconfigured", "error #0: client: endpoint https://<IP_ADDRESS>:4001 exceeded header timeout", "error #1: client: endpoint https://<IP_ADDRESS>:2379 exceeded header timeout"], "stdout": "cluster may be unhealthy: failed to list members", "stdout_lines": ["cluster may be unhealthy: failed to list members"]}

I will try reproducing the problem locally, and then log in to one of the nodes during a CI if I can not.

I am beginning to think that the problems I having reliably forming etcd clusters may be related to the recent change to use etcd 3.0.17 (instead of 3.2.5 and 3.1.0 before it). Debugging this may be easier if we complete samsung-cnct/kraken-lib#825 first.

One reason the etcd health checks are failing is that the ssh_config is incomplete. There was an issue about this sometime ago: https://github.com/samsung-cnct/kraken-lib/issues/722

LGTM, going to merge.


Reorganized ML module documentation

fixed one BiB record
moved algorithms overview to separate page
added docs for some enumerations
fixed some small documentation errors

:+1:


GLTF Exporter

Is your feature request related to a problem? Please describe.
The engine i'm using supports GLTF files natively and allows me to do one singular click to create collision across the entire file, plus they embed all the textures.
Describe the solution you'd like
I'd love to see the option to export the model in the GLTF format while embedding all textures.
Describe alternatives you've considered
For now I think i'd keep making maps in Trenchbroom.

Thanks for your feedback!
I'll add it to my TODO list:

rewrite the existing OBJ exporter to use Assimp for the job, so models can be saved in any format that Assimp can output.

Thank you for testing Elit3D. As for me, I've been rewriting the project from scratch in Godot for a while now. I hope to have a stable version soon that contains the minimum to work well. The GLTF model import is already functional and now I'm working on the GLTF and OBJ export. Once this is finished and stable, I will update the repository with the new project.
Thanks for your time!


Dont duplicate CHECK constraints and DEFAULTs when altering column type

Do not attempt to duplicate CHECK constraints and column DEFAULTs as part of an 'alter column' operation that changes a column type. CHECKs and DEFAULTs may not be compatible with the new type of the column.
In the case where a DEFAULT or CHECK should be retained, it is now possible to restate it as part of the same 'alter column' operation (as of https://github.com/xataio/pgroll/pull/338 which allows multiple changes as part of one 'alter column' operation).
Fixes https://github.com/xataio/pgroll/issues/348

I'm wondering if this will be unexpected for compatible types, ie when I cast from integer to int8 or float 🤔

That was my concern too.
If we had done this before supporting multiple sub-operations in one alter column statement it would probably not have been the right decision.
Now however if you change type to a compatible type you can recreate the DEFAULT and CHECK in the same alter as the type change:
{
  "name": "35_alter_column_multiple",
  "operations": [
    {
      "alter_column": {
        "table": "users",
        "column": "age",
        "type": "bigint",
        "default": "0",
        "check": {
          "name": "age_check",
          "constraint": "age > 21"
        },
        "up": "...",
        "down": "..."
      }
    }
  ]
}

which mitigates the problem, although it might still be unexpected that you have to do this.
Ideally we would be to try to preserve checks and defaults iff they are compatible with the new type.

We should preserve defaults and check constraints when they are compatible.
Putting this back into draft in order to do that.

Updated this to preserve DEFAULT and CHECK constraints when they are compatible, and ignore errors on duplication of DEFAULT and CHECK constraints when the error code suggests the failure is a result of column type change.


Testings Tasks

How do you suggest I might write unit tests against the tasks I create?  And other code calling the tasks.
I'm thinking I'd need a way for all the tasks to  be run immediately and ignore the delay parameter during unit testing?

Here's an example:
from django.test import TestCase

from background_task import background
from background_task.tasks import tasks

completed_tasks = []

@background()
def my_background_task(message):
    completed_tasks.append(message)

class MyBackgroundTaskTestCase(TestCase):
    def test_run_task(self):
        my_background_task(message="test")
        tasks.run_next_task()
        self.assertTrue("test" in completed_tasks)

The management command process_tasks might be a helpful reference: https://github.com/arteria/django-background-tasks/blob/master/background_task/management/commands/process_tasks.py#L85

Thanks.  That would work.  But let's say I have a function used all over the code like send_email.  And I decorate it to now be a background task.
Would I need to go into every single test and tell it to run_next_task().  Would there be a way to do this automatically everywhere?

You can use .task_function field of your task:
@background
def add(x,y):
    return x+y

add(1,2)  # will create a task
add.task_function(1,2)   # will launch your function immediatly and synchronously

also check out this (PR)[https://github.com/lilspikey/django-background-task/pull/17#issuecomment-163598417]. I accidentally pushed it to wrong repo.

@Zagrebelin could you please send the PR#17 to this repo as well? Thanks


Fix getting full paths cross platform

Previously readlink -f was used, however this does not work on OSX because a different version is installed. The command realpath is available on both Linux and OSX and so can be used cross-platform.

Peter if this works for you on your system, this can be merged.

Realpath does work on my system (ubuntu) but it is not a default a installation so a user would have to install it in order to run the tests.

see issue #44


Remove const_get from StringIdentifierArgument target

[Fix #425]
const_get doesn't accept symbols that are something like :"Foo::Bar", and that's intended according to this issue. https://bugs.ruby-lang.org/issues/12319
Therefore, changing String to Symbol might not work as expected. As far as I know that's related to const_get only so we can remove it from the target of StringIdentifierArgument

Before submitting the PR make sure the following are checked:

[x] The PR relates to only one subject with a clear title and description in grammatically correct, complete sentences.
[x] Wrote good commit messages.
[x] Commit message starts with [Fix #issue-number] (if the related issue exists).
[x] Feature branch is up-to-date with master (if not - rebase it).
[x] Squashed related commits together.
[x] Added tests.
[x] Ran bundle exec rake default. It executes all tests and runs RuboCop on its own code.
[x] Added an entry (file) to the changelog folder named {change_type}_{change_description}.md if the new code introduces user-observable changes. See changelog entry format for details.

I don't think this is the right solution.
This should just skip interpolated strings for const_get. In addition it's easily analyzable if the string literal contains double-colons to prevent a false positive.
From testing, this also applies to:

const_source_location
const_defined?

These don't:

public_const/private_const don't accept nested constants
deprecate_constant as well
Same for remove_const and set_const

You also need to update tests.

@Earlopain You are right. At least this change must cover all affected methods.
One concern is that the rule will be a little complex. Some might be confused about why it's not applied for some cases. We need to have a good documentation to explain.

@Earlopain What about this case:
const_get("#{process(name)}Resource")

Here, process method returns a string that MIGHT contain ::. Is it analyzable?

Perhaps in some trivial cases but generally no. I would just ignore those.

I'll close this PR in favor of #430, which proposals a better resolution. Thank you.


Build a miror behind a proxy

Hello,
Unless I missed something, but I can't build a mirror behind a proxy
In streamhandler.php there is a dns request with php_network_getaddress()
The proxy works well with env and curl export https_proxy=http://X.X.X.X:port
This feature could be very useful inside company without direct internet access

Replace https://my-proxy-server:8080 with tcp://my-proxy-server:8080 works

Can you send a PR to demonstrate or post a snipet code?

Just an proxy export is needed in shell
export http_proxy=tcp://my-proxy-server:8080
export https_proxy=tcp://my-proxy-server:8080
This syntax is unusual (http_proxy=http://my-proxy-server:8080 works usualy)

Thanks!


The ordinal 345 could not be located in the dynamic link library

As of the last two (or three?) updates, I'm receiving the following error when trying to launch VS Code Insiders.
The ordinal 345 could not be located in the dynamic link library C:\Program Files (x86)\Microsoft VS Code Insiders\code - insiders.exe

The only error recorded in the windows event logs was an Application popup log entry in the System logs. It simply contained a copy of the message - no further information.
Are there logs recorded anywhere that I can post for assistance?
I've searched everywhere and the only references I can find relate to multiple ComCtl32.dll versions in windows xp. (e.g.: http://stackoverflow.com/questions/37016803/calling-taskdialogindirect-loader-says-ordinal-345-not-found)

VSCode Version: 1.12.0.insider (2017-04-27T06:12:15.095Z)
OS Version: Windows 10 Pro v10.0.14393

Steps to Reproduce:

Launch Code - insiders.exe <- throws error.
Launch Code - insiders.exe a second time <- runs ok.

Repeat this process over and over with same result every second time.

@doublehelix does a complete uninstall and reinstall help at all?

Hi @bpasero,
Yes! uninstalling and removing all settings, then re-installing from the 1.12 insiders installer did not cause the error any more. (v1.12.0-insiders - 2017-04-28T18:57:52.518Z)
The error originally occurred three versions in a row during the re-launch after automatic upgrade.
I will just confirm that the issue is resolved by performing the next automatic upgrade when available (in the next day or two I presume).
Not sure if it's the new version or the uninstall that fixed it. But thanks.


[Documentation] Istio Authorization Policy "principals" works without mTls

Bug description
Hi all,
As the documentation says, we should have mtls enabled in our cluster or the microservices that want to user AuthiorizationPolicy using the principal field.
Istio uses mutual TLS to securely pass some information from the client to the server. Mutual TLS must be enabled before using any of the following fields in the authorization policy:

the principals field under the source section

Reference: https://istio.io/v1.7/docs/concepts/security/#dependency-on-mutual-tls
The point is, we apply this configuration bellow and the AuthorizationPolicy is working without mTls enabled
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: identity-metadata-service
spec:
  selector:
    matchLabels:
      app: identity-metadata-service
  action: ALLOW
  rules:
  - from:
    - source:
        principals: ["cluster.local/ns/default/sa/identity-service-account"]
    to:
    - operation:
        methods: ["GET"]
        paths: ["/actuator/info"]

The result works as declared on the AuthorizationPolicy file but without any mTls enabled between the pods.

Expected behavior
The AuthorizationPolicy should not work?
Steps to reproduce the bug
Apply any authorization policy using principals rule without mtls enabled
Version
istio 1.7
kubernetes 1.18.8
How was Istio installed?
Using istio operator 1.7.2
http://github.com/istio/istio/operator
Environment where the bug was observed (cloud vendor, OS, etc)
Azure AKS

mTLS is enabled between sidecars where possible by default: https://preliminary.istio.io/latest/docs/ops/configuration/traffic-management/tls-configuration/#auto-mtls

mTLS is enabled between sidecars where possible by default: https://preliminary.istio.io/latest/docs/ops/configuration/traffic-management/tls-configuration/#auto-mtls


Whitelist spectrum.ieee.org

Energized Protection - block ⚡

Let's make an annoyance free, better open internet, altogether!

Issue Submit Form

  Provide the following info properly, which will help us to resolve your issue quickly.

Issue(s):
Type x in between [ ] and make sure there isn't any space between brackets. Example; for Your Selected Issue(s), type like this - [x]
You can select more than one category of issues if you need to!

[x] Whitelist
[ ] Blacklist
[ ] App Broken
[ ] Website Broken
[ ] Request
[ ] Other Issue

Pack(s):
Write the name of the pack(s) you are using.

Pack(s) Name: Ultimate

Extension(s):
Also name the extension(s) if you are using any. If you don't then leave it blank.

Extension(s) Name: None

Domain(s):
If you are submitting this issue for whitelist/blacklist issue, send us the domain(s) for whitelisting/blacklisting here. Kindly use the Code Tag to prevent tracking.

Domain(s): spectrum.ieee.org

Your Config:
Just to ensure there is no issues or conflicts with other app/software/magisk module/extension/source list.
Make sure you are running Energized Protection Service only.

Client: Adguard Home Server

Version: Updated 29/03/22

Detail:
Write us a lil bit more about your issue or query. You can attach any screen shot or log of the issue or advert, this will help to highlight it.

Your Issue Detail: spectrum.ieee.org is well know website.

Thank you for making Energized Protection great, with your kind help!
A project by Ador with ❤

Is this still an issue? I don't see it in the unified lists.


Add density plot to dashboard

As a user of Gemdat,
I want to be view or download the density plot,
So that I can do my analysis without having to write any code
I think we have 2 options:

Add a volume viewer to the dashboard
Add a way to generate and download the volume in vasp format

Option 1. depends on whether we can find a good volume viewer for the dashboard to display volumes/atomic structures. Otherwise, option 2 would be a good fallback in combination with dedicted crystallographic software like Vesta.

This links to the code @AstyLavrinenko showed us yesterday right?
Then I suppose we can do option 1. and work together to import the code .

Yeah, makes sense. Could be added under a different tab with a link to download the density in vesta format.

Yes, and I've just added the function to get the plot that I showed yesterday. But I need some help to integrate it with the main class.

I will start working on this now that #160 is implemented.


Create OpenAI API Key and add it to your .env file: openai

Could you please clarify step 4 in instructions properly. Can you create a sample .env file and add sample env names and value pairs. so openai API key can be replaced with correct value.
Currently it is not clear what would be the enviornment variable name, like openapikey, open_api_key or simply open_api etc.

@starpunjabi Thank you for reaching out! Will update it soon.
In the meantime, in your env, specify the variable as follows :
OPENAI_API_KEY = "Enter the Secret Key Here"

Completed the upgrade!


Docs: accessibility and IE fixes for SVG-based top navigation

fixes issue of some browser/AT combinations not reading an <svg>'s <title> for the link text (by adding an explicit aria-label to the link)
fixes issue of IE explicitly focusing on <svg> elements (resulting in two tab stops for each <svg>-based link - one for the link, one for the nested <svg>)

hmmm, travis is taking an inordinately long time to run here it seems...


maybe use isolated pg session for each slt in e2e test

Currently all slt file in e2e test runs in one session. This might be a problem when we test SET xxx, all later slt test will use the earlier slt test parameters.
Of course, we can set a parameter back after finishing using it, but it is error-prone.

We already support this by setting -j 1.

    -j, --jobs <JOBS>        Whether to enable parallel test. The `db` option will be used to create
                             databases, and one database will be created for each test file

It's one db for one file, but maybe one session for one file should be enough?

Sounds good. May have a try.

@BugenZhao We had encountered to the smae issue before, right? Did you do some fix?

@BugenZhao We had encountered to the smae issue before, right? Did you do some fix?

I think this has been resolved in https://github.com/risingwavelabs/risingwave/pull/4902.


Add computers.scorptec.com.au

computers.scorptec.com.au is the url for searching for products on scorptec.com.au which is already on the list.

Thanks!


Release/v4.0.7

READY FOR REVIEW
Summary

#369

Review By (Date)

EOD August 1

Criticality

How critical is this PR on a 1-10 scale? Also see Severity Assessment.
E.g., it affects one site, or every site and product?

Review Tasks
Setup tasks and/or behavior to test

Look at code diff

@forrestglick Release has been prepped - please let us know when it's ready to be push to production.

@yvonnetangsu Thanks for the quick work! I reviewed on Dev and noted one small issue: on other entries, including an adjacent entry, it says: "To Be Named". On this new entry it says: "To be Named" – noting the lowercase letter "b". Would it be possible to update the case to match the other entries? With this change, I approve the push to Prod. Thank you!
The page where you can see the entry: https://dev--adapt-giving.netlify.app/endowed-positions/school-of-humanities-and-sciences/?page=3

Thanks for the review @sherakama!
@forrestglick Updated to "To Be Named" to match the other entries.

@forrestglick Changes are live!

Awesome! Thanks to @yvonnetangsu and @sherakama for the quick work. I'll let the client know it's live.


ReactNative 😅

Is it possible to use it with ReactNative ?

I don't know, but if your KUTE.js animation code runs after DOM changes, it should be fine.

Thnx for the reply - kute depends on DOM, right? if yes it won't work :)

Also requires window for the to() method where getComputedStyle() is used.

Thanks for explaining 🤗


What is "Properties" parameter value??

Missing the value~!

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 7dc8ef75-1f1d-2d0c-12ad-20159bfd26b9
Version Independent ID: 03ecc613-7ab0-b470-6995-acc8583bb683
Content: Add tenants for usage and billing to Azure Stack
Content Source: articles/azure-stack/azure-stack-csp-howto-register-tenants.md
Service: azure-stack
GitHub Login: @mattbriggs
Microsoft Alias: mabrigg

Thanks for your feedback! We will investigate and update as appropriate.

@niceysj good catch!
@mattbriggs please take a look. They are referring to the code sample here:
New-AzureRmResource -ResourceId "subscriptions/{registrationSubscriptionId}/resourceGroups/{resourceGroup}/providers/Microsoft.AzureStack/registrations/{registrationName}/customerSubscriptions/{customerSubscriptionId}" -ApiVersion 2017-06-01 -Properties

@niceysj  Thanks for your feedback. I'll find out the answer and update! #in-process

-- Properties  Specifies properties for the resource. Use this parameter to specify the values of properties that are specific to the resource type. Updating doc. Thanks! #please-close


Move Group or multiple channel at once

please add this feature. and I think column 3  should be named a "Group" not "Title".
thanks.

Found a bug in Ctrl-I, otherwise you could use Ctrl-x, Ctrl-I to move a whole block of lines. Will fix it in next version.

Found a bug in Ctrl-I, otherwise you could use Ctrl-x, Ctrl-I to move a whole block of lines. Will fix it in next version.

great, thank you.

great, thank you.

bugfix released. Don't know if I can do this function as a button in UI, because with shortcuts you can bundle rows together. The Move up/down button cannot do this in an easy and logic way (?)

bugfix released. Don't know if I can do this function as a button in UI, because with shortcuts you can bundle rows together. The Move up/down button cannot do this in an easy and logic way (?)


Changelog v1.17.0

blocked by https://github.com/go-gitea/gitea/pull/20530

Could we wait for #20548 and #20549 ? Some users need that for v1.17.

Could we wait for #20548 and #20549 ? Some users need that for v1.17.

Backporting features on the last moment? Looks risky for me, no time to iron out bugs etc.

we did reject a lot of "can we have this in v1.17" ... at that rate we just could v1.18 ... - I'm also against it


Port number incorrect in custom tab documentation

[Version]
16.1 fix4 2016/12/16
[How to reproduce]

check documentation for add service/ custom tab section
help/en/help/tasks/service_add.htm

[Observed result]
example for  URL of a custom tab
For example:
https://MyAPPHost:8880/oscm-app-openstack/serverInformation.jsf
[Expected]
Either make it http: or use 8881 as a port number

Corrected the sample. It now reads https and 8881
Modified source files:
UserDoc/DocSources12/en/help/tasks/service_add.xml
UserDoc/DocSources12/en/help/tasks/service_edit.xml
UserDoc/DocSources12/de/help/tasks/service_add.xml
UserDoc/DocSources12/de/help/tasks/service_edit.xml
UserDoc/DocSources12/ja/help/tasks/service_add.xml
UserDoc/DocSources12/ja/help/tasks/service_edit.xml
UserDoc/DocSources12/en/ReleaseNotes/WhatsNew/c_ExtensionInterface.xml

[Version]
16.1 fix6 2017/02/17 verified


Where does the score file in FastRerank come from?

In config.py, on line 26-28
it seems like the preprocessing step need article.txt, title.txt, template, samples.index.json and _score.json to be prepared in config.py to run the whole process
But after doing retrieve, I only got train/test/dev.sample.index other than the original article and title file
So how can I get all the other data I need such as  sample.index.json and _score.json ?

You can use the index file to get the templates (summaries of the corresponding training article), and each line contains 30 indices of one sample. The score is the ROUGE-1 of template evaluated with true summary. I'm sorry I didn't include this code, because when I implemented this project, there was no suitable python wrapper for ROUGE evaluation. I had to use the perl version seperatedly for this job.

Could you give an quick example of what does these 2 json file look like?
Cause I’m now getting a text file after doing Retrieve part and not sure how the require .json file suppose to format.

It looks like this,
[{"art_idx":"0","scores":[0.25,0.1333333333,0.25,0.25,0.1111111111,0.125,0.1428571429,0.2666666667,0.2857142857,0.2666666667,0.2666666667,0.2,0.25,0.125,0.2666666667,0.2666666667,0.2666666667,0.2,0.1538461538,0.375,0.625,0.25,0.5,0.2666666667,0.125,0.5333333333,0.2666666667,0.2666666667,0.25,0.125],"tp_idx":[280563,468740,2977802,2978740,1305283,810428,143628,3305902,96755,227145,227356,228893,2668569,2669230,2669605,2579854,2579826,86884,54116,88311,186211,342885,414963,558914,1305361,897042,2608945,2832328,554728,98514]}]

Thanks
Also, what does the samples.index.json like?


Smaller benchmarks

In recent PRs, we've never needed to benchmark the performance of individual set types.
At this point in development, we should focus on benchmarks of our defaults instead. The only exception in this PR are shared Hessian tracers, which don't have a public API yet.

:warning: Please install the  to ensure uploads and comments are reliably processed by Codecov.
Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 41.22%. Comparing base (ba2f2fe) to head (168795a).
Report is 1 commits behind head on main.

:exclamation:  There is a different number of reports uploaded between BASE (ba2f2fe) and HEAD (168795a). Click for more details.
HEAD has 3 uploads less than BASE

Flag
BASE (ba2f2fe)
HEAD (168795a)

4
1

Additional details and impacted files
@@             Coverage Diff             @@
##             main     #157       +/-   ##
===========================================
- Coverage   92.21%   41.22%   -50.99%
===========================================
  Files          38       36        -2
  Lines        1554     1453      -101
===========================================
- Hits         1433      599      -834
- Misses        121      854      +733

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Notice: Undefined variable: message in .../eden/core/src/Argument.php on line 54

Hi!
in Argument.php line 54 there is:
public static function i()
{
    $class = __CLASS__;
    return new $class($message, $code);
}

since $message and $code is undeclared, it should be
public static function i()
{
    $class = __CLASS__;
    return new $class();
}

+1 please implement this simple fix :-)

Thank you. This works


Drop secret from spec

The SECRET_KEY parameter is sensitive information and should not be part of spec. Because the value is internal and is not exposed to clients, this makes the value auto-generated without user input.

Verified this change in my local env.
$ oc describe secrets/horizon
Name:         horizon
Namespace:    openstack
Labels:       horizon.openstack.org/name=horizon
              horizon.openstack.org/namespace=openstack
              horizon.openstack.org/uid=65dc80f0-29d4-4f01-8a7f-0c8153039a67
Annotations:  <none>

Type:  Opaque

Data
====
horizon-secret:  10 bytes

$ oc exec -t horizon-5bff8d5fd5-x9nbn -- cat /etc/openstack-dashboard/.horizon-secret
mlmjj5g47k


fix iso_gradient_mode in constructor call

Description
I have spotted that iso_gradient_mode value provided to constructor is not used in Labels layer.
This PR fix it and add missed docstring and type annotation

I don't think so - I must have just missed it!


⚠ Bofrost.fr has degraded performance

In 26fd230, Bofrost.fr (https://www.bofrost.fr) experienced degraded performance:

HTTP code: 200
Response time: 874 ms

Resolved: Bofrost.fr performance has improved in fe4ce6b.


Use filerGroup for s3 buckets collection prefix

What problem are we solving?
link to discussion
We can't have two buckets with the same name in two different filers with different filer stores because they use the same collection name in the volume server  and of course, we want this change to be backward compatible
How are we solving the problem?
Use filerGroup as a prefix for buckets collections name (filerGroup_bucket)
I added these flags:

s3.collectionPrefix in filer

weed filer  -filerGroup abc -s3 -s3.collectionPrefix

collectionPrefix and filerGroup in  s3

weed s3 -filerGroup abc -collectionPrefix

s3CollectionPrefix in shell

weed shell -filerGroup abc -s3CollectionPrefix

And when s3 gateway uses filerGroup as a prefix of the collection name and sets the collection name query param
And change the collection of buckets in other files
How is the PR tested?
with these commands:
Run two filer, one volume server and, one master
go run ./weed master -volumeSizeLimitMB 100
go run ./weed volume -port 5555 --dir /tmp/abc -max 0
go run ./weed filer -s3 -port 5566 -s3.port 6655 -s3.collectionPrefix -filerGroup first
go run ./weed filer -port 5567 -filerGroup second -defaultStoreDir second-store
go run ./weed s3 -port 6656 -collectionPrefix -filerGroup second -filer localhost:5567

/tmp/a is a big file
echo "s3.bucket.list" | go run ./weed shell -filerGroup first -s3CollectionPrefix
echo "s3.bucket.list" | go run ./weed shell -filerGroup second -s3CollectionPrefix

No bucket
aws --endpoint http://localhost:6655 s3 mb s3://test
echo "s3.bucket.list" | go run ./weed shell -filerGroup first -s3CollectionPrefix
>   test	size:0	chunk:0
echo "s3.bucket.list" | go run ./weed shell -filerGroup second -s3CollectionPrefix
>
aws --endpoint http://localhost:6656 s3 mb s3://test
echo "s3.bucket.list" | go run ./weed shell -filerGroup first -s3CollectionPrefix
>   test	size:0	chunk:0
echo "s3.bucket.list" | go run ./weed shell -filerGroup second -s3CollectionPrefix
>   test	size:0	chunk:0

Now we have two bucket with same name and we are going to upload files to theme
aws --endpoint http://localhost:6655 s3 cp /tmp/a s3://test
aws --endpoint http://localhost:6656 s3 cp /tmp/a s3://test
echo "collection.list" | go run ./weed shell
> collection:""	volumeCount:7	size:186600	fileCount:3	deletedBytes:0	deletion:0
collection:"first_test"	volumeCount:7	size:8025168	fileCount:411	deletedBytes:0	deletion:0
collection:"second_test"	volumeCount:7	size:8025168	fileCount:411	deletedBytes:0	deletion:0
Total 3 collections.

Now we are going to enforce quota on first bucket
echo "s3.bucket.quota -name test -sizeMB 1" | go run ./weed shell -filerGroup first -s3CollectionPrefix
echo "s3.bucket.quota.enforce -apply" | go run ./weed shell -filerGroup first -s3CollectionPrefix
>   test	size:8025168	quota:1048576	usage:765.34%
    changing bucket test to read only!
echo "s3.bucket.quota.enforce -apply" | go run ./weed shell -filerGroup second -s3CollectionPrefix
>
aws --endpoint http://localhost:6655 s3 cp /tmp/a s3://test
aws --endpoint http://localhost:6656 s3 cp /tmp/a s3://test
> upload failed: ../../../../../tmp/a to s3://test/a
> upload: ../../../../../tmp/a to s3://test/a

Only first bucket is readonly
echo "s3.bucket.list" | go run ./weed shell -filerGroup first -s3CollectionPrefix
>   test	size:8025168	chunk:411	quota:1048576	usage:765.34%
echo "s3.bucket.list" | go run ./weed shell -filerGroup second -s3CollectionPrefix
>   test	size:16063432	chunk:411
echo "s3.bucket.delete -name test" | go run ./weed shell -filerGroup first -s3CollectionPrefix
>
echo "s3.bucket.list" | go run ./weed shell -filerGroup first -s3CollectionPrefix
>
echo "collection.list" | go run ./weed shell
> collection:""	volumeCount:7	size:593272	fileCount:7	deletedBytes:0	deletion:0
collection:"second_test"	volumeCount:7	size:16063432	fileCount:822	deletedBytes:8011551	deletion:411
Total 2 collections.

Checks

[ ] I have added unit tests if possible.
[x] I will add related wiki document changes and link to this PR after merging.

Looks like there is some network issue with archive.ubuntu.com so End to End / FUSE Mount fails  I event pushed another commit to trigger builds again but again build failed.
I removed those flags from s3 and shell and get filerGroup from filer response


Automatic pagination

Java SDK v2 added support for automatic pagination of (paginated) responses. We should look into possibly implementing this as it would be a better customer experience.
internal id: 172743731

Note: SdkAsyncIterable ought to probably be renamed to SdkAsyncSequence since it is lazy like Sequence (Iterable on the other hand implies eager which it is not)
Update design as necessary when implementing to make this distinction clear.


Start a Gitter Chat Room for BIDMach / BIDMat

Please start a https://gitter.im/ Chat Room for BIDMach / BIDMat. Documentation for the projects is sparse, and adoption will remain limited unless the user community has an opportunity to interact with the project authors and other community members.
This approach is used very successfully by a large number of github-hosted projects.

Thanks for this suggestion. I will check out gitter. I never heard of it before, and if it looks like it might be helpful I can bring this up with John later.
By the way, do you have a specific area or high-level suggestion for our documentation that I can clarify?

Thanks, Daniel. I'm having difficulty using Word2Vec load / save Google vector files, and would love to see some sample code.

It should be there now. Can you find it and post to it?

Hi, thanks for creating the room. I can't seem to find either a BIDMach or BIDMat room in the search. Could you perhaps add the Gitter link or badge to the README of the project, or the wiki?
Thanks!

The Gitter room is called BIDData. I just joined via an email invitation, but I assume there's a way for you to search it? Let us know if you still cannot find it.
With regards to Word2Vec, there are a couple of examples from the Berkeley Institute for Data Science (BIDS) destress project repository somewhere, but it's hard to learn from that so I agree, some form of documentation will be helpful. I've been meaning to learn how to use that code for a while so maybe this will motivate me to write something. I'll keep this issue here open for future reference.

By the way, I remember this old issue you posted: https://github.com/BIDData/BIDMat/issues/31
Is this still an issue for you?

Thanks. I took a look at destress. Appears you're also using gensim. I'm looking for a means to convert the MAT output from BIDMach Word2Vec to Google C Word2Vec format. Am going to wade in deeper.

When I navigate to: https://gitter.im/BIDData/
I get the error below. Would you like me to speak to the Gitter team? I know one of the cofounders.

BIDData
You've tried to access BIDData's private organization room and you do not belong to this organization.

 Unfortunately this organization has no public rooms.

Yeah, I tried to add you to the room, and it says you don't have permission. By hovering over the lock icon next to the name, it appears that only organization members (i.e., only those that have push access) can actually join.
Unfortunately, I'm not sure how to change these settings; there's a settings menu there but nothing seemed to work. Also, I don't even know if I can change these settings since John was the one who created it. @jcanny can you modify the gitter settings by, say, clicking on that lock icon next to the BID Data name, or that settings icon?

Ah, I think I figured out the issue: https://gitter.zendesk.com/hc/en-us/articles/200178951-Gitter-rooms
Apparently, we need a BIDMach (not BID Data) gitter, because BID Data is the organization-level room.

This would be very helpful.

Yeah I think there are two approaches: an org or team room (which is locked
down), or a public project room.
The latter is likely what we're looking for.
On Sun, Jul 5, 2015 at 2:38 PM kovasb<EMAIL_ADDRESS>wrote:

This would be very helpful.
—
Reply to this email directly or view it on GitHub
https://github.com/BIDData/BIDMach/issues/55#issuecomment-118649455.

OK, I added BIDMach and BIDMat. THey're supposed to be public. Do you
see them now?
-John
On 7/5/2015 8:53 PM, Daniel Chalef wrote:

Yeah I think there are two approaches: an org or team room (which is
locked
down), or a public project room.
The latter is likely what we're looking for.
On Sun, Jul 5, 2015 at 2:38 PM kovasb<EMAIL_ADDRESS>wrote:

This would be very helpful.
—
Reply to this email directly or view it on GitHub
https://github.com/BIDData/BIDMach/issues/55#issuecomment-118649455.

—
Reply to this email directly or view it on GitHub
https://github.com/BIDData/BIDMach/issues/55#issuecomment-118650449.

I do. Brilliant. Thank you!