Add payloadType option to file

This adds the option to specify payloadType: binary for a file, which uploads the blob as application/octet-stream without any multipart/form-data encoding.
As discussed in #9.

Some testing is probably in order to make sure this doesn't break any existing functionality. I don't have a setup to test the multipart/form-data method.

This is interesting! I'm not sure if this does everything, since it should pass along all the data as well as the payload.

Good point. The only way to pass additional data would be query parameters. That's what this library did when they added support for this. Personally, I'm just using HTTP headers which works fine.
I agree that at a minimum, the docs should specify that the data option is ignored in binary mode. Thoughts?

HTTP headers sound good to me!

Do you mean that the library should set the contents of data as HTTP headers? Or the docs should advise to use the headers option instead of data? My vote would be for less magic, and to just document that data is ignored.

I think that'd make the most sense to use headers.

Thanks, this is a nice feature. I extended it slightly so that it's still possible to pick the content type. application/octet-stream is not the only thing one might want to directly submit. I'm using this with s3, where it's nice to just PUT any file.

@ef4 I can merge this, would you like to add your changes after?

Looks like you have already merged them in https://github.com/tim-evans/ember-file-upload/pull/23
Thanks, looks good.

Is it possible that s3 upload does not work without this binary option?
I've been trying it out and the file on s3 has content type of form data and is not usable that way.

Yes, I believe that is the case. The S3 API expects the payload of the request to be just the binary file. It's not expecting form encoding.

Thanks @jlevycpa.
I'm not sure what the right way is of helping to get this merge request into master, so i made a little fork of the current and added your logic to it, so i could try and use it.
Seems to work fine so far: https://github.com/maxigs/ember-file-upload
Please let me know if i can help to get it into a proper released version :)

@maxigs I think I'm trying to understand the ramifications of this. How does extra data get passed along in this use case?

In the upload itself there is no way of adding extra data as it's just uploading the file itself to s3.
I'm not sure if there is way to attach any additional data to be stored at s3, i'm pretty new to using it.

Here's the documentation around the headers that are sendable for that:
http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUT.html

Thanks so much for this library. Wish it featured an option to set Content-Type instead. Unfortunately, hardcoding content type to application/octet-stream as proposed in this PR wouldn't do. S3 requires the content type hashed in a signed request to match that of the actual upload request. I.e., if it was Content-Type: image/png, the upload needs to be same.

Including a big +1 on getting this merged.  The use case this would primarily support is uploading to S3 via pre-signed urls.
http://docs.aws.amazon.com/AmazonS3/latest/dev/PresignedUrlUploadObject.html
Using these, the Content-Type on the request needs to match what was provided when creating the PresignedUrl. In this case, "application/octet-stream" is entirely appropriate, as it represents a stream of data to be saved to a file defined in the PresignedUrl. The ability to specify the Content-Type header explicitly would also work, of course. But really, S3 doesn't care.
I have tried it out, and maxigs's implementation works great. Without it, I can't use this great (seriously, thanks for the hard work) module.

In this case, "application/octet-stream" is entirely appropriate

@bissleb It seems you are ignoring the original content types of objects in your S3 bucket. However, that wouldn't work for applications that require original Content-Type to be stored (For example, CloudFront serving image assets with S3 buckets as origin. See http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/AddingObjects.html).
I might have a solution to this. I will spend some time to raise a PR.

Thanks @alexnj

Yes, that's true, @alexnj. We are copying the files from the upload bucket to another and setting the appropriate metadata on the new object. So, probably a narrower use case than other people.
Thanks! Let me know if I can help at all.

Hi there. Once the version from @maxigs is in, and the mime-type option from @alexnj, I can update the README with a Node JS getSignedUrl pattern and the updated URL pattern. I've got the thing working but it was a long way around.
In particular (this bit took the most time) you need to set the method to 'PUT'.
        return file.upload(response.url, {
          data: response.credentials,
          method: 'PUT',
          payloadType: 'binary',
          contentType: file.blob.type,
        });

If I get a moment I'll PUT all this together in a single PR.

@queenvictoria that'd be awesome. I'll get to merging this soon. Sorry for delays on this; it's a large feature that I want to be sure about the API with.

Hey all, would you be fine if I made a new function called uploadBinary or uploadRaw instead of having a payloadType?

Fine with me!

So for future readers: although this commit https://github.com/tim-evans/ember-file-upload/commit/f2f01b91d7214174146dd41978894db30db7781c looks like it supports payloadType, if you're installing 2.0.0+ then as @tim-evans says, there is a new method uploadBinary.
So this is what was happening with < 2.0.0:
        return file.upload(response.url, {
          payloadType: 'binary',
          method: 'PUT',
          contentType: file.blob.type,
          data: JSON.stringify(response.credentials),
        });

And this is what is happening now:
        return file.uploadBinary(response.url, {
          method: 'PUT',
          contentType: file.blob.type,
          data: JSON.stringify(response.credentials),
        });

So good news! It all works out of the box.

So for future readers: although this commit https://github.com/tim-evans/ember-file-upload/commit/f2f01b91d7214174146dd41978894db30db7781c looks like it supports payloadType, if you're installing 2.0.0+ then as @tim-evans says, there is a new method uploadBinary.
So this is what was happening with < 2.0.0:
        return file.upload(response.url, {
          payloadType: 'binary',
          method: 'PUT',
          contentType: file.blob.type,
          data: JSON.stringify(response.credentials),
        });

And this is what is happening now:
        return file.uploadBinary(response.url, {
          method: 'PUT',
          contentType: file.blob.type,
          data: JSON.stringify(response.credentials),
        });

So good news! It all works out of the box.

Yay! Glad to hear!!

Hey @tim-evans I'd like to write up a S3, NodeJS/Express and Ember pattern. I see you're moving away from the README. So I can't make a PR correct? Anywhere I can do it more sensibly?

Feel free to add to the README; I'll migrate what you've written to the recipes in the docs app.
The docs are the dummy app in tests/dummy/app, FYI


Vim for pipeline filtering

I made a 40-line script that lets you call Vim commands for Unix pipeline filtering, like sed or awk: https://github.com/MilesCranmer/vim-stream.
As an example,
cat file.txt | vims -e 'label' 'f|d$'

will filter the output of "cat file.txt" by running the Vim command ‘f|d$’ on each line matching 'label' to delete everything after "|". The repo gives the other options.
I’m eager to hear: are you potentially interested in integrating this command line mode into Vim itself? The code looks tiny but pulls a lot of weight; I use "vim-stream" for batch text processing in nearly every script I write. I no longer spend time hunting for awk/sed commands, since Vim is ingrained in my fingers.
Is your feature request about something that is currently impossible or hard to do? Please describe the problem.
Yes. To do this with vanilla vim, you would have type:
cat file.txt | vim - -nes -c ':%g/label/exe "norm f|d$"' -c ':%p' -c ':q!' | tail -n +2

Describe the solution you'd like
Perhaps a flag could activate this pipeline filter mode, and this functionality could be integrated into vim.
I'm interested to hear your thoughts.
Cheers,
Miles

This article may be useful on implementation (or moving to):
https://vimways.org/2018/vims-social-life/#under-the-tree-neovim

Thanks @oblitum, this looks useful! I should emphasize, though, that the main additions of the script aren't just to make command-line vim less verbose, but to also make it easier to call commands like %g/{...}/exe "norm {...}" on piped text. Such a command is very very useful for batch text processing in a script, but is tedious to write out.
This is why I think it would be useful to integrate such short-hand in vim's execution arguments (and neovim potentially!).

The command could look like this: cat file.txt | vim -P ...., where the remaining commands would be the shorthand. Here, the "-P" flag stands for "pipe mode" and would essentially run the remaining arguments to vim according to the short hand defined in https://github.com/MilesCranmer/vim-stream/.
For example,
cat file.txt | vim -P '%s/foo/bar/g'

^Replaces foo with bar using vim syntax.
cat file.txt | vim -P -l 'f|d$' -e 'label' 'AHello World'

Goes through every line (-l), and deletes everything after |. Then, every line with "label", it appends "Hello World".
-l usually turns on Lisp mode, but since we have called -P as the first argument, every other argument to vim is interpreted as a vim-stream function.
I'm happy to provide code for this to integrate it to vim, by the way.

It's an interesting idea.  Your first example shows an Ex command, while the second one is a Normal mode command. So -l implies that?
The match & append could be done with 'g/label/s/$/Hello World'  Is it needed to add another way to select lines?

Exactly! The heart of the script (in bash) looks like this:
none)        vim_cmds+=(-c "$1");;
-s)          vim_cmds+=(-c ":exe \"norm ""$1""\"");;
-l)          vim_cmds+=(-c ":%g/.*/exe \"norm ""$1""\"");;
-e)          vim_cmds+=(-c "%g/$1/exe \"norm $2\""); shift;;
-r)          vim_cmds+=(-c "%v/$1/exe \"norm $2\""); shift;;

So by default, you run Ex commands. The "-s" flag lets you type in vim command mode (it just runs :exe "norm {...}"), which gets applied to the piped text. "-l" runs a command mode command on every line. "-e" runs a command on every line matching a regex, and "-r" not matching the regex.
It's simple but it's easy to fall in love with. You are right; there is usually an equivalent Ex command, but for more complicated jobs (e.g., requiring copy-pasting around a command output), command-mode vim seems to require less thinking. For example, say I run "ls -lath" and want to isolate the time of modification for each file. I could write out an awk/sed/Ex command, but that would require more thinking (and counting columns) than just running a vim command on every line:
ls -lath | vims -l 'f:hhd0f d$'

This runs "f:" to get to the colon (in the printed time), "hh" to move left twice, "d0" to delete to the start of the line, then "f " to move in front of the time, then "d$" to delete to the end.
It feels much more natural than using sed/etc, so I use this script all the time when text processing. I don't think I have used awk or sed ever since; I only need vim (and occasionally python) for scripted text parsing.


Make use of secure port when accessing Kubelet API

What does this PR do?
This PR switches Metricbeat k8s manifests and docs to point to Kubelet secure port over https instead of the insecure port.
Why is it important?
Insecure port of Kubelet (10255/TCP) is now less common and discouraged and also in most cases it is not enabled by default (requiring to restart kubelet with --read-only-port flag)
Related to https://github.com/elastic/beats/pull/16063

jenkins test this please

Hey @jmlrt . What is the status on this? Is this valid or is anything else missing here?

jenkins test this please

Hi @ChrsMark,
I'll take a look at your PR today.
Meanwhile tests are failing, can you merge master on your branch?

Updated with the latest master. Not sure if the latest failure is related though.

@ChrsMark This is really strange, we have a test which queries elasticsearch for q=metricset.name:container%20AND%20kubernetes.container.name:metricbeat and verify that it match the metricbeat index here.
This test is failing in your PR because there is no document matching metricset.name:container and kubernetes.container.name:metricbeat. In addition of our failing test on GKE, I could also reproduce it locally on Docker for Mac.
In the same time, The test on master branch is still working well (see logs here).

Hi @jmlrt ! Could I somehow check what is the output of Metricbeat pod? If there is a way to reproduce it locally feel free to mention and I could give it a shot.
To share some content here, with this change Metricbeat will try to query for metrics from Kubelet's API secure port instead of the insecure which was the previous one in the configuration. So I'm wondering if for some reason this port is not accessible in the testing env maybe because Kubelet is configured without this port enabled or for some reason this port is not exposed.

Hi Chris,
Here are the logs of a metricbeat pod when deploying your metricbeat chart from this PR on a GKE 1.15 cluster with default config:
2020-04-03T10:38:59.361Z        INFO    instance/beat.go:622    Home path: [/usr/share/metricbeat] Config path: [/usr/share/metricbeat] Data path: [/usr/share/metricbeat/data] Logs path: [/usr/share/metricbeat/logs]
2020-04-03T10:38:59.361Z        INFO    instance/beat.go:630    Beat ID: fb4eeb21-eef7-49ae-9878-0fb6643c76d3
2020-04-03T10:38:59.396Z        INFO    [api]   api/server.go:62        Starting stats endpoint
2020-04-03T10:38:59.397Z        INFO    [api]   api/server.go:64        Metrics endpoint listening on: <IP_ADDRESS>:5066 (configured: localhost)
2020-04-03T10:38:59.397Z        INFO    [seccomp]       seccomp/seccomp.go:124  Syscall filter successfully installed
2020-04-03T10:38:59.397Z        INFO    [beat]  instance/beat.go:958    Beat info       {"system_info": {"beat": {"path": {"config": "/usr/share/metricbeat", "data": "/usr/share/metricbeat/data", "home": "/usr/share/metricbeat", "logs": "/usr/share/metricbeat/logs"}, "type": "metricbeat", "uuid": "fb4eeb21-eef7-49ae-9878-0fb6643c76d3"}}}
2020-04-03T10:38:59.397Z        INFO    [beat]  instance/beat.go:967    Build info      {"system_info": {"build": {"commit": "d57bcf8684602e15000d65b75afcd110e2b12b59", "libbeat": "7.6.2", "time": "2020-03-26T05:27:27.000Z", "version": "7.6.2"}}}
2020-04-03T10:38:59.397Z        INFO    [beat]  instance/beat.go:970    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.13.8"}}}
2020-04-03T10:38:59.398Z        INFO    [beat]  instance/beat.go:974    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-04-02T12:49:11Z","containerized":true,"name":"metricbeat-metricbeat-6nlxd","ip":["<IP_ADDRESS>/8","<IP_ADDRESS>/24"],"kernel_version":"4.19.104+","mac":["46:ba:fe:e1:f9:e9"],"os":{"family":"redhat","
platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":7,"patch":1908,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
2020-04-03T10:38:59.399Z        INFO    [beat]  instance/beat.go:1003   Process info    {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fs
etid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid"
,"setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/metricbeat", "exe": "/usr/share/metricbeat/metricbeat", "name": "metricbeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-04-03T10:38:58.630Z"}}}
2020-04-03T10:38:59.399Z        INFO    instance/beat.go:298    Setup Beat: metricbeat; Version: 7.6.2
2020-04-03T10:38:59.399Z        INFO    [index-management]      idxmgmt/std.go:182      Set output.elasticsearch.index to 'metricbeat-7.6.2' as ILM is enabled.
2020-04-03T10:38:59.399Z        INFO    elasticsearch/client.go:174     Elasticsearch url: http://elasticsearch-master:9200
2020-04-03T10:38:59.399Z        INFO    [publisher]     pipeline/module.go:110  Beat name: metricbeat-metricbeat-6nlxd
2020-04-03T10:38:59.426Z        INFO    add_kubernetes_metadata/kubernetes.go:70        add_kubernetes_metadata: kubernetes env detected, with version: v1.15.11-gke.1
2020-04-03T10:38:59.426Z        INFO    kubernetes/util.go:94   kubernetes: Using pod name metricbeat-metricbeat-6nlxd and namespace default to discover kubernetes node
2020-04-03T10:38:59.438Z        INFO    kubernetes/util.go:100  kubernetes: Using node gke-test-jmlrt-15-default-pool-7204fd38-4rd1 discovered by in cluster pod node query
2020-04-03T10:38:59.539Z        WARN    tlscommon/tls_config.go:79      SSL/TLS verifications disabled.
2020-04-03T10:38:59.541Z        INFO    kubernetes/util.go:79   kubernetes: Using node gke-test-jmlrt-15-default-pool-7204fd38-4rd1 provided in the config
2020-04-03T10:38:59.541Z        WARN    tlscommon/tls_config.go:79      SSL/TLS verifications disabled.
2020-04-03T10:38:59.542Z        WARN    tlscommon/tls_config.go:79      SSL/TLS verifications disabled.
2020-04-03T10:38:59.543Z        INFO    kubernetes/util.go:79   kubernetes: Using node gke-test-jmlrt-15-default-pool-7204fd38-4rd1 provided in the config

It seems that secure ports isn't available from inside the pods with default GKE config.

After discussing and debugging it with @jmlrt, it was found that the problem was a missing hostNetwork: true from the pod which is present in Beat's repository manifests (https://github.com/elastic/beats/blob/master/deploy/kubernetes/metricbeat-kubernetes.yaml#L112).
Since it is not possible to add it in helm charts too (see https://github.com/elastic/helm-charts/pull/391#issuecomment-562059286 for more info), we switch from using HOSTNAME to NODE_NAME in order to access the Kubelet API.
@exekias any comments on this, since we have it with HOSTNAME in Beat's repo?

This sounds reasonable. Also since we are now exposing NODE_NAME we can switch to using it too to avoid confusion, WDYT?


Consider side of the road and crossing it via zebra&traffic lights for foot routing

We do not take care about the direction for foot routing but especially when going opposite direction we have to take care regarding sidewalk=no taggings like here
Not sure what exactly is the problem here.

I've not paid great attention to GH development in this direction since my last examples, but does this also cover situations like the following? (this has only sidewalk=left on a section, at least!)
http://www.openstreetmap.org/directions?engine=graphhopper_foot&route=45.49529%2C-122.82148%3B45.49389%2C-122.82631#map=18/45.49457/-122.82384
Practically this is a complete pain, using transit from Millikan way to the building, since there is no sidewalk in front of Millikan business center on millikan way (particularly if in a wheelchair, though I appreciate that is not a current routing profile!)
I am not aware of the specific legal situation, but given 'jaywalking' law in some areas, one would have to stay on the north side of millikan way and then cross at the junction with murray where there are crossing lights for sure...and there is no sidewalk there! In this case there is a very uneven dirt path one could take (and probably is by some), but it is not a sidewalk and in many cases 'no sidewalk' may imply a building or fence right up against the road. Millikan road can be very busy - separate to the legality, I would not prefer to cross it except at that junction, and would prefer to avoid walking in the road! (the route is completely intractable for a wheelchair!)

Whould you share the link with some added via points how you'd prefer to take the route (as I'm not sure I understand the ground truth situation fully :) )

@karussell Basically the north side of Millikan way essentially has no proper sidewalk. However, I do not know if any router (including GH) takes into account left/right sidewalks and hence can meaningfully represent where to cross from one side to another, at a junction, pedestrian crossing, or wherever. Having discussed this somewhat on irc (#osm), the best route would probably be something like:
https://graphhopper.com/maps/?point=Millikan Way Eastbound MAX Platform%2C 97077%2C Beaverton%2C United States of America&point=45.493341%2C-122.821704&point=45.493792%2C-122.826247&locale=en-US&vehicle=foot&weighting=fastest&elevation=true&use_miles=false&layer=Omniscale
This is based on the fact that there are pedestrian crossings (marked on the road, and tagged) at the 141st place junction (where the via point is), and a stop sign on the junction itself.
A number of people might choose to walk the currently-suggested route, since it is the most direct, but there is only a very rough un-paved "sidewalk" in front of 14305 on the north side and no path to speak of (just where people have cut across, themselves) in front of the milikan business center.
Note that I have since added a footway cut-through from the MAX station to schottky terrace, which I suspect would be preferred compared to the current suggestion, but still will require a 'short' section of either walking on a rough dirt/garden area or on the road in front of milikan business center, which I would suggest is not desirable for a router to suggest.
If travelling from that MAX station, and requiring to cross only at a light-controlled junction, that would probably involve going east to Hocken Ave! Alternatively, of course, a transit-aware router might suggest a different stop, or taking a bus from the transit center ;)
Again, please paste/link into a different issue if that's better!

takes into account left/right sidewalks

We do - see here

hence can meaningfully represent where to cross from one side to another

That is not done and is probably the biggest issue in this example (have therefor changed the title if that is okay for you).
Also this is a tricky example as the service road in my opinion is a less 'frequent' and 'safe' way to walk on. Is this really a service road?

I have read (and participated in!) quite a bit of these issues; my reference to left/right was specifically with regard to sidewalk=left|right, which I believe is not really touched on. IMO this may well be quite a significant development to implement, both as a non-GH developer (implementation) but also with the great potential, as I am not aware of any rules for routing in any other engine. I suppose java is not much different from C++, so one day I will have to look at the code and development environment myself ;)
Re title, that is more appropriate for the issue I reported, but I'm not sure if it covers your original example!
Re service road, do you mean "Southwest Millikan way transit center road" ? That is a one-way road with no sidewalks, and walking south along it does not feel very safe as the traffic is coming up behind you! (that is perhaps another 'safety' issue which could be included - preferring walking against the flow of traffic on a one-way road?). I did not pick it as a service road, but I'm not sure if you object to that tagging on the basis of foot routing issues (which I addressed above) or just if it is such a road?

I suppose java is not much different from C++, so one day I will have to look at the code and development environment myself ;)

Started with C++ too I can say it is similar but both have their quirks ;) - just let me know if you have questions (probably better in the forum)

Re title, that is more appropriate for the issue I reported, but I'm not sure if it covers your original example!

Uups, indeed. I'll rename once again and would ask you to create yet another issue?

do you mean "Southwest Millikan way transit center road" ?

I meant this service road here

preferring walking against the flow of traffic on a one-way road?

I'm not sure here as it is e.g. required by German law to walk on the left side of the road outside of a village (because they say it is safer)

but I'm not sure if you object to that tagging on the basis of foot routing issues (which I addressed above)

Yes, I just meant that the foot routing we expect service roads to be relative safe and so we have a hard time to reduce priority. But as I now understand: our aim should be to increase priority of the other road instead (?)

Re issue creation/split, while I appreciate your initial report might take priority, the majority of our discussion is on the newer title, so would it make sense to keep that here and duplicate the small portion of yours in another issue? (apologies for hijacking!)
That service road you indicate is apparently named as I said, on the general OSM map layer at least ;)
Re service roads being 'safe' and preferred, I see the general benefit and approach of doing that, including eg. that residential roads are similar. I think the big change that I'm seeing here and in foot routing generally via highways (roads) is that, as far as foot (or hike?) routing is concerned, explicit sidewalk tagging should override a lot of other priorities re class of road. For a fastest/shortest route (no difference for pedestrians?), a service or residential road with no sidewalks is less desirable than a major road with them.

so yes, prioritise the other road ;)

Re the 'left side of the road', I believe it is advised (not sure if law) in the UK to do the opposite (due to other traffic direction); I would say the general principle is to walk in the direction opposite the 'nearest' traffic flow, so you can see oncoming traffic on your side (excepting blind corners, maybe?). If walking 'sides' are included in the routing on sidewalk=no roads, that could be detailed in directions too, and factored in to the crossing rules? Re the one-way service no-sidewalk road, I hope you can see now that I just meant it is less safe walking with traffic coming from behind you - and that 'safety' rule could be another option (or included by default), so eg. travelling north on that service road would be considered much better than going south. This is only for roads with no sidewalk, of course, so as long as a sidewalk can be used, that would probably be the preferred road-section to use? (if safety is prioritised)

so would it make sense to keep that here and duplicate the small portion of yours in another issue

Sure, done :) #847

explicit sidewalk tagging should override a lot of other priorities re class of road.

Yes

For a fastest/shortest route - no difference for pedestrians

Not always. As soon as you have priorities like preferring safer or more silent roads, the 'short' route is shorter ;)

you can see oncoming traffic on your side

That is highly likely a research topic as you would need to take into account buildings, tunnels, road width, average speed, road borderline, ...

that could be detailed in directions too, and factored in to the crossing rules

yes, either this or directly model the "sideway roads" as separate ways, not sure.

I hope you can see now that I just meant it is less safe walking with traffic coming from behind you

Yes, I think I understand this better now :)

Thanks for #847 :)
Agreed that 'short' is affected by 'safe' :)
Re seeing traffic on the side you're walking, a quick search seems to agree, at least for US & UK - no idea about other areas, but I'd assume it was similar!
eg. http://www.saferoutesinfo.org/program-tools/what-side-road-should-i-walk
It could be a research topic - but also, could be somewhere for OSM tags to help hint (maybe) eg. if sidewalk=no (or none) then consider prefer_road_side=left|right, or failing that some crazy analysis of curves in the road, buildings, etc ;)
"sideway roads"? Do you mean the "not a sidewalk but people need to travel by foot here somehow" ways, added in the routing model?
Re implementation, do you have any ideas at this stage? Initially I imagine a duplicate graph for left & right, with extra connections at key points, but that probably isn't very memory-efficient. That said, my gut feeling would be that using attributes for each way to hold that info might make it very complicated - albeit perhaps more tractable ;)

"sideway roads"? Do you mean the "not a sidewalk but people need to travel by foot here somehow" ways, added in the routing model?

I mean what you described here:

Initially I imagine a duplicate graph for left & right, with extra connections at key points, but that probably isn't very memory-efficient.

my gut feeling would be that using attributes for each way to hold that info might make it very complicated - albeit perhaps more tractable ;)

Yeah, as always we have the choice of making the import complex or mess with the algorithm ;)
But as the side of the road is also important if no sidewalk is existing (postman delivery) we should probably try to tweak the algo. See #680

Another example: this route is an important connection from a popular residential area in Bielefeld to the main railway station. As soon as the intermediate point is removed, the route wrongly uses the other side of the main road. The sidewalk tags for the two sides of the road are set correctly, i.e. sidewalk:right everywhere except for a section of the southern half where the route uses the separate footway slightly off the road. The route with the intermediate point is what people use.

I'm working on a project with foot routing and we're using Graphhopper for it. One of the functionalities we needed was for road crossings to be taken into account to indicate the user where to cross the road, if it is needed. This seemed like the correct issue to post on.
What's the current state of this functionality? Is it being worked on? If not, could I try implementing it (should it be implemented in another routing profile & would you be able to support the functionality in future updates)?


Getbalance can't show balance for specific address

Getbalance can show only balance for account or total wallet. What if someone want to check balance of specific address?

This repository is unmaintained (see https://github.com/laanwj/bitcoin-python#not-actively-maintained) so there is no one to handle issues. Closing this, sorry.


Webpack,  datepickers dom not cleared when watcher detects changes

Running local version a dev server. When you save something and the page gets updated - a new instance of the datepicker gets added to the dom right under body tag every time until you fully reload the browser.

@Ketec the issue should be fixed by v2.0.4


Uncaught Error: Scss compiler error: Undefined variable: "$medium-and-down".

Just got this after updating Meteor to <IP_ADDRESS> and Materialize to 0.97.8.

+1

+1

@Dogfalo I can reproduce. This is NOT an issue with the Meteor update.
Seems like this is an issue with a Materialize SCSS definition in 0.97.8. Downgrading to 0.97.7 fixes this error.
Anyone upgrading meteor versions is also prompted for package updates, such as materialize:materialize.

How to downgrade? Mine will auto install latest materialize:materialize 0.97.8
Even if specify meteor add<EMAIL_ADDRESS>

@Phixiys
You have to ru meteor add<EMAIL_ADDRESS>You can read more about it here
https://atmospherejs.com/i/installing

@Dogfalo Yeah, searched stackoverflow and found it. Thank's though.
:-)

Meh, the easiest way is to manually edit your .meteor/versions file :stuck_out_tongue: Then meteor will install the proper Materialize package automatically without updating. This can even be done running hot.

https://github.com/Dogfalo/materialize/issues/3905#event-858604013 Yeah! we got a bug label. A solution is hopefully near!

Can't seem to reproduce locally. I fresh installed Node, NPM, and Meteor.
I did run into some issues when trying to install the Materialize package fresh. But after I installed the fourseven:scss package first, installing Materialize went smoothly.

I encountered the same error $medium-and-down... and downgraded materializecss to version 0.97.7.
But now I got another issue: Failed to decode downloaded font: ...Roboto...
As seen here -> #2217

submitted a pull request to correct this #3924

@acburst I did the same as you, and I definitely reproduce the same error.
@charlesdagenais  can you tell us a bit more about the fix?

@keyscores Looks like the error happen when sass try to compile sass/style.scss  . The compiler can't resolve any variable, because they're not imported. style.scss is only used for documentation website style. Renaming it to _style.scss and correcting the import statement in ghpages-materialize prevent it from been compiled to css

@charlesdagenais @acburst
Now I get
=> Errors prevented startup:

   While processing files with fourseven:scss (for target web.browser):
   /client/css/index.scss: Scss compiler error: Undefined variable:
   "$small-and-down".

   /client/css/login.scss: Scss compiler error: ".forgot .login-row" failed to
   @extend ".row".
   The selector ".row" was not found.
   Use "@extend .row !optional" if the extend should be able to fail.

   /client/css/map.scss: Scss compiler error: Undefined variable:
   "$small-and-down".

=> Your application has errors. Waiting for file change.

@tomscholz you forgot to rename /sass/style.scss  file to sass/_style.scss

@charlesdagenais Thats true, but it didn't change anything :/

@tomscholz Just tried your fork of the materialize package in my app and it's working perfectly fine. looks like the problem is in your index.scss and login.scss and map.scss

@charlesdagenais Do you spot any errors?
https://gist.github.com/tomscholz/ae4b3894e3e7963888f7cffdb9c5a892 // map.scss
https://gist.github.com/tomscholz/61883623a63e49c6dcc59159589abc89 // login.scss (I tried to replace the screen variable with the pure values)
https://gist.github.com/tomscholz/9606216e9112f1d74d4cd1c5851f9ef3 // index.scss

@tomscholz if you want to use variables in your custom scss files, you need to import them from the materialize package

But then I end up with a huuuuge compiled css file! I guess we can now close this issue and the rest is my problem ;)
Von: charlesdagenais<EMAIL_ADDRESS>Antworten an: Dogfalo/materialize<EMAIL_ADDRESS>Datum: Montag, 21. November 2016 um 19:50
An: Dogfalo/materialize<EMAIL_ADDRESS>Cc: Tom Scholz<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Betreff: Re: [Dogfalo/materialize] Uncaught Error: Scss compiler error: Undefined variable: "$medium-and-down". (#3905)
@tomscholz if you want to use variables in your custom scss files, you need to import them from the materialize package
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Hi everybody!
I have the same problem and I cannot find the style.scss file in my project directory.
Where are those files going from the .meteor directory ?
Thanks gor your help

@MuyBien Clone the materialize repo into the /packages folder inside your project folder.
Via shell:
cd /path/to/YOURPROJECTFOLDER
mkdir packages
cd packages
git clone https://github.com/Dogfalo/materialize.git

Then open this file inside the materialize folder in your packages folder.

and change
  name: 'materialize:materialize',  // http://atmospherejs.com/materialize/materialize

to something like this:
  name: 'tomscholz:materialize',  // http://atmospherejs.com/materialize/materialize

Then run:
meteor remove materialize:materialize
meteor add tomscholz:materialize

Than edit the style.scss
et voilà

Huuuuuuuuuuuu OK!!!
Thanks ! And when your PR will be accepted I can run:
meteor remove tomscholz:materialize
meteor add materialize:materialize

@MuyBien Yes ;D. Totally. You got it

https://github.com/Dogfalo/materialize/issues/2217#issuecomment-268100489

Both PR have been merged, will be closing this issue for now unless further issues pop up.


[BUG] azure-core-1.23.1 logs version conflict with Jackson 2.13.0, but that version is marked as supported in maven

Describe the bug
A new INFO warning appears when using Jackson libraries on version 2.13.0 and azure-core library on version 1.23.1. In maven azure-core 1.23.1 supports Jackson 2.13.0.
Exception or Stack Trace
2021-12-20 11:09:41,205 [http-nio-8080-exec-6] INFO  com.azure.core.implementation.jackson.JacksonVersion - Package versions: jackson-annotations=2.13.0, jackson-core=2.13.0, jackson-databind=2.13.0, jackson-dataformat-xml=2.13.0, jackson-datatype-jsr310=2.13.0, azure-core=1.23.1, Troubleshooting version conflicts: https://aka.ms/azsdk/java/dependency/troubleshoot

To Reproduce
Steps to reproduce the behavior:

Install azure-core 1.23.1 and Jackson 2.13.0.
Try to start the application, that INFO log appears

Code Snippet
The code that should be updated is in JacksonVersion.java verifier file, on line 28:
    private static final SemanticVersion MAX_SUPPORTED_VERSION = SemanticVersion.parse("2.12.5");

should instead be:
    private static final SemanticVersion MAX_SUPPORTED_VERSION = SemanticVersion.parse("2.13.0");

Expected behavior
A warning should not appear when having azure-core 1.23.1 and Jackson libraries on version 2.13.0, because this version is marked as supported for that version of azure-core.
Setup (please complete the following information):

OS: Ubuntu LTS 20.04
IDE: IntelliJ IDEA
Library/Libraries: com.azure:azure-core:1.23.1, com.fasterxml.jackson.core:jackson-core:2.13.0 and the rest of jackson libraries on 2.13.0
Java version: 11
App Server/Environment: Tomcat
Frameworks: None

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

[x] Bug Description Added
[x] Repro Steps Added
[x] Setup information Added

I've also prepared this pull request to fix this issue: https://github.com/Azure/azure-sdk-for-java/pull/26127

Thanks for reporting it @Kryword. The log you see is not an indication of any error, this is information in case we'll need to investigate any issues regarding serialization and version conflict.
I agree  MAX_SUPPORTED_VERSION needs to change for clarity, but functionally-wise currently only the major version of it is used
https://github.com/Azure/azure-sdk-for-java/blob/7a38eddc5cbdc43b5694a44cbdce91e0ff82332b/sdk/core/azure-core/src/main/java/com/azure/core/implementation/jackson/JacksonVersion.java#L90-L95
It should be logged once when your process starts (or when Azure SDKs are first time initialized) and never logged again if everything is fine.  Does it cause any inconvenience to you?

Hello, I think this recently merged MR should fix this issue: https://github.com/Azure/azure-sdk-for-java/pull/26129

@Kryword thanks, I'm going to close this issue then, and please reopen as you see fit


feat: Moonshot supported function call & function call unit tests

PR content:

Moonshot supported function call
function call unit tests

related issue: https://github.com/spring-projects/spring-ai/issues/1058

HI @mxsl-gr  and thank you for the update.
As we are deprecating AbstractFunctionCallSupport in favor of AbstractToolCallSupport would you be interested to re-work your PR to support AbstractToolCallSupport?
For examples, you can check the OpenAI/AzureOpenAI/Anthropic/Mistral/Ollama/VertexAIGemini function calling implementations, or ask me for support ;)
Would appreciate your help as you own those models.

hi, @tzolov
it worked for now, and this PR has been squash and force pushed

@mxsl-gr thanks, sorry I haven't gotten to merging this yet

@mxsl-gr thanks, sorry I haven't gotten to merging this yet

hi @markpollack , is ok, i went on a trip and just back now. please let me know if there's anything else to adjust.
other some PRs related to function calls: https://github.com/spring-projects/spring-ai/pull/1116, https://github.com/spring-projects/spring-ai/pull/1163

I'm merging this despite test failure as this PR has improved the support so significantly.  We can circle back to figure out what is going on with the function calling failure in the test.  The streaming function calling test did pass in my testing.

merged in 611c9493e372f96129dd1ada3a0ca82445ba5994


after type in git commit - my terminal crashed

VSCode Version: 1.37.0-insider
OS Version: Ubuntu WSL Remote on Windows_NT x64 10.0.18956

Steps to Reproduce:

Open a command window.
Change into a git directory.
Type in git commit - (No matter how, like type in g- and type it commit  between them)
The terminal will crash. The editor is fine.

Does this issue occur when all extensions are disabled?: Yes(I disabled everything except Remote:WSL because this happens in a WSL window)

Note: This will happen in other terminal window. I may need to open an issue in Microsoft/WSL repository. If you want, close this issue(because I don't know if I could close it here)

Spotted problem.

Possibly a set of dotfile causes this problem. The maintainers may close this issue.

@KsRyY How did you fix it? i have same issue.

Hi guys, i'm having the same problem here, what was the fix for it?

Also having the same issue.

FYI @craigloewen-msft

This seems to be related to an issue when running zsh.
We also have an issue on the WSL Github repo: https://github.com/microsoft/WSL/issues/4430
And there's one on the Terminal repo too: https://github.com/microsoft/terminal/issues/2518
You can follow those issues as well for more updates! :)


Refactoring

/cc @puffnfresh @jdegoes
I broke apart runUIWith into componentProcess and mainLoop in the hope that it would expose some structure we could exploit to build widgets. At least now we can use SF and Process to build the top level functions which update things like Ace and ECharts.
Thoughts?

Seems like a good place to start :+1:

Can you explain the simplest way to listen to all events?? It's still not clear to me. :frowning:

Sure - the function passed to runUIWith now takes the input which caused any rerendering to happen.

Ah, I missed that, great we'll give it a go!


fix: Move URI resolution to render() method

componentDidMount() is called once for each component. We can use it to initialize the ILocationService, but the workItemUrl needs to be evaluated for each work item, so it has now been moved to the render() method.
Fixes #88

I cannot test this right now, but this approach (in general) should fix the linked issue.

I just noticed that #85 tackles the same issue in a bit different way. Feel free to choose whichever is the better approach. 🙂

Thanks for your help @hangy ! We decided to go for the fix in #92 but your input helped to evaluate the options. New release coming soon!


Allow lowercase variables

What does this PR do?
This adds the ability to use lowercase variables to E2.
They syntax highlight correctly and should work everywhere PascalCased variables work.
Resolves #17
Only thing is vclk is a restricted variable name as it is used internally.
Currently not reserving any future keywords, shouldn't break that much code to add keywords later down the line.

[ ] Documentation changes
[x] Code changes

How did you verify your code works?

[x] I created tests and the test suite (e2test) pass on a local server

Marked as draft: Needs support for using lowercase identifiers for lambda / string calls.


adresses in draft messages not saved

Expected behavior
Copy addresses from existing message, pasted into new message, new draft message saved, all addresses should be saved.
Actual behavior
Only one of the addresses saved.
Steps to reproduce

Copy addresses
Paste into draft
Save draft
Open draft
Find addresses disappeared

Environment
K-9 Mail version:
5.403
Android version:
7.0
Account type (IMAP, POP3, WebDAV/Exchange):

I have had a similar issue. When composing an email and it gets set to draft the addresses don't stay. The last email I tried this on it did work so it's unknown if the issue has been fixed or it's a fluke.


Disable CONFIG_DISTRIBUTION_TRACKING

Please provide a description of this PR:
This is to disable config distribution tracking, which is not mature, especially on CPU andMem cost.
https://github.com/istio/istio/issues/24521
https://github.com/istio/istio/issues/38716#issuecomment-1119187983
It is used for debug interface, so I think not much risk

@hzxuzhonghu: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-pilot_istio
6ba31e5748fa38162b93a269b59ef043bbe19836
link
true
/test integ-pilot

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@hzxuzhonghu: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-pilot_istio
6ba31e5748fa38162b93a269b59ef043bbe19836
link
true
/test integ-pilot

integ-cni_istio
6ba31e5748fa38162b93a269b59ef043bbe19836
link
true
/test integ-cni

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Should this have a release notes?

I think it's also used by istioctl wait  so it's not just debug interface.
I am not necessarily opposed to it though as wait also has some unaddressed issues making it unreliable

Yeah, since we all agreed to disable it, will add a release note

As pointed out, this change breaks istioctl functionality, which is relied on widely in docs tests.  We should either revert this commit or update our docs and go through the process of deprecating the command.

We can definitely disable doc tests or set env for the test

This did indeed cause a test failure: https://github.com/istio/istio.io/issues/11780
That is a release blocker PR as https://preliminary.istio.io/latest/docs/ops/configuration/mesh/config-resource-ready/ is no longer accurate for the main branch. I'm not sure if it's adequate so have the user have to add another set or if that would work.


Partial App Loading Example ES6 Classes

Looking at https://github.com/rackt/react-router/tree/0.13.x/examples/partial-app-loading gives you a good idea of how to do the dynamic component loading controlled by the router. However, this uses the "old style" React.createClass() style and not ES6 classes.
Does anyone have an example of doing the same feature but using ES6 classes? I haven't been able to get it to work.

You can check out https://github.com/dferber90/webapp-starter-pack. It’s WIP, but I already have the code splitting part partially done. Should be good enough as an example.

I should have specified I wanted to use react-router 0.13.x. It's hard to get sign-off on beta versions of modules in a corporation. Also, this was more a comment on updating documentation and providing a simple example.
Thanks for the link though, I will check it out.


Shared location for VSTSSDK

Environment
azure-pipelines-task-lib version:  current
Issue Description
I'm writing an extension with multiple tasks, it feels very hackish to include a ps_modules folder in every single task. I'm unable to find an example where the SDK is shared for several tasks. I have attempted to define it in the files section of the vss-extension file, but that appears to not do anything, or I'm defining it incorrectly.
Expected behaviour
I would love to be able to add this into my files section in vss-extension.json so that all my shared powershell modules would have access to it.
  {
   "path": "mytask/ps_modules"
  }

Actual behaviour
It expects to have the module file locally in each task.

##[error]File not found: 'D:\a\_tasks\MyTask_00f968f6-fb8b-475a-903b-7b492eb7d48f\1.0.0\ps_modules\VstsTaskSdk\VstsTaskSdk.psd1'

Logs

You unfortunately can't do that. To better understand, when installing the extension, each task is installed separately by zipping up the contents of the contribution into a taskzip. It's that taskzip that's installed into the ProjectCollection/Organization and it's that taskzip that's sent to the agent.

Hi @jeffpatton1971, do you have any other question?

Thanks for this, I do have a follow up, that is slightly related. Is there a way to have the task run PowerShell core? The code I'm using has .net 7.27 and it does not run properly on the regular PowerShell.

Not directly, but you can call & pwsh ./your.other.ps1 from the script that is called by PowerShell 5.

Not directly, but you can call & pwsh ./your.other.ps1 from the script that is called by PowerShell 5.

OOOO i didn't think of that! good call, thanks!


[Feature][MasterServer] Add cache manager for Master

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Description
Now when Master run a workflow, it will do many db select operation, such as processDefinition, taskDefinition, processTaskRelation, tenant, user, workGroup, project......
I think these data will not be changed easily, so if we can use cache instead of selecting from db, it will speed up the processing of workflow and improve performance.
function design:

cache config for enable switch and other params like expire time, cache size.
cache annotation, which can be use easily, and it's helpful to manage the unified cache logic -> use spring cache + caffeine
cache hit rate statistics
cache update notify, now is from api-server

sub-issue:

[x] #6985
[ ] #6987
[ ] #6988

Use case
No response
Related issues
No response
Are you willing to submit a PR?

[x] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Now when Master run a workflow, it will do many db select operation, such as processDefinition, taskDefinition, processTaskRelation, tenant, user, workGroup, project......

If the cache is only for DB operations, maybe we can investigate whether the ORM framework supports cache itself, I remember MyBatis supports some basic cache mechanism.

function design:

[ ]  add the cache switch, it can be controlled by user freely. If false, the cache will not effect
[ ]  cache expire config, it will decide how long the cache will last
[ ]  cache annotation, which can be use easily, and it's helpful to manage the unified cache logic
[ ]  cache hit rate statistics
[ ]  cache update notify, now is from api-server

All the functionalities above can be easily achieved by introducing
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-cache</artifactId>
</dependency>

@caishunfeng which cache system do we want to use?

@kezhenxu94 yes, I found too, the spring cache maybe is suitable.

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-cache</artifactId>
</dependency>

try to take #6987

try to take #6987

OK, looking forward to your PR.


How to slice a tensor for custom loss?

Trying to implement a custom loss function that works as following in Theano:
        def sparse_sqe(y_true, y_pred):
            ix = y_true.nonzero()
            return (y_pred[ix] - y_true[ix]) ** 2

e.g. y_true = [[0, 0, 0, 0, 5], [9, 0, 0, 0, 0]]
y_pred = [[0.2, 0.1, 0.2, 0, 4.8], [9.5, 0.3, 0.1, 0, 0]]
would return ([4.8, 9.5] - [5, 9]) ** 2 = 0.29
The loss should only be calculated for the y_pred values at the index of the nonzero y_true values.
I tried
z = tf.constant(0, tf.float32)
ix = tf.where(tf.not_equal(y_true, z))
(y_pred[ix] - y_true[ix]) ** 2

but you cannot slice like a numpy array. I also tried tf.gather, but this applies to a collection of tensors, and it will slice the vectors, instead of the elements of those vectors.
Any help would be greatly appreciated.

In the future you should ask usage questions like these on StackOverflow. However...
You probably are looking for gather_nd does indirect indexing,. On the other hand, that is not necessary and gather_nd does not yet have a gradient. So instead do
(y_true - tf.cast(tf.not_equal(y_true, 0), tf.float32) * y_pred)**2


Add more Aspnetcore namespace docs

Addresses the remaining items in https://github.com/dotnet/AspNetApiDocs/compare/wtgodbe/Docs?expand=1

Docs Build status updates of commit c78ddfb:
:warning: Validation status: warnings

File
Status
Preview URL
Details

aspnet-core/xml/Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption/EncryptionAlgorithm.xml
:warning:Warning
View
Details

aspnet-core/xml/Microsoft.AspNetCore.Razor/PartialParseResult.xml
:warning:Warning
View
Details

aspnet-core/xml/Microsoft.AspNetCore.Mvc/CompatibilityVersion.xml
:warning:Warning
View
Details

aspnet-core/xml/Microsoft.AspNetCore.Razor.TagHelpers/TagStructure.xml
:warning:Warning
View
Details

aspnet-core/xml/Microsoft.AspNetCore.Razor.Language/TagStructure.xml
:warning:Warning
View
Details

aspnet-core/xml/Microsoft.Extensions.Logging.Console/LoggerColorBehavior.xml
:warning:Warning
View
Details

aspnet-core/xml/Microsoft.Extensions.DependencyInjection/ServiceLifetime.xml
:warning:Warning
View
Details

aspnet-core/xml/ns-Microsoft.AspNetCore.CookiePolicy.xml
:white_check_mark:Succeeded
View

aspnet-core/xml/ns-Microsoft.AspNetCore.Cors.Infrastructure.xml
:white_check_mark:Succeeded
View

aspnet-core/xml/ns-Microsoft.AspNetCore.Cors.xml
:white_check_mark:Succeeded
View

aspnet-core/xml/ns-Microsoft.AspNetCore.Http.Headers.xml
:white_check_mark:Succeeded
View

aspnet-core/xml/ns-Microsoft.AspNetCore.HttpOverrides.xml
:white_check_mark:Succeeded
View

aspnet-core/xml/ns-Microsoft.AspNetCore.HttpsPolicy.xml
:white_check_mark:Succeeded
View

aspnet-core/xml/Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption/EncryptionAlgorithm.xml

Line 0, Column 0: [Warning-ECMA2Yaml_Enum_NoRemarks] Please note: <remarks> node on Enum fields will be ignored.

aspnet-core/xml/Microsoft.AspNetCore.Razor/PartialParseResult.xml

Line 0, Column 0: [Warning-ECMA2Yaml_Enum_NoRemarks] Please note: <remarks> node on Enum fields will be ignored.

aspnet-core/xml/Microsoft.AspNetCore.Mvc/CompatibilityVersion.xml

Line 0, Column 0: [Warning-ECMA2Yaml_Enum_NoRemarks] Please note: <remarks> node on Enum fields will be ignored.

aspnet-core/xml/Microsoft.AspNetCore.Razor.TagHelpers/TagStructure.xml

Line 0, Column 0: [Warning-ECMA2Yaml_Enum_NoRemarks] Please note: <remarks> node on Enum fields will be ignored.

aspnet-core/xml/Microsoft.AspNetCore.Razor.Language/TagStructure.xml

Line 0, Column 0: [Warning-ECMA2Yaml_Enum_NoRemarks] Please note: <remarks> node on Enum fields will be ignored.

aspnet-core/xml/Microsoft.Extensions.Logging.Console/LoggerColorBehavior.xml

Line 0, Column 0: [Warning-ECMA2Yaml_Enum_NoRemarks] Please note: <remarks> node on Enum fields will be ignored.

aspnet-core/xml/Microsoft.Extensions.DependencyInjection/ServiceLifetime.xml

Line 0, Column 0: [Warning-ECMA2Yaml_Enum_NoRemarks] Please note: <remarks> node on Enum fields will be ignored.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Docs Build status updates of commit da45f55:
:warning: Validation status: warnings

File
Status
Preview URL
Details

aspnet-core/xml/Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption/EncryptionAlgorithm.xml
:warning:Warning
View
Details

aspnet-core/xml/Microsoft.AspNetCore.Razor/PartialParseResult.xml
:warning:Warning
View
Details

aspnet-core/xml/Microsoft.AspNetCore.Mvc/CompatibilityVersion.xml
:warning:Warning
View
Details

aspnet-core/xml/Microsoft.AspNetCore.Razor.TagHelpers/TagStructure.xml
:warning:Warning
View
Details

aspnet-core/xml/Microsoft.AspNetCore.Razor.Language/TagStructure.xml
:warning:Warning
View
Details

aspnet-core/xml/Microsoft.Extensions.Logging.Console/LoggerColorBehavior.xml
:warning:Warning
View
Details

aspnet-core/xml/Microsoft.Extensions.DependencyInjection/ServiceLifetime.xml
:warning:Warning
View
Details

aspnet-core/xml/ns-Microsoft.AspNetCore.CookiePolicy.xml
:white_check_mark:Succeeded
View

aspnet-core/xml/ns-Microsoft.AspNetCore.Cors.Infrastructure.xml
:white_check_mark:Succeeded
View

aspnet-core/xml/ns-Microsoft.AspNetCore.Cors.xml
:white_check_mark:Succeeded
View

aspnet-core/xml/ns-Microsoft.AspNetCore.Http.Headers.xml
:white_check_mark:Succeeded
View

aspnet-core/xml/ns-Microsoft.AspNetCore.HttpOverrides.xml
:white_check_mark:Succeeded
View

aspnet-core/xml/ns-Microsoft.AspNetCore.HttpsPolicy.xml
:white_check_mark:Succeeded
View

aspnet-core/xml/Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption/EncryptionAlgorithm.xml

Line 0, Column 0: [Warning-ECMA2Yaml_Enum_NoRemarks] Please note: <remarks> node on Enum fields will be ignored.

aspnet-core/xml/Microsoft.AspNetCore.Razor/PartialParseResult.xml

Line 0, Column 0: [Warning-ECMA2Yaml_Enum_NoRemarks] Please note: <remarks> node on Enum fields will be ignored.

aspnet-core/xml/Microsoft.AspNetCore.Mvc/CompatibilityVersion.xml

Line 0, Column 0: [Warning-ECMA2Yaml_Enum_NoRemarks] Please note: <remarks> node on Enum fields will be ignored.

aspnet-core/xml/Microsoft.AspNetCore.Razor.TagHelpers/TagStructure.xml

Line 0, Column 0: [Warning-ECMA2Yaml_Enum_NoRemarks] Please note: <remarks> node on Enum fields will be ignored.

aspnet-core/xml/Microsoft.AspNetCore.Razor.Language/TagStructure.xml

Line 0, Column 0: [Warning-ECMA2Yaml_Enum_NoRemarks] Please note: <remarks> node on Enum fields will be ignored.

aspnet-core/xml/Microsoft.Extensions.Logging.Console/LoggerColorBehavior.xml

Line 0, Column 0: [Warning-ECMA2Yaml_Enum_NoRemarks] Please note: <remarks> node on Enum fields will be ignored.

aspnet-core/xml/Microsoft.Extensions.DependencyInjection/ServiceLifetime.xml

Line 0, Column 0: [Warning-ECMA2Yaml_Enum_NoRemarks] Please note: <remarks> node on Enum fields will be ignored.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

@Rick-Anderson is it alright if I merge this?

@Rick-Anderson is it alright if I merge this?

@Rick-Anderson is it alright if I merge this?

Yup, that's what the green check means

@Rick-Anderson is it alright if I merge this?

Yup, that's what the green check means


Separate http and ws metrics

Problem
Currently the http and ws server both utilize the relay and other business logic.
However, since the registries are created in the classes it's not clear the metrics are intended for the ws server when it's running
Solution
Separate metrics flags or make sure there's a different layer for calls originating for one over the other
Alternatives
No response

Just one idea, there are some default labels that are added when ingesting the metrics using prometheus
I believe that currently the labels "namespace" and "cluster" are being added in that way, just an idea, but could be possible to add a label, either in code or at ingestion via  prometheus configuration for service-type and current values could be api, websocket


⚠ Bofrost.fr has degraded performance

In 3636a8d, Bofrost.fr (https://www.bofrost.fr) experienced degraded performance:

HTTP code: 200
Response time: 972 ms

Resolved: Bofrost.fr performance has improved in 1686256 after 13 minutes.


Add mod specific endpoints & ReadMe update

I have looked through all used endpoints of the qmod. For many of them all data is used and only for the three created here I saw a greater added value due to a lower network traffic.
Also, this PR adds an array of all rank votes to the new map/modinterface endpoint.
While editing I noticed that some code is repetitive or doesn't use async for sql queries. So that there are not more changes in the PR at the end, I have not touched these places for now. Should I change them next time or just focus on the content of the PR?

If you want to fix some repetitive stuff or performance issues - feel free to do so in a separate PR. It's just easier to manage this way.👍


Getting device Error

DeviceError: The analytic argument has been replaced by shots=None. Please use shots=None instead of analytic=True.

using pennylane-honeywell-0.15.0
Installing collected packages: pennylane-honeywell
Successfully installed pennylane-honeywell-0.15.0

this is my code dev = qml.device("honeywell.hqs", machine='HQS-LT-S1-APIVAL',wires = 5)

Hi @ab124585, what version of PennyLane are you using? The analytic keyword was removed from PennyLane itself in version 0.15, so if you happen to be using an earlier version, you will need to update (the current version is 0.16). Also, does the error occur immediately when you create the device with the line of code you provided?

python3.7/site-packages (0.16.0)

Name: PennyLane
Version: 0.16.0
Platform info:           Linux-4.14.232-177.418.amzn2.x86_64-x86_64-with-glibc2.2.5
Python version:          3.7.10

honeywell.hqs (PennyLane-Honeywell-0.11.0)

Hi @ab124585 , thanks for the info! It looks like the Honeywell plugin just needs to be updated in your environment; the most recent version is 0.15.0. Please try
pip install --upgrade pennylane-honeywell

and let us know if you are still running into trouble.

Updated - honeywell.hqs (PennyLane-Honeywell-0.15.0). Let me check if that resolved the problem

it resolved the Error. I am gettign No valid api key for HQS platform found

Where should I set up my HW credential? It is not clear from the docs. I am on linux container.

I only see example of config.toml for IBM access but not for HW

can I have example of config.toml for honeywell?

it resolved the Error.

Great! Thanks for letting us know.

Where should I set up my HW credential? It is not clear from the docs. I am on linux container.

Right, you will need to obtain an API key directly from Honeywell to access their devices. Once you've got that, there are three options:

set the environment variable HQS_TOKEN
pass the API key directly to the device upon construction using the api_key argument (device = ('honeywell.hqs', api_key = "...")
use the config.toml file. It be much the same as the Qiskit example, except it should be a [honeywell.global] section with the variable api_key = "..." set beneath.

ty. Let me try

Per HW they are not supporting api_key any more only id (e-mail) and pwd

Per HW they are not supporting api_key any more only id (e-mail) and pwd

Thanks so much for bringing this to our attention! Our team will look into it :slightly_smiling_face:

Thanks. Waiting for new release :)

Hi @ab124585, thank you for uncovering this issue!
Short update here: we've started investigating how the new API of Honeywell will integrate with our plugin and likely a more substantial change will be required to how the PennyLane-Honeywell device handles API requests. We have ideas for going forward, it will likely just require some time to get the new features required for the new API support through the implementation and the review phases. We'll be updating this issue once the support is ready. :)

Hi @ab124585, we've merged an update into the master branch that should help with connecting to the Honeywell service using the new API. To use this feature, after cloning the repository the master branch can be checked out locally and installed via pip install -e .. We'll include this feature in the upcoming release.
To provide credentials when connecting, the email registered with Honeywell needs to be specified. One way to specify the email is to pass it as the user_email to the device. Later, on the first login (and roughly every 30 days) the associated password will be asked to connect to the Honeywell service.
I'll close this issue for now, but let us know how connecting goes! :slightly_smiling_face:


[Checkbox] Update state when 'checked' props change

Continuation of #1403.
@oliviertassinari Please take a look.

@oliviertassinari Please do a final review and merge it if it's alright.

@subjectix @salarkhan Thanks

Thank you :+1: :tada:


AVG Antivir treats SublimeTextTrans as a virus

Two days ago Avira started to think that SublimeTextTrans has a virus, is it safe to use?

I don't know if it has to do with this package. If that is the case, that can only be a false positive related to SetSublimeLayered.exe executable included in this package.
That executable is very tiny (1024 bytes size), and its only purpose is to change SublimeText window into layered mode so it can be applied opacity.
Unfortunately, sometimes some antiviruses will erroneously detect it as malware.


workflow: enhance runner self-initialization

e.g, if some docker images are missing, the workflow should be able to check them and pull them automatically, rather than simply being failed.

The related issue:
https://github.com/alibaba/inclavare-containers/issues/1055


Windows - Under FAQ: A few  spelling errors

Under FAQ: "Make sure that you remove all traces on your computer once en export has been done." Spelling error. I guess you know you should do some change on the "About" page. A few other spelling errors too.

Fixed! Thanks Loop!


Allow to configure Server fluently

As #112, users can configure Server using configurator, but they should use a DSL way.
Instead, some users may want to configure Server fluently, like:
Gateway.builder()
       .server().http(8080)
                .https(8443)
                .tlsSelfSigned()
                .then() // ???
       .upstream("/foo", Upstream.of("http://example.com")
       .build();

I could have done together in #112, but there was no proper naming for a method returning to GatewayBuilder. Here are my candidates: then, and, gateway, build, ...

Seems like build(), standby() and start() 😆


TintTransformation.HexColor Binding

Description
I would like to bind a fft:TintTransformation.HexColor to a constant string or DynamicResource
Steps to Reproduce
<fft:TintTransformation
    HexColor="{Binding Source={x:Static res:Colours.ICON_TINT_HEX}}"
    EnableSolidColor="true" />

Expected Behavior
It binds.
Actual Behavior
No property, bindable property, or event found for 'HexColor', or mismatching type between value and property

I have the same problem

Idem here.

I found the reply is here
https://github.com/luberda-molinet/FFImageLoading/issues/491#issuecomment-279891483

@MosCD Thanks!

Yes, transformations properties are not bindable as they must also work in a non Xamarin.Forms projects. But that will change soon, as we're including Transformations packages in a base nugets, so we can add Xamarin.Forms specific Transformations for that which can have bindable properties :) I created issue for this: #499


How to integrate systeminformation?

@frederikbugglin We have a page called "Systeminformation" at the path /systeminformation . It is currently not integrated into the menu system because it is mostly usefull for us to review the system configuration, check if we are on the newest version etc. I think we will expand that page in the future a bit, similar to "System Status" or "Health Check" pages of other systems.
Do you think we should integrate that page as a link somewhere or leave it as is for now? I is not too relevant for end-users I think.

Also: We might want some information about background-jobs, in order to detect problems e.g. with synchronization.

Example:

Okay, we have now a basic version of this inside the basxconnect project. I think we will move the according views inside bread.

Closing in favor of #91


order doesn't support number types

I was writing a unit test for the peach builtin to improve unit test coverage since that command had zero test coverage. That caused me to notice this unexpected behavior:
> range 5 | peach [x]{ + 1 $x } | order
Exception: bad value: inputs to "order" must be comparable values, but is uncomparable values

Ideally the number comparison used by order would be unified with the numeric comparison commands in pkg/eval/builtin_fn_num.go.


Authentication without SSL certificate validation

Hello the Jenkins server is using https. Unfortunately I can't use the lib as it is.
Inside the libraries that Jenkinsapi use there is an option Check_SSL="True".
Can somehow expose this to your api ?

import jenkinsapi
print(jenkinsapi.version)
0.2.26
from jenkinsapi.jenkins import Jenkins
J = Jenkins(baseurl='https://myurl')
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/dist-packages/jenkinsapi/jenkins.py", line 51, in init
JenkinsBase.init(self, baseurl, poll=not lazy)
File "/usr/local/lib/python2.7/dist-packages/jenkinsapi/jenkinsbase.py", line 34, in init
self.poll()
File "/usr/local/lib/python2.7/dist-packages/jenkinsapi/jenkinsbase.py", line 57, in poll
data = self._poll(tree=tree)
File "/usr/local/lib/python2.7/dist-packages/jenkinsapi/jenkinsbase.py", line 67, in _poll
return self.get_data(url, tree=tree)
File "/usr/local/lib/python2.7/dist-packages/jenkinsapi/jenkinsbase.py", line 77, in get_data
response = requester.get_url(url, params)
File "/usr/local/lib/python2.7/dist-packages/jenkinsapi/utils/requester.py", line 95, in get_url
return requests.get(self._update_url_scheme(url), **requestKwargs)
File "/usr/lib/python2.7/dist-packages/requests/api.py", line 55, in get
return request('get', url, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 456, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 559, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 382, in send
raise SSLError(e, request=request)
SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)

You can use ssl_verify=False as ma,ed c'tor parameter.


chore: increase logLevel for lambda functions

:tada: This PR is included in version 1.2.3 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Stepping away from the project.

Greetings 👋🏻
I'm @djdefi and I created the Docker image for Cachet back in 2015 to implement Cachet as a status page for the company I worked for at the time, and have been the semi-official maintainer of this Docker image since.
Shortly after that, I took a new job as a Support Engineer at GitHub, and no longer personally had a need to run Cachet. I continued to push the Docker image along as a side project when I was able. As time has gone on, I've been able to contribute less and less, and with no "production" need for it, I essentially stopped being active here.
Just wanted to drop a note here to say that I no longer have access to the repository or Docker hub, and I don't know what the maintainership status is going forward.
Thank you to everyone who contributed over the years, hopefully this project was helpful to you in some way. This was the largest open source project I have ever attempted to maintain, and ya'll made it possible.

☮ ❤ :octocat:  @djdefi

Greetings 👋🏻
I'm @djdefi and I created the Docker image for Cachet back in 2015 to implement Cachet as a status page for the company I worked for at the time, and have been the semi-official maintainer of this Docker image since.
Shortly after that, I took a new job as a Support Engineer at GitHub, and no longer personally had a need to run Cachet. I continued to push the Docker image along as a side project when I was able. As time has gone on, I've been able to contribute less and less, and with no "production" need for it, I essentially stopped being active here.
Just wanted to drop a note here to say that I no longer have access to the repository or Docker hub, and I don't know what the maintainership status is going forward.
Thank you to everyone who contributed over the years, hopefully this project was helpful to you in some way. This was the largest open source project I have ever attempted to maintain, and ya'll made it possible.

peace_symbol heart :octocat:  @djdefi

Sad to see you go, but I hope you will have a lot of fun and success going forward! I personally want to thank you for maintaining the docker image as I use it at the co. I work at. Really appreciate your work!
P.S.: Going to try and fix the current problems and make a PR next week.


Add sim_afu

sim_afu loads the ASE version of libopae-c, turning software compiled for use
with hardware into a simulation run.

What do you all think of this? I've been struggling with the problem of having to write Makefiles for examples that either build both ASE and FPGA versions of a binary or a mode that builds one or the other. This script lets us just compile for FPGA and invoke it for ASE by forcing it to load libopae-c-ase.
Any suggestions for the script name? sim_afu is ok, but maybe there is something better.

Why can't you use a flag in the Makefile? E.g. make USE_ASE=1 builds the ASE version.
If you still want to keep the script, how about "run_as_ase", "run_in_ase" or "use_ase" instead of sim_ase?

I like "run_in_ase".
The reason I don't like the USE_ASE=1 flag, especially, is because it builds a single named binary that has different semantics depending on the flag. You never know how it was compiled last, so are forced to do "make clean" every time you do a make in order to be sure it is compiled how you need it.

I also like run_in_ase and this script in general.
It'd be an atypical case, but I think we should extend LD_PRELOAD rather than overriding entirely:
env LD_PRELOAD="libopae-c-ase.so${LD_PRELOAD:+ $LD_PRELOAD}" "$@"

Is there a consistent methodology on the motivations for naming these scripts? I think it would be useful to see what is being envisioned here.

Not to dwell on something inconsequential, but another name occurred to me. How does plain with_ase sound?
As a sanity check to potentially add later, it might be worth verifying that libopae-c-ase.so appears on the search path prior to launching the command.

@chipb I think you are probably right about with_ase. I was finding run_in_ase tedious to type.
Any disagreement? Not of deep consequence, but typed a lot so worth a couple extra minutes.

I'm fine with with_ase

@OPAE/ci waiting for you to approve one line make change.


Add MergeCal to open source projects

name: Add a new project
about: Suggest a new Django project for the Awesome Django list
title: "[NEW] Project Name"
Project Information

Project Name:
MergeCal

Project URL:
https://github.com/abe-101/mergecal
https://mergecal.habet.dev

Description:
(Please provide a brief description of the project.)

Criteria
Please answer the following questions about the project you are submitting. This will help us evaluate if the project should be included in the Awesome Django list.
Note: If your project is only a few months old or has under 50 to 100 stars on GitHub, it may NOT meet the criteria for inclusion in the Awesome Django list. We recommend that you promote your project more and then consider a submission at a later date when it has gained more visibility and community support.

Is the project new?

[ ] Yes
[x] No

How long has the project been maintained?
(Please provide an approximate duration in months or years.)
2 years

How many releases has it had if it's a library or package?
(Please provide the number of releases, or link to the project's release history.)
its a webapp

Are you the author or are you submitting the project on behalf of a company?

[x] I am the author
[ ] I am submitting on behalf of a company
[ ] Other (please specify)

What makes it awesome?
(Please provide a brief explanation of why you believe this project is a valuable addition to the Awesome Django list.)
It's a elegant example demonstrating how Django is a great choice for building webapps

Additional Information
Please provide any additional information you believe is relevant to the project or its evaluation for inclusion in the Awesome Django list. This might include information about the project's documentation, test coverage, community support, or any unique features that set it apart from other Django projects.
(Please provide any relevant additional information here.)
Thank you for your submission!

Hi @abe-101, This is a very cool project. We'll feature it in Django News next week. I think it's still not quite big enough for Awesome Django; unfortunately, we have to draw the line somewhere or the curated list would quickly turn into Django Packages. Hopefully, the newsletter placement brings you a lot more eyeballs.


Remove pervasives when appropriate.

This removes the leading Pervasives module when matching on built-in types. It solves issues with ppx_import on 4.03 (where types are fully qualified).

Also, any tests?

It's surprisingly annoying to test, since only ref (and result) is in pervasive. I added a test with show.

I pushed more test and fixed deriving fold for ref.

Thanks


[1.10 WIP] Custom checks for DC/OS

Description

https://jira.mesosphere.com/browse/DOCS-1987
Urgency

[ ] Blocker
[x] High
[ ] Medium

Requirements

Test all commands and procedures.
Build content locally and test for formatting/links.
Add redirects to dcos-website/redirect-files.
Change all affected versions (e.g. 1.7, 1.8, and 1.9).
See the contribution guidelines.

@sascala when you get a chance, please review. thanks!


Error: "ignore unknown room" while using socket.io-emitter

`let io = require('socket.io-emitter')(redis_config);
io.to(socket_id);`
https://github.com/socketio/socket.io-emitter/blob/master/index.js#L145

Closed by https://github.com/socketio/socket.io-emitter/pull/55.
Thanks for the heads-up :+1:


CI: capture additional information if the server tests fail

Is your feature request related to a problem? Please describe.
Occasionally server tests (make pytest) fail because a new version of a package dependency was released. We should consider adding an additional CI step to https://github.com/elyra-ai/elyra/blob/main/.github/workflows/build.yml#L76-L95, which is only executed if the previous step https://github.com/elyra-ai/elyra/blob/main/.github/workflows/build.yml#L94 failed. This step could collect additional information that would enable us to determine more quickly if a failure might be the result of a dependency change that the build picked up.
Describe the solution you'd like
A clear and concise description of what you want to happen.

append a step to https://github.com/elyra-ai/elyra/blob/main/.github/workflows/build.yml#L76 that is executed only if https://github.com/elyra-ai/elyra/blob/main/.github/workflows/build.yml#L94 fails
the step parses the output of pip freeze, retrieves for each package its metadata by sending an HTTP GET request to https://pypi.org/pypi/{PKG_NAME}/json, scans the release history and issues a message if a new package version was released within the past X days

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

I'd like to do this. PR later today


Resolve ActiveRecord schema format deprecation

Running the ar:migrate task with ActiveRecord 7.0 will provide the following deprecation warning:
DEPRECATION WARNING: ActiveRecord::Base.schema_format is deprecated and will be removed in Rails 7.1.
Use `ActiveRecord.schema_format` instead.

Now that 7.1 is released, ActiveRecord::Base.schema_format has been removed in the latest version of ActiveRecord.
This checks to see if at least ActiveRecord version 7.0 is being used. If not, there is no change to the task. However, if so, it now uses ActiveRecord.schema_format to check if the schema dump task should be invoked.

The test suite failures are unrelated, but addressed in #2276.

Thanks!


Several questions to long_term_average

Info:
Currently I try to calculate the LTA as a reference for the enso_nino34-operation.
Approaches:
CLI:
cate --traceback res set cl_lta long_term_average source='esacci.CLOUD.mon.L3C.CLD_PRODUCTS.AVHRR.NOAA-17.AVHRR_NOAA.1-0.r1' year_min=2007 year_max=2008 file='/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/UC06/cl_lta.nc' var='cc_total'
Returns:
LTA: done
....traceback_start
Traceback (most recent call last):
File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/util/cli.py", line 274, in run_main
args_obj.command_class().execute(args_obj)
File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/util/cli.py", line 171, in execute
return sub_command_function(command_args)
File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/cli/main.py", line 945, in _execute_set
monitor=cls.new_monitor())
File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/webapi/wsmanag.py", line 171, in set_workspace_resource
data=self._post_data(op_name=op_name, op_args=json.dumps(op_args)))
File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/webapi/wsmanag.py", line 63, in _fetch_json
raise error_type(message)
cate.core.workspace.WorkspaceError: ('Unsupported protocol', None)
Cate WebAPI service traceback
Traceback (most recent call last):
File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/webapi/rest.py", line 240, in post
monitor=_new_monitor())
File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/core/wsmanag.py", line 307, in set_workspace_resource
workspace.execute_workflow(res_name=res_name, monitor=monitor)
File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/core/workspace.py", line 520, in execute_workflow
Workflow.invoke_steps(steps, value_cache=self._resource_cache, monitor=monitor)
File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/core/workflow.py", line 577, in invoke_steps
steps[0].invoke(value_cache=value_cache, monitor=monitor)
File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/core/workflow.py", line 911, in invoke
return_value = self._op_registration(monitor=monitor, **input_values)
File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/core/op.py", line 225, in call
return_value = operation(**input_values)
File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/ops/average.py", line 124, in long_term_average
data_source.sync(dt_range, monitor=monitor.child(work=step * 0.9))
File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/ds/esa_cci_odp.py", line 481, in sync
raise ValueError('Unsupported protocol', protocol)
ValueError: ('Unsupported protocol', None)
cate res: error: ('Unsupported protocol', None)
Cate WebAPI service traceback
Traceback (most recent call last):
File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/webapi/rest.py", line 240, in post
monitor=_new_monitor())
File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/core/wsmanag.py", line 307, in set_workspace_resource
workspace.execute_workflow(res_name=res_name, monitor=monitor)
File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/core/workspace.py", line 520, in execute_workflow
Workflow.invoke_steps(steps, value_cache=self._resource_cache, monitor=monitor)
File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/core/workflow.py", line 577, in invoke_steps
steps[0].invoke(value_cache=value_cache, monitor=monitor)
File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/core/workflow.py", line 911, in invoke
return_value = self._op_registration(monitor=monitor, **input_values)
File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/core/op.py", line 225, in call
return_value = operation(**input_values)
File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/ops/average.py", line 124, in long_term_average
data_source.sync(dt_range, monitor=monitor.child(work=step * 0.9))
File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/ds/esa_cci_odp.py", line 481, in sync
raise ValueError('Unsupported protocol', protocol)
ValueError: ('Unsupported protocol', None)
....traceback-end
API:
from cate.core.ds import DATA_STORE_REGISTRY
import cate.ops as ops
from cate.util import ConsoleMonitor
monitor = ConsoleMonitor()
data_store = DATA_STORE_REGISTRY.get_data_store('esa_cci_odp')
local_store = DATA_STORE_REGISTRY.get_data_store('local')
name_oz = 'esacci.OZONE.mon.L3.NP.multi-sensor.multi-platform.MERGED.fv0002.r1'
tr_uc6 = '2007-01-01, 2008-12-31'
vr_names_oz = 'O3_du_tot'
ozone_remote_sources =  data_store.query(name_oz)
ozone_remote_sources[0].make_local(local_name=name_oz, time_range=tr_uc6, var_names=vr_names_oz, monitor=monitor) #year 2006
name_cl = 'esacci.CLOUD.mon.L3C.CLD_PRODUCTS.AVHRR.NOAA-17.AVHRR_NOAA.1-0.r1'
vr_names_cl = 'cc_total'
cloud_remote_sources =  data_store.query(name_cl)
cloud_remote_sources[0].make_local(local_name=name_cl, time_range=tr_uc6, var_names=vr_names_cl, monitor=monitor)
oz06 = ops.open_dataset('local.' + name_oz)
cl06 = ops.open_dataset('local.' + name_cl)
#Co-Register:
cores_output = ops.coregister(oz06,cl06)
cl06 = cores_output
#Subset:
region_uc6 = '-170,-5,-120,5'
oz_tmp = ops.subset_spatial(ds=oz06,region=region_uc6) #,mask=bool)
cl_tmp = ops.subset_spatial(ds=cl06,region=region_uc6) #,mask=bool)
cl_lta = ops.long_term_average(name_cl,var=vr_names_cl, year_min=2007, year_max=2008, file='/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/UC06/cl_lta.nc', save=True, monitor=monitor)
LTA: started
LTA:   0%
LTA: done
....traceback-start
Traceback (most recent call last):
File "", line 1, in
File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/ops/average.py", line 124, in long_term_average
data_source.sync(dt_range, monitor=monitor.child(work=step * 0.9))
File "/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/lib/python3.5/site-packages/cate/ds/esa_cci_odp.py", line 481, in sync
raise ValueError('Unsupported protocol', protocol)
ValueError: ('Unsupported protocol', None)
....traceback-end
Question:
Can someone explain me what I did wrong?

I know what's going on. Cate 8 rc1 is too old. It still has the old long_term_average operation that tries to fetch data on it's own, and it's a mismatch with the new ds implementation.
@mzuehlke Can we make an rc2 release with the current state of the software?
Another option would be if the installed cate-core can somehow be updated from master. Is this possible? This would be great, as then Stephan and co would always have the latest state of the software, yet wouldn't have to bother with manual installation!

In addition it seems that other operations are also not updated:
E.g. temporal_agg:
CLI:
cate op info temporal_agg
Returns:
[…]Operation cate.ops.average.temporal_agg […]
Isn't it the case that .average.py is obsolete and it belongs to .aggregate.py by now?

OK, so I think you should be able to update cate from git. I think you have already checked out the repository, so git checkout master, git pull. As you have installed cate, you should have a conda installation with all the required packages installed. So, you should just be able to do python setup.py install in cate-core root directory, where setup.py is located.
Then you should have an up to date cate-core.

the installer does not automatically make it a git-rep.

Yes. So checkout the repository (cate-core) somewhere else, unrelated to the installation. And do python setup.py install there. I think (hehe :)) it should overwrite that installed cate-core in your conda environment.
The crucial thing here is that setup.py is being executed with the correct python installation. In general though, if the command line cate runs, then running the above line should over-write the installed version.

On github->cate-core/cate/ops are all the .py’s listed which later on go into the ‘/cate-core/lib/python3.5/site-packages/cate/ops/’-directory – right?
The directory setup by the installer does not contain the same list of .py’s as listed on github. When I pull the latest version of the software into a directory named 'checkout_master' I do find a very different list of .py's (operations) within the directory 'checkout_master/cate-core/cate/ops'
(Even though I did 'python setup.py install', which updated some parts of ./bin from the installer.)
That is why I think I did something wrong or it did not work. Maybe I need a new installer ...

Yes, I think that's the correct directory. And I would've thought it should've updated that directory when you ran 'python setup.py install'.
Ah, so maybe you can try to give the full path to the correct python installation? I don't have it installed through an installer on Linux, but on windows it's
C:/User/name/cate/python.exe setup.py install
While located in the directory checked out from master.

I use the python that comes with the installer:
/cmsaf/cmsaf-ops8/sherzog/ESA_CCI_Toolbox/cate_v08_3/cate-core/bin/python

I do not know what went wrong trying updating it the first time. However, now I do have an updated TBX containing the latest OP-versions. Thanks for that!


Use Nix shell in CI

@ashelkovnykov PTAL. Note that this will require a bit of additional work in the urcrypt PR as well


How important is Axis and QLLM support

Hi HQQ team, Happy new year.
I's a great work and thanks for  the insightful idea of which is really faster during quantization. I installed hqq and ran examples in a minutes.
When I dive into your code and I found there is a parement named "axis" to indicate which direction the compression will go on. I got a question on my mind, would axis=0 be better then axis=1? I just checked that it seems GPTQ and AWQ both used axis=1 by default. However it's 0 in HQQ.
I tried to set it to 0 and do the quantization, it looks good as well. So, Do you have any comments on the quantization axis?
Besides, I integrate HQQ into my general quantization QLLM, so it works for all models in huggingface for now. The same time, it could be easily support mix-bits quantization which is naturelly supported by qllm.

Hi @wejoncy ! Thanks for your message and happy new year!
Regarding the axis question, I guess it doesn't matter or maybe for some models some axis would work better, difficult to say. We did all our benchmarks and tests with axis=0 so it would be better to just keep it that way. Since the quantization is very fast, you can even run it with both and choose the one that minimizes the error between the reconstructed and original weights!
That's great! For now, we do it manually for each architecture, but one can just loop over the layers and if it's nn.Linear -> quantize. The reason why we also do it manually is because for some models like Mixtral, giving different quantization settings to different layers works much better, see this thread: https://github.com/mobiusml/hqq/issues/2

Hi, @wejoncy, maybe you can reference this paper: https://arxiv.org/pdf/2309.15531.pdf. Quantization along axis=0, may be helpful for limiting the outlier in fewer groups as figure 1 illustrated. In fact, the activation outlier is quite common in different LLM models.

And I did some experiments using HQQ on PPL for OPT 125m->13B using LLM eval harness based on my own implementation for HQQ, the results shows that quantization along axis=0 is constantly better than axis=1. The results(w4 g128 asym):

OPT
125M
1.3B
2.7B
6.7B
13B

axis=1
39.19
18.16
15.75
13.34
11.91

axis=0
36.07
17.04
14.65
12.46
11.63

Thanks @rainyBJ for the reference! It will be interesting to do the comparison by keeping the quantized weights/scale/zero in float32 and remove the clipping. I noticed some instabilities with fp16 which could contribute to this.
To stay agnostic to the axis parameter, you can run the quantize step with both and pick the one that minimizes the error for each layer separately in: https://github.com/mobiusml/hqq/blob/master/hqq/core/quantize.py#L294
@torch.inference_mode()
def quantize(self, W, weight_quant_params, scale_quant_params, zero_quant_params):
	quant_scale = scale_quant_params is not None
	quant_zero  = zero_quant_params  is not None

	self.in_features, self.out_features = W.t().shape

	if('axis' in weight_quant_params):
		del weight_quant_params['axis']

	#Quantize - axis agnostic
	W_q_0 , meta_0 = Quantizer.quantize(W, **weight_quant_params, axis=0)
	err_0          = torch.abs(W - Quantizer.dequantize(W_q_0, meta_0)).mean()

	W_q_1 , meta_1 = Quantizer.quantize(W, **weight_quant_params, axis=1)
	err_1          = torch.abs(W - Quantizer.dequantize(W_q_1, meta_1)).mean()

	if(err_0<err_1):
		W_q, meta = W_q_0, meta_0
		del W_q_1 , meta_1
	else:
		W_q, meta = W_q_1, meta_1
		del W_q_0 , meta_0

	meta.update({'quant_scale':quant_scale, 'quant_zero':quant_zero})
	if(meta['quant_scale']):
		meta['scale_q'] , meta['meta_scale'] = Quantizer.quantize(meta['scale'], **scale_quant_params); del meta['scale']
	if(meta['quant_zero']):
		meta['zero_q'], meta['meta_zero']    = Quantizer.quantize(meta['zero'],  **zero_quant_params);  del meta['zero']

	self.W_q   = W_q
	self.meta  = meta
	self.cuda()
	self.ready = True

Really Thanks @mobicham  and @rainyBJ for your reply and the detail explanation.
It's move obvious for small model to have better PPL with axis=0 from your experiment results. I guess we can also get better results for the other algorithms.
@mobicham  Yeah, It's really amazing for the quantization speed of HQQ, it wouldn't take too long to pick the best axis for different layers. Do you think if we can have an algorithm to select the best paraments, say bits, group-size, axis, and we can have the Pareto optimality between size and PPL.
May the easiest way be to sort all layers by loss and re-quantize those candidate layers.

Yeah , not all layers are equally sensitive to quantization. For example, for the Mixtral model, the experts can be quantized to 2-bit and the attention layers to 4-bit and there's only an additional 0.20GB compared to using 2-bit for all the layers but the performance is significantly better. So it would make sense to give different parameters to different layers.
The easiest way is to use quant_config and set it for each layer type like in here : https://huggingface.co/mobiuslabsgmbh/Mixtral-8x7B-v0.1-hf-attn-4bit-moe-2bit-HQQ

Got it. Thank you.
I am thinking if there is a way to do it without human involved.


"Partition key paths cannot be empty." error when using ReplaceDocumentCollectionAsync

I am getting the following error intermittently when using DocumentClient.ReplaceDocumentCollectionAsync()
Error: One or more errors occurred., Message: Message: {"Errors":["Partition key paths cannot be empty."]}
ActivityId: 62466c1f-aaf7-4731-a07f-1fc805c6286f, Request URI: /apps/DocDbApp/services/DocDbMaster0/partitions/780e44f4-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, RequestStats: , SDK: Microsoft.Azure.Documents.Common/<IP_ADDRESS>, documentdb-dotnet-sdk/1.22.0 Host/32-bit MicrosoftWindowsNT/6.2.9200.0

Neither the existing collection nor the collection passed in have been created with a partition key.
Repro steps

Create a console app using the CollectionManagement Example from this repo.
Between steps 5 and 6, add the following line of code:
await client.ReplaceDocumentCollectionAsync(collection);
Set a breakpoint on this line and run the app.
After a short amount of time continue past the breakpoint.

Can you tell us which SDK you are using?

I'm using the Microsoft Azure Cosmos DB .NET SDK (Version 1.22.0). Previously I was using version 1.19.1 with no issues. It was only after the upgrade that I began to see problems.


Issue & PR filtering by language & product

In a monorepo, we will have a challenge scaling PR and Issue queues to a wider scope than most individuals will need. More aggressive labeling will be needed, as well as quick mechanisms to get to the filtered views.

[x] Enable automatic product labeling for Issues & PRs
[ ] Enable automatic language labeling for PRs
[ ] Create guidance/mechanism for quick lookup of relevant language/product issues

Automatic Labeling: Labeling is enabled, but there's a need to add the lang: prefix to the auto-label mechanism. Also, there are a disjoint set of language labels with and without the prefix in the repo. Will all of them be in use? Getting this perfect is out of scope, but getting our primary languages working as expected is necessary.
Guidance to Relevant Issues: I'm currently thinking about a pinned issue with guidance on how to use labels to filter, investigating if GitHub has a saved search option that would be relevant, looking into Chrome custom search engines, and whether GitHub projects might have a role in this.