Low RTX20 tensor core performance

Hi, I'm running your square mm tutorial/bench_matmul.py on an RTX2080 (compute capability 7.5) with CUDA 11.4 and pytorch 1.9 and both the triton-1.0.1.dev20210802 wheel and triton-1.0.0 (pip). I get roughly half the TFLOPs from from Triton vs CUBLAS.
For the softmax tutorial I see the expected outperformance of Triton.
thanks.
matmul-square-nn:
M     cuBLAS     Triton  CUTLASS
0    512.0  16.384000   6.909891      NaN
1    768.0  28.948417  15.798857      NaN
2   1024.0  29.228599  15.632161      NaN
3   1280.0  35.281829  16.388097      NaN
4   1536.0  40.905258  23.600336      NaN
5   1792.0  38.781708  19.718289      NaN
6   2048.0  41.979116  21.509251      NaN
7   2304.0  40.128716  23.282526      NaN
8   2560.0  39.766991  22.594238      NaN
9   2816.0  40.091195  22.432408      NaN
10  3072.0  47.434208  24.280919      NaN
11  3328.0  42.087097  23.059814      NaN
12  3584.0  43.395461  23.450199      NaN
13  3840.0  44.456862  24.021342      NaN
14  4096.0  46.487361  23.773495      NaN
15  4352.0  45.007942  21.663536      NaN
16  4480.0  44.281429  23.805984      NaN
17  4736.0  44.714373  24.007716      NaN
18  4992.0  44.933972  24.113050      NaN
19  5248.0  46.043116  22.751770      NaN
20  5504.0  45.356638  23.978917      NaN
21  5760.0  45.166731  23.875121      NaN
22  6016.0  44.986523  23.995386      NaN
23  6272.0  44.936866  23.735443      NaN
24  6528.0  45.392516  23.870419      NaN
25  6784.0  45.753316  23.779448      NaN
26  7040.0  45.605717  23.826782      NaN
27  7296.0  45.360904  24.030071      NaN
28  7552.0  44.805316  23.973715      NaN
29  7808.0  45.230768  23.980246      NaN
30  8064.0  41.971233  24.024032      NaN
31  8192.0  46.473655  24.004187      NaN

Hey!
This is actually expected. Triton is not tuned for Turing. On the RTX2080 it's using the Volta mma.884 instructions which operates at half the throughput of mma.1688. This is why you see it showing roughly half the performance of cuBLAS in your benchmark. @daadaada any interest in improving support for Turing? If RTX2080 supports mma.16816, we could just re-use the same code-path as A100 when lowering the dot instruction.
Philippe

Hi!
@axeln31459 Yep. I think it's because Triton is now using mma.884 on Turing GPUs.
@ptillet I think it's interesting since I'm also using a Turing GPU :). Though ptx doesn't allow mma.16816 on Turing :(, I can add support for mma.1688.

Hi, thanks for your quick response. Much appreciated.

Hi all, @daadaada. I am trying to understand what is meant by adding support for Turing in Triton. Do you mean that you have to introduce the instruction PTX in triton for code generation? Does this also mean that you will have to implement the functions to load data from shared memory into registers using some modified ldmatrix instruction? Will you have to swizzle data in shared memory buffers to prevent bank conflicts? What about the WMMA API?
Is adding the inline PTX instruction the only thing, or is it much more than that?
Thanks!

Hi @navdeepkk!
I think adding the inline PTX (mma.1688) is part of the work. We will also need to tweak the logic of loading data from shared memory a little. I believe we don't need to modify the data swizzling part since now Triton already swizzles data to prevent bank conflict.

Hi @daadaada, thanks for the reply. Please correct me if I am wrong. Doesn't the swizzling of data in shared memory depend on the PTX instruction which will finally use the data. Each PTX instruction may have a different thread data mapping and to have minimum bank conflicts the swizzling function might also need some tweaking, i.e., a swizzling function for one PTX instruction may not result in the same bank conflicts for a different PTX instruction. Is that correct?
Thanks!

@navdeepkk
Oh, that's correct.
I forgot to mention that I think the same swizzling method can help both Turing and Ampere GPUs to prevent bank conflict.

@daadaada oh okay. Were they different for volta? Also are these sizzling functions standard, is there a work that I can refer too?
Thanks!

@navdeepkk  Triton uses https://developer.nvidia.com/gtc/2020/video/s21745-vid for swizzling. The swizzling is a bit different for Volta, but it was designed in such a way that both swizzling mechanism call the same code paths with different parameterizations

Hi @daadaada ,
I see current triton is using inline ptx assembly for the codegen. As your previous work on gas and turingas prove that sass level is more effective, will triton add sass codegen in the near future?
Thx


💭 Question: zsh: parse error near '\n'

Hi everyone,
I could need some help fixing the zsh: parse error near '\n' issue which occurs when typing the following command:
brew install --build-from-source <codemaster-cli>

cc: @Devansh-Singh-Parmar

No expertise.


Change admin data

Well... I made a local instalation... users can't change his data (pass for example) and remember a screen to add mail configuration easily.

Now you can ADD new administrators, and EDIT administrators.

Done delete admins.


Fix quoted mediatype from content type

For some sites we were getting the following errors during crawling:
java.nio.charset.IllegalCharsetNameException: "UTF-8"
The reason is that these sites, for example facebook, are sending the charset in the content-type within quotes, like
content-type: text/html; charset="UTF-8"
When creating a Java Charset from the String ""UTF-8"" the exception is thrown.
The RFC states, that all forms are correct, with or without quotes:
https://tools.ietf.org/html/rfc7231#section-3.1.1
The same issue also is discussed here:
https://github.com/crystal-lang/crystal/issues/6353
I pushed a fix, that delegates parsing of the content-type header to httpclient, that handles all the forms correctly and added a JUnit-Test that checked the old behavior (null values) and the new one with the supported forms

Thanks for reporting and providing a fix for this issue!  It is now part of the latest snapshot release.


support http healthcheck for api lb

/kind feature
Describe the solution you'd like
I would like to use a http healthcheck for the api lb. This prevents the healthcheck to be ready before the apiserver itself is ready, because the port is opened before the apiserver is ready (and tcp healthcheck will be ready before apiserver is ready).
Anything else you would like to add:
Maybe this could be added like this:
spec:
  controlPlaneLoadBalancer:
    healthcheck:
      type: http # (or tcp / https (http with tls enabled))
      interval: 15s
      timeout: 10s
      retries: 3
      domain: ""
      path: "/readyz"

if omitted use the hetzner default values (tcp, 15s interval, 10s timeout, 3 retries)

@simonostendorf please elaborate a bit more in which use-case you encountered the error. What kind of cluster did you create, how many control-planes?
Do you have logs/events/conditions where the error is visible?
You propose to extend the CRD HetznerCluster, is that correct?

You propose to extend the CRD HetznerCluster, is that correct?

Yes.

please elaborate a bit more in which use-case you encountered the error. What kind of cluster did you create, how many control-planes?

I didn't get any errors as far as I can remember but I heard that other clusters (e.g. openstack) have problems at cluster updates where the api is unavailable because the lb adds the nodes before the api server is ready and I think to prevent errors like this and to use all the features provided by the hcloud lb, this should be added to the HetznerCluster as its a better check than tcp. I usually use 3 hcloud nodes as control plane.

I heard that other clusters (e.g. openstack) have problems at cluster updates where the api is unavailable because the lb adds the nodes before the api server is ready

@simonostendorf I hope that would be a problem here ;)
https://github.com/syself/cluster-api-provider-hetzner/blob/741b61f0fdadf2231819c7ce44add204111150ff/pkg/services/hcloud/server/server.go#L296

But what do you do if the apiserver becomes unhealthy later and the port is still reachable via tcp?

@simonostendorf I don't have all the details in my mind. But overall the code is readable. Could you please have a look at the code?
If you don't understand the code, then please tell us! The code should be readable, if comments are missing, we want to add them.
Is that ok for you?

@guettli what would be your answer to this question here?

But what do you do if the apiserver becomes unhealthy later and the port is still reachable via tcp?

If this is the case, you should usually have a handler in your stack to detect this and mark the node as unhealthy. Then you can configure machine health checks or a custom remediation strategy to fix this problem.
Usually the port will be unavailable and then the LB healthcheck will see this as well and stop routing traffic to that node. So this would be a very very rare edge that would only happen if the port is still available and for the time until your stack detects the problem and only then it will affect 1/3 or 1/5 of the connections for a small period of time.
When using Syself Autopilot: You would be affected on 1/5 of the connections for about 3 minutes. Actually I never saw a hanging port on the kube-apiserver that was not detected by the loadbalancer health check, so in the real world I guess it would be lower than 30s. We also use a reboot strategy, so the node will probably be back up in less than a minute in most cases

@simonostendorf we have not heard from you since some weeks. Were you able to solve your issue? Please tell us!
I will close the issue now. Please re-open it or create a new issue, if something is not working or not easy to understand - thank you!

Sorry for the late reply. I haven't got the time to test the lbaas failure behavior. I will reopen this issue if I have any problems with that.

@guettli
I tested the feature now: the node is added to the LB after it is marked "ready" or if it is the first node (this is the result of my test. I haven't checked if this is intended and correct).
This works for adding the node to the LB.
But I could not figure out what happens if the api server of, say, node-3 goes down from time to time. A http healthcheck would catch this and remove the node from load balancing. The caph controller does not seem to do this. So maybe supporting http healthchecks is still a good idea, but not as important as first thought.

Your observations are correct @simonostendorf . We do have a check like this in the code that you have the exact behavior you describe. The first node has to be added and after that we wait until that node is actually reachable.
No comment on the healthcheck from my side though :sweat_smile:

@simonostendorf

But I could not figure out what happens if the api server of, say, node-3 goes down from time to time. A http healthcheck would catch this and remove the node from load balancing.

I am unsure about which component is responsible for that. Maybe this is already handled at a higher level (cluster-api, kubernetes). Can you please check how other cluster-api infra providers do that?

@simonostendorf

But I could not figure out what happens if the api server of, say, node-3 goes down from time to time. A http healthcheck would catch this and remove the node from load balancing.

I am unsure about which component is responsible for that. Maybe this is already handled at a higher level (cluster-api, kubernetes). Can you please check how other cluster-api infra providers do that?

I don't think that's an infra provider responsibility. I just wanted to clarify why supporting http healthchecks is still a good idea but not as necessary as initially thought.


Update VEX URL

 Issue by abyrd
Wednesday Jun 28, 2017 at 04:13 GMT
Originally opened as https://github.com/catalogueglobal/datatools-server/issues/20

Data tools is the only Conveyal product still using the old C VEX URL pattern. We should update it so our VEX nginx proxy doesn't need to contain URL rewrite rules. Alternatively update everything to use the old VEX URL pattern which @mattwigway pointed out is probably better since it uses named URL parameters.

 Comment by landonreed
Wednesday Jun 28, 2017 at 13:28 GMT

What is the new pattern it should follow @abyrd ?

 Comment by aguiraf
Friday Jul 28, 2017 at 17:08 GMT

It should be:  server:port/min_lat,min_lon,max_lat,max_lon[.pbf|.vex]
see: https://github.com/conveyal/osm-lib
just pushed a PR

 Comment by abyrd
Tuesday Aug 01, 2017 at 10:41 GMT

Thanks @aguiraf


Swift 5 update, keyboard issue resolved, Unwanted warnings fixed

@deejeay143
there are some conflicts. can you fix them?


Enable asserts in the tests

Pull Request Description
Enso unit tests were running without -ea check enabled and as such various invariant checks in Truffle code were not executed. Let's turn the -ea flag on and fix all the code misbehaves.
Checklist
Please include the following checklist in your PR:

[ ] The documentation has been updated if necessary.
[ ] All code conforms to the
Scala,
Java,
and
Rust
style guides.
All code has been tested:

[ ] Unit tests have been written where possible.
[ ] If GUI codebase was changed: Enso GUI was tested when built using BOTH
./run ide build and ./run ide watch.

I have commited a small change that fixes the assertion failures in HashCodeTest and EqualsTest

I've noticed as I got collision when updating. Your 8a36fdf1e886be0763d383a5dcc5d34aee4f5c5d is however nicer. Keeping it. Thanks.

There are three tests failing:

TypeMembersTest - https://github.com/enso-org/enso/actions/runs/4003518106/jobs/6876561370#step:10:5175
AvoidIdInstrumentationTagTest - https://github.com/enso-org/enso/actions/runs/4003518106/jobs/6876561370#step:10:5484
RuntimeVisualizationsTest - https://github.com/enso-org/enso/actions/runs/4003518106/jobs/6876561370#step:10:5543

AvoidIdInstrumentationTagTest is still failing. Others seem to be fixed.


[ROCm] Enable gpu unrolling test

Discovered during an audit of our fork:
The //tensorflow/compiler/xla/service/gpu/tests:gpu_unrolling_test  test was fixed in https://github.com/ROCmSoftwarePlatform/tensorflow-upstream/commit/9d859f5f313ec5a0d9f626d2c0dd78e061fa036d but never enabled.
Enable this test for ROCm to increase coverage.
Other details:

the gpu_kernel_tiling_test was also "fixed" in the above commit and remains disabled as well. That test fails and will need some extra attention so I left it off here.

Commits to go upstream:
9d859f5f313ec5a0d9f626d2c0dd78e061fa036d
234b54661230b9db8169977bd88b1d81aa1cc8c7 (this commit)

@ekuznetsov139  just an FYI as it looks like you fixed these tests in the original commit.

@deven-amd FYI, here was another one I found.

LGTM

@deven-amd @jayfurmanek
This test appears to be causing failures: http://ml-ci.amd.com:21096/job/tensorflow/job/github-prs-rocmfork-develop-upstream/job/rocm-latest-ubuntu-gpu-single/603/consoleFull
It appears that it doesn't recognize string.
I have made changes in https://github.com/ROCmSoftwarePlatform/tensorflow-upstream/pull/1530 , changing string to std::string.

was about to do the same fix on my end....thank you for fixing this!

Strange, I didn't see that in my testing. Thanks for fixing!


Vmss: fix creation failure when use unmanaged disk

Found this when went over the live test results. The error is Parameter 'dataDisks' is not allowed..  I also fix a test broken by that Standard_A4 no longer supports Premium sku.
Moving forward let us still check out live test result daily so we can fix it as early as possible,
//cc: @tjprescott, @troydai

[x] The PR has modified HISTORY.rst with an appropriate description of the change (see Modifying change log).

Command Guidelines

[x] Each command and parameter has a meaningful description.
[x] Each new command has a test.

(see Authoring Command Modules)

Codecov Report

Merging #3844 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #3844      +/-   ##
==========================================
+ Coverage   70.91%   70.91%   +<.01%
==========================================
  Files         424      424
  Lines       26895    26896       +1
  Branches     4094     4094
==========================================
+ Hits        19073    19074       +1
+ Misses       6581     6578       -3
- Partials     1241     1244       +3

Impacted Files
Coverage Δ

.../azure/cli/command_modules/vm/_template_builder.py
85.85% <100%> (+0.06%)
:arrow_up:

...dback/azure/cli/command_modules/feedback/custom.py
31.25% <0%> (ø)
:arrow_up:

...nent/azure/cli/command_modules/component/custom.py
16.23% <0%> (ø)
:arrow_up:

src/azure-cli-core/azure/cli/core/util.py
70.06% <0%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a9f6f42...3d91436. Read the comment docs.

Looks good. Approved. Btw, please rebase the pull request because the locations of the tests were moved in https://github.com/Azure/azure-cli/pull/3848.


[android] build libxml2 and curl with -fPIC on Windows

Purpose
Defines CMAKE_POSITION_INDEPENDENT_CODE=YES when building libxml2 and curl on Windows with build.ps1. Defining this property makes build.ps consistent with the Python build scripts swift_build_support\products\libxml2.py and swift_build_support\products\curl.py.
Problem Details
Android i686/x86 binaries fail to link against libraries compiled without -fPIC.
Validation
Successfully built swift toolchain locally with the following command on Windows:
S:\SourceCache\swift\utils\build.cmd -AndroidSDKs x86_64,aarch64,i686

This command previously failed linking with i686 libcurl due to not being compiled with -fPIC.

@swift-ci please smoke test


Cron Job is not working - 'No such file or directory' error

I wrote below content in Crontab -e for my rails project.
That project is deployed on production(CentOS- Amazon).
0,5,10,15,20,25,30,35,40,45,50,55 * * * * /bin/bash -l -c 'cd /var/www/houseq/current && RAILS_ENV=production bundle exec bin/rails runner -e production '\''TaskSchedulerJob.perform_now'\'''
but there is no working. thus I execute TaskSchedulerJob.perform_now command by manually.
if I execute /bin/bash -l -c 'cd /var/www/houseq/current && RAILS_ENV=production bundle exec bin/rails runner -e production '\''TaskSchedulerJob.perform_now'\''' command, they shows below error
No such file or directory
help!!

Although question was created few month ago...
Simple way - create symlink to bash directory (if bash installed)

Find current bash directory - 'whereis bash'
Returns example: '/usr/local/bin/bash'. Can be more then one folder, we need folder with 'bin'

Create symlink - 'ln -s /usr/local/bin/bash /bin/bash'

It's works, but it's may be wrong way.


Access Token 403 forbidden

I tried all login methods every time I login successfully but the udeler dosen't detect it, even extension still not detecting the login

Check the updated version here, the issue has been fixed.
https://github.com/Puyodead1/udemy-downloader


Validate topic names

Catch the common case of someone sticking an invalid char in the topic
line.
Topic: catch1-./Q
Reviewers: brian-k

Reviews in this chain:
└https://github.com/Skydio/revup/pull/103 Validate topic names

#
head
base
diff
date
summary

0
9736a989
fa4e5a9a
diff
Apr 4 2:21 AM
1 file changed, 4 insertions(+)


Hazelcast K8s DNS based pod discovery has been removed

Describe the issue:
Hazelcast K8s DNS based pod discovery has been removed with https://github.com/wso2/carbon-kernel/pull/3290/files, it was supposed to be deprecated[1] and not removed entirely without any notice.
Expected behavior:
DNS based pod discovery mechanism should not be removed without any notice.
Environment information (Please complete the following information; remove  any unnecessary fields) :

Product Version: [IS 5.11.0]

[1]https://github.com/wso2/kubernetes-common/issues/22

We don't have any plans to re-introduce DNS based pod discovery back to the next product release. Hence closing the issue.


while loop

I am trying to implement a while loop using promises.
The method outlined here seems to work.
http://blog.victorquinn.com/javascript-promise-while-loop
it uses a function like this
var Promise = require('bluebird');

var promiseWhile = function(condition, action) {
    var resolver = Promise.defer();

    var loop = function() {
        if (!condition()) return resolver.resolve();
        return Promise.cast(action())
            .then(loop)
            .catch(resolver.reject);
    };

    process.nextTick(loop);

    return resolver.promise;
};

This seems to use anti-patterns and deprecated methods like cast and defer.
Does anyone know a better or more modern way to accomplish this?
Thanks

Yes there is
Please ask support questions on Stack Overflow with the promise and bluebird tags.

Ah, http://stackoverflow.com/questions/29375100/while-loop-using-bluebird-promises


login page: missing docs

we need content for:

[ ] an About page
[ ] Legal Notice
[ ] Data privacy policy
[ ] A brief description of the CRR on the page

Just an idea for general information:
The clinical research repository is a file based repository - capable of holding any type of 'digital object' - be it a data set, a document, a computer program, or a media file.
The secure infrastructure for file storage is provided by TSD, an established, secure, data infrastructure that is part of the University of Oslo.
The interface of the repository is provided by ECRIN. We liaise with both data providers, for the transfer of digital objects to the repository, and data requesters, browsing the objects under managed access. Both processes are governed by formal agreements, one for the data providers, and another for the secondary users. ECRIN uses a Repository Management System, developed in-house, to manage and record this activity.
The system developed so far is intended to provide and support: the management of repository content and its associated metadata; the transfer of data objects into the repository, including details of data contributors and their requirements; the quality control processes applied to uploaded data objects; the handling of data requests, in particular the responses of the repository in answering data requests; the usage of all types of stored data objects; the repository’s performance against pre-agreed quality criteria; interfacing with other systems, for example TSD and ECRIN’s MDR.
The aim of these efforts is to provide a single system to support and record all the workflows associated with managing the repository.

added 2 placeholder pages for
legal notice:  https://crr.gitbook.io/crr/legal-notice
and
data privacy policy: https://crr.gitbook.io/crr/data-privacy-policy
TO BE FILLED

another version for @Gerd-ECRIN 's general info page:
The clinical research repository is a file based repository - capable of holding any type of 'digital object' - be it a data set, a document, a computer program, or a media file.
The secure infrastructure for file storage is provided by TSD, an established secure data infrastructure that is part of the University of Oslo.
ECRIN liaises with both data providers, for the transfer of digital objects to the repository, and data requesters, browsing the objects under managed access. Both processes are governed by formal agreements, one for the data providers, and another for the secondary users. ECRIN uses a Repository Management System, developed in-house, to manage and record this activity.
The system developed so far is intended to provide and support: the management of repository content and its associated metadata; the transfer of data objects into the secure infrastructure; the collection of details of data contributors and their requirements; the quality control processes applied to uploaded data objects; the handling of data requests, in particular the responses of the repository in answering data requests; the usage of all types of stored data objects; the repository’s performance against pre-agreed quality criteria.

link to ecrin data privacy for our data privacy?
https://ecrin.org/data-privacy-policy
for the moment added the link in the gitbook doc
no idea where to find a legal notice, or what it should be.


cannot open browser when executing hub pull-request --browse with emacsclient

Environment

Mac

Mac OS Sierra
git version 2.10.1 (Apple Git-78)
hub version 2.2.9

Linux

Ubuntu 16.10
git version 2.9.3
hub version 2.2.0-416

editor: emacsclient

Overview
I couldn't open browser when executing hub pull-request --browse after creating a pull-request.
I expected to open pull request page I created, but I could open Emacs.
I found this problem on MacOS Sierra, but I also found this problem on Ubuntu.
I think it is because git.core.editor is emacsclient.

I couldn't open browser when executing hub pull-request --browse after creating a pull-request.

What happened instead when you ran that command?
Protip: you could use HUB_VERBOSE=1 hub pull-request --browse to see extra debugging information.
The --browse option shouldn't be affected at all by your core.editor configuration. They are separate operations. First, the text editor opens so you can write the title and the body of your pull request. Then, the pull request is created using GitHub API. Finally, the --browse option causes the web browser to open with the URL of the just-created pull request. On macOS, this gets executed:
open https://github.com/YOURNAME/PROJECT/pull/123

I could open text editor instead when I ran hub pull-request --browse even if  the Pull Request exists.
It was HUB_VERBOSE=1 hub pull-request --browse on Ubuntu 16.10.
$ git rev-list --cherry-pick --right-only --no-merges origin/topic-branch...
$ git remote -v
$ git remote -v
$ git rev-list --cherry-pick --right-only --no-merges upstream/master...origin/topic-branch
$ git rev-parse --show-toplevel
$ git config core.commentchar
$ git var GIT_EDITOR
$ git config core.commentchar
$ emacsclient /path/to/github-repo/.git/PULLREQ_EDITMSG

Oh, so you're saying that the pull request for this branch already exists, and you expected its web page to open immediately, but instead your text editor opened?
The hub pull-request command is not yet aware of existing pull requests. It will open a text editor every time because it thinks you want to create a new pull request.
I do want to make pull-request aware of existing pull requests. This will likely get addressed at the same time as https://github.com/github/hub/issues/688. Does this answer your questions?

Thank you for your reply.
It can be my answer.


chore: update otel go dependancies and resolve breaking change

To unblock the same upgrade in the collector https://github.com/open-telemetry/opentelemetry-collector/pull/7324
Closes: https://github.com/influxdata/influxdb-observability/pull/181, https://github.com/influxdata/influxdb-observability/pull/180, https://github.com/influxdata/influxdb-observability/pull/179, https://github.com/influxdata/influxdb-observability/pull/178

Once this is merged, we don't need another influx release. We can just point the contrib to the latest commit for now

Integration tests are fixed by upgrading OTel Collector core dependency, which required the go version upgrade to 1.19.0


Add support for local dependencies detection

Fixed the remaining bugs with local dependencies detection in : https://github.com/Robinson04/serverlesspack/commit/f221e7fffa53ebc0c8be06c232b9ece199310d5c


[#168567141] Enable CORS on the requests from the frontend client

This PR aims to enable CORS on the requests coming from the client.

Affected stories

🌟 #168567141: Abilitazione di CORS per le chiamate dal frontend

New dependencies added: cors and @types/cors.
cors
Author: Troy Goode
Description: Node.js CORS middleware
Homepage: https://github.com/expressjs/cors#readme

Createdover 6 years agoLast Updated26 days agoLicenseMITMaintainers2Releases34Direct Dependenciesobject-assign and varyKeywordscors, express, connect and middleware

README
cors

CORS is a node.js package for providing a Connect/Express middleware that can be used to enable CORS with various options.
Follow me (@troygoode) on Twitter!

Installation
Usage

Simple Usage
Enable CORS for a Single Route
Configuring CORS
Configuring CORS Asynchronously
Enabling CORS Pre-Flight

Configuration Options
Demo
License
Author

Installation
This is a Node.js module available through the
npm registry. Installation is done using the
npm install command:
$ npm install cors

Usage
Simple Usage (Enable All CORS Requests)
var express = require('express')
var cors = require('cors')
var app = express()

app.use(cors())

app.get('/products/:id', function (req, res, next) {
  res.json({msg: 'This is CORS-enabled for all origins!'})
})

app.listen(80, function () {
  console.log('CORS-enabled web server listening on port 80')
})

Enable CORS for a Single Route
var express = require('express')
var cors = require('cors')
var app = express()

app.get('/products/:id', cors(), function (req, res, next) {
  res.json({msg: 'This is CORS-enabled for a Single Route'})
})

app.listen(80, function () {
  console.log('CORS-enabled web server listening on port 80')
})

Configuring CORS
var express = require('express')
var cors = require('cors')
var app = express()

var corsOptions = {
  origin: 'http://example.com',
  optionsSuccessStatus: 200 // some legacy browsers (IE11, various SmartTVs) choke on 204
}

app.get('/products/:id', cors(corsOptions), function (req, res, next) {
  res.json({msg: 'This is CORS-enabled for only example.com.'})
})

app.listen(80, function () {
  console.log('CORS-enabled web server listening on port 80')
})

Configuring CORS w/ Dynamic Origin
var express = require('express')
var cors = require('cors')
var app = express()

var whitelist = ['http://example1.com', 'http://example2.com']
var corsOptions = {
  origin: function (origin, callback) {
    if (whitelist.indexOf(origin) !== -1) {
      callback(null, true)
    } else {
      callback(new Error('Not allowed by CORS'))
    }
  }
}

app.get('/products/:id', cors(corsOptions), function (req, res, next) {
  res.json({msg: 'This is CORS-enabled for a whitelisted domain.'})
})

app.listen(80, function () {
  console.log('CORS-enabled web server listening on port 80')
})

If you do not want to block REST tools or server-to-server requests,
add a !origin check in the origin function like so:
var corsOptions = {
  origin: function (origin, callback) {
    if (whitelist.indexOf(origin) !== -1 || !origin) {
      callback(null, true)
    } else {
      callback(new Error('Not allowed by CORS'))
    }
  }
}

Enabling CORS Pre-Flight
Certain CORS requests are considered 'complex' and require an initial
OPTIONS request (called the "pre-flight request"). An example of a
'complex' CORS request is one that uses an HTTP verb other than
GET/HEAD/POST (such as DELETE) or that uses custom headers. To enable
pre-flighting, you must add a new OPTIONS handler for the route you want
to support:
var express = require('express')
var cors = require('cors')
var app = express()

app.options('/products/:id', cors()) // enable pre-flight request for DELETE request
app.del('/products/:id', cors(), function (req, res, next) {
  res.json({msg: 'This is CORS-enabled for all origins!'})
})

app.listen(80, function () {
  console.log('CORS-enabled web server listening on port 80')
})

You can also enable pre-flight across-the-board like so:
app.options('*', cors()) // include before other routes

Configuring CORS Asynchronously
var express = require('express')
var cors = require('cors')
var app = express()

var whitelist = ['http://example1.com', 'http://example2.com']
var corsOptionsDelegate = function (req, callback) {
  var corsOptions;
  if (whitelist.indexOf(req.header('Origin')) !== -1) {
    corsOptions = { origin: true } // reflect (enable) the requested origin in the CORS response
  } else {
    corsOptions = { origin: false } // disable CORS for this request
  }
  callback(null, corsOptions) // callback expects two parameters: error and options
}

app.get('/products/:id', cors(corsOptionsDelegate), function (req, res, next) {
  res.json({msg: 'This is CORS-enabled for a whitelisted domain.'})
})

app.listen(80, function () {
  console.log('CORS-enabled web server listening on port 80')
})

Configuration Options

origin: Configures the Access-Control-Allow-Origin CORS header. Possible values:

Boolean - set origin to true to reflect the request origin, as defined by req.header('Origin'), or set it to false to disable CORS.
String - set origin to a specific origin. For example if you set it to "http://example.com" only requests from "http://example.com" will be allowed.
RegExp - set origin to a regular expression pattern which will be used to test the request origin. If it's a match, the request origin will be reflected. For example the pattern /example\.com$/ will reflect any request that is coming from an origin ending with "example.com".
Array - set origin to an array of valid origins. Each origin can be a String or a RegExp. For example ["http://example1.com", /\.example2\.com$/] will accept any request from "http://example1.com" or from a subdomain of "example2.com".
Function - set origin to a function implementing some custom logic. The function takes the request origin as the first parameter and a callback (which expects the signature err [object], allow [bool]) as the second.

methods: Configures the Access-Control-Allow-Methods CORS header. Expects a comma-delimited string (ex: 'GET,PUT,POST') or an array (ex: ['GET', 'PUT', 'POST']).
allowedHeaders: Configures the Access-Control-Allow-Headers CORS header. Expects a comma-delimited string (ex: 'Content-Type,Authorization') or an array (ex: ['Content-Type', 'Authorization']). If not specified, defaults to reflecting the headers specified in the request's Access-Control-Request-Headers header.
exposedHeaders: Configures the Access-Control-Expose-Headers CORS header. Expects a comma-delimited string (ex: 'Content-Range,X-Content-Range') or an array (ex: ['Content-Range', 'X-Content-Range']). If not specified, no custom headers are exposed.
credentials: Configures the Access-Control-Allow-Credentials CORS header. Set to true to pass the header, otherwise it is omitted.
maxAge: Configures the Access-Control-Max-Age CORS header. Set to an integer to pass the header, otherwise it is omitted.
preflightContinue: Pass the CORS preflight response to the next handler.
optionsSuccessStatus: Provides a status code to use for successful OPTIONS requests, since some legacy browsers (IE11, various SmartTVs) choke on 204.

The default configuration is the equivalent of:
{
  "origin": "*",
  "methods": "GET,HEAD,PUT,PATCH,POST,DELETE",
  "preflightContinue": false,
  "optionsSuccessStatus": 204
}

For details on the effect of each CORS header, read this article on HTML5 Rocks.
Demo
A demo that illustrates CORS working (and not working) using jQuery is available here: http://node-cors-client.herokuapp.com/
Code for that demo can be found here:

Client: https://github.com/TroyGoode/node-cors-client
Server: https://github.com/TroyGoode/node-cors-server

License
MIT License
Author
Troy Goode<EMAIL_ADDRESS>
@types/cors
Author: Unknown
Description: TypeScript definitions for cors
Homepage: http://npmjs.com/package/@types/cors

Createdover 3 years agoLast Updated15 days agoLicenseMITMaintainers1Releases21Direct Dependencies@types/express

README
Installation

npm install --save @types/cors

Summary
This package contains type definitions for cors (https://github.com/troygoode/node-cors/).
Details
Files were exported from https://github.com/DefinitelyTyped/DefinitelyTyped/tree/master/types/cors
Additional Details

Last updated: Mon, 19 Aug 2019 00:51:10 GMT
Dependencies: @types/express
Global values: none

Credits
These definitions were written by Alan Plum https://github.com/pluma.

  Generated by :no_entry_sign: dangerJS against e24521b3827dc0d63a3c2653b3e21d42ec1e8b8a


"Could not implement aligned_alloc..." when linking VMA as a CMake target

[...]/vk_mem_alloc.h:2791: void* vma_aligned_alloc(size_t, size_t): Assertion 0 && "Could not implement aligned_alloc automatically. Please enable C++17 or later in your compiler or provide custom implementation of macro VMA_SYSTEM_ALIGNED_MALLOC (and VMA_SYSTEM_ALIGNED_FREE if needed) using the API of your system."' failed.
This seems to happen because the CMake project for VMA is setup for C++14, and when using GCC on Linux it needs C++17 for the aligned_alloc() function. The project I'm using VMA for is using C++20, but the setting is not propagated back to VMA's CMake project.
As an example, VulkanMemoryAllocator-Hpp links VMA this way.
My solution to this problem is overriding VMA's C++ standard after including it:

add_subdirectory(vma)

set_target_properties(VulkanMemoryAllocator PROPERTIES CXX_STANDARD 17)

I don't know how desirable it is for the VMA's CMake project to stay at C++14, if it's not a big deal then I suggest simply bumping the standard in the src/CMakeLists.txt file.

Thank you for this suggestion. I agree it may be a good idea to default to C++17 in src/CMakeLists.txt to fix the issue with aligned_alloc. I committed this change.


[many ports] Add GnuTLS and its dependencies

This PR provides a port for GnuTLS library and its dependencies.

Which triplets are supported/not supported? Have you updated the CI baseline?
x64-linux only, for now. Yes.

Does your PR follow the maintainer guide?
Yes

can you please hurry up to port this library faster?!

@odskinz488 You can open a new PR, inherit these changes and continue.

Pinging @wrobelda for response. Is work still being done for this PR?

@JackBoosY I intend to finish this work, got held back by migrating to Apple Silicon last month.

@JackBoosY ready for review. Linux only, for now.

@JackBoosY I just got it to work on macOS, about to commit those changes

@JackBoosY the CI build is being skipped for libgnutls: "libgnutls:x64-osx:    cascade"

@wrobelda  Because libgnutls dependency can't be built on OSX now.

This LGTM, thanks for the PR!


RSortSet donot implement RExpire interface

RSortSet donot implement RExpire interface.
So I can't  set  expire time  to  it

Done


NEXT-36499 - Fix cms data mapping for nested translations

1. Why is this change necessary?
In the shopping experience I set a data mapping for an mapped entity value (e.g. product.manufacturer.name).
In storefront for the default language works everything fine (because of product.manufacturer.name is filled). But in any other language it don't works (because of product.manufacturer.name is not filled and it don't reads translated value).
2. What does this change do, exactly?
It reads the correct translated value of nested entties inside the cms data mapping.
3. Describe each step to reproduce the issue or behaviour.

Create a text shopping experience element inside a cms product detail page.
Inside the shopping experience add a new default text element. Select the data mapping and map product.manufacturer.name there.
Assign this shopping experience to a product, which has a manufacturer assigned
Open the product in the storefront in default language -> there everything works fine and you see the manufacturer name like it should be.
Now change the language in the storefront and check againg -> now you see the product name and not the manufacturer name.

4. Please link to the relevant issues (if any).
https://issues.shopware.com/issues/NEXT-36499
5. Checklist

[x] I have rebased my changes to remove merge conflicts
[ ] I have written tests and verified that they fail without my change
[x] I have created a changelog file with all necessary information about my changes
[ ] I have written or adjusted the documentation according to my changes
[ ] This change has comments for package types, values, functions, and non-obvious lines of code
[x] I have read the contribution requirements and fulfil them.

hey @acris-cp
thanks for your contribution. would you mind adding a test for this?

Greetings! Any updates on this ticket?

Currently I don't have time to program a test.
Hopefully anyone of Shopware will test and do that issue where I provided you the solution.

Currently I don't have time to program a test. Hopefully anyone of Shopware will test and do that issue where I provided you the solution.

Okay, Thank :+1:

Your Pull Request has been merged! :tada:
Thanks for your contribution! :blue_heart:


Form validation about business category details

Add server side form validation to filter only subcategories of the selected category and common subcategories.

I'll do this.


NixOS option to supply environments attribute set.

so that update of iohk-nix is not required
for changes in environments config

bors r+


Re-export the (>$<) operator

The >$< operator works well with (.) unlike the infix version of contramap.

Probably not, things like >$ or the newtype wrappers are not very useful in logging context.


SPIKE: how to build Module for DNS and certificates on the platform

User Story
As a user, I want to be able to quickly set up DNS for my services
What is the smallest thing we can build that will be useful?
User Type(s)
Developer or engineer user
Value
Make setting up DNS and certificates easier
Questions / Assumptions / Hypothesis
The way to set up DNS and certificates is documented but requires a few different things to work properly which can easily cause trouble for first time users.

What trouble is caused?
Why is it only first time users and what are the different things?
Does this affect other first time set up processes?

If we create a module, we expect that it will make it easier for first time users to create DNS and certificates and require less input from mp engineers to resolve queries.
Definition of done

[ ] module created
[ ] readme has been updated
[ ] user docs have been updated
[ ] Tested in example
[ ] Terratests written and passing
[ ] another team member has reviewed
[ ] tests are green
[ ] UR test OR added to continual research plan

Reference
How to write good user stories

PR Created to create a new repo
https://github.com/ministryofjustice/modernisation-platform/pull/3858

initial code pr on new repo
https://github.com/ministryofjustice/modernisation-platform-terraform-dns-certificates/pull/3

Definition of done here is maybe a bit big for a spike.

Senarios:

Non production uses mp domains and certificates, production uses delegated hosted zone domain and ACM certificates
Non production uses mp domains and certificates, production uses delegated hosted zone domain and Gandi certificates
Non production uses uses delegated hosted zone, production uses delegated hosted zone domain and ACMs certificates
Non production uses uses delegated hosted zone, production uses delegated hosted zone domain and Gandi certificates


house cleaning

Some changes revert to leaning more on configlet, since we're beholden to its wishes anyway. Others remove unnecessary code. The rest update the README.


Some minor updates to Multi-cluster documentation

Indicate that cross-cluster WireGuard is not compatible with in-cluster encryption.
Indicate that Multi-cluster Gateway is not compatible with in-cluster WireGuard.
Remove stale information from Helm doc about "encap" mode being required for Multi-cluster Gatway.

/skip-all


Unable to console into containers

Bug description
Since the upgrade to version 1.17.0 Portainer no longer allows console access into any containers when sitting behind a nginx reverse proxy. The web page pops up with an error in the top right reading:
Failure
no such exec

It's also worth noting that rolling back to version 1.16.5 restores Console access.
Expected behavior
Console for respective container to launch for executing commands.
Steps to reproduce the issue:
Steps to reproduce the behavior:

Go to 'Containers'
Click on any container
Click on 'Console >_'
Click on 'Connect'
See error in top right

Technical details:

Portainer version: 1.17.0
Docker version (managed by Portainer): 18.04.0-ce
Platform (windows/linux): Linux
Command used to start Portainer (docker run -p 9000:9000 portainer/portainer):

  portainer:
    image: portainer/portainer
    container_name: portainer
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /mnt/nerv01/.config/.portainer:/data
    networks:
      - nerv-net
    expose:
      - 9000
    restart: unless-stopped

Browser: Google Chrome 66.0.3359.139

Additional context
The portainer logs show the following error each time a request to console is executed (clicking the Connect button when a shell has been selected).
2018/05/12 07:45:17 http error: websocket: request origin not allowed by Upgrader.CheckOrigin (code=500)
2018/05/12 07:45:17 http: multiple response.WriteHeader calls

The instance of Portainer is reverse proxied behind an SSL secured nginx with the following configuration:
server {
    server_name portainer.example.com;
    listen 443 ssl http2;
    include /config/nginx/options-ssl.conf;

    location / {
        proxy_pass http://portainer:9000;
        include /config/nginx/proxy.conf;
        proxy_set_header Connection "";
    }

    location /api/websocket {
        proxy_pass http://portainer:9000/api/websocket;
        include /config/nginx/proxy.conf;
        # websocket configuration
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

proxy.conf:
client_max_body_size 10m;
client_body_buffer_size 128k;

#Timeout if the real server is dead
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503;

# Advanced Proxy Config
send_timeout 5m;
proxy_read_timeout 240;
proxy_send_timeout 240;
proxy_connect_timeout 240;

# Basic Proxy Config
proxy_set_header Host $host:$server_port;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Ssl on;
proxy_redirect  http://  $scheme://;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_cache_bypass $cookie_session;
proxy_no_cache $cookie_session;
proxy_buffers 32 4k;

FYI, I'm seeing the same behavior connecting directly to Portainer (i.e. not behind a proxy)

@fslDev please open a new issue if you're not using Nginx here.

@nightah could you try with portainer/portainer:1887 ?
@fslDev feel free to try as well although I'm not sure your problem is related.

@deviantony pr1901 still fails with the same error:
2018/05/16 00:53:04 Starting Portainer 1.17.0 on :9000
2018/05/16 00:53:51 http error: websocket: request origin not allowed by Upgrader.CheckOrigin (code=500)
2018/05/16 00:53:51 http: multiple response.WriteHeader calls

The frontend still displays the no such exec error also.

The problem is that nginx does not allow the websocket connection.
The problem can be solved easily if the configuration of Nginx adapt.
location /wsapp/ {
proxy_pass http://wsbackend;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
}

@intergate thank you this worked for me.
Don't forget to rebuild with the docker-compose up --build option for changes to take effect

because websocket  can not work with http2.


fastlane match doesn't create a Provisioning Profile after adding 'Sign in with Apple' capability

New Issue Checklist

[/] Updated fastlane to the latest version
[/] I read the Contribution Guidelines
[/] I read docs.fastlane.tools
[/] I searched for existing GitHub issues

Issue Description

I added 'Sign in with Apple' capability and I revoked the existed certificate and provisioning profile using match nuke distribution Then I created a new certificate and a provisioning profile using match appstore. But it doesn't create a Provisioning Profile on Apple Developer account.
 Or just in specific circumstances? Is this reproducible?

Are you reporting that creation of profisioning profiles is broken? Or just in specific circumstances? Is this reproducible?

Hi. Previously I used fastlane match successfully before I updated Xcode 11. Then I added 'Sign in with Apple' capability. Then Xcode shows the provisioning profile doesn't include the capability. So I revoked it and tried to create it again. By the way it doesn't create a new Provisioning Profile to Apple Developer account. If my understanding is correct, match should create a new one to Apple Developer account.
So I think that it can be reproduced in Xcode 11.

@idreamer have you tried to use the option generate_apple_certs?


812xx ci release notes

Before publish:

Update date in 2 rn topics
Update delegate version in 2 rn topics
Update CI version in 2 rn topics

Previews

CI release notes
What's new page
K8s cluster self-signed cert set up
CI FAQ - self-signed cert question

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app


Failing lint

/tmp/awesome-memento$ awesome-lint
✖ Linting

  README.md:7:86
  ✖   1:1    Missing file contributing.md                                                remark-lint:awesome-contributing
  ✖   1:1    The repository should have "awesome" as a GitHub topic                      remark-lint:awesome-github
  ✖   1:1    Missing or invalid Table of Contents                                        remark-lint:awesome-toc
  ✖   1:20   Invalid badge source                                                        remark-lint:awesome-badge
  ✖   7:86   Link to https://addons.mozilla.org/en-US/firefox/addon/mementofox/ is dead  remark-lint:no-dead-urls
  ✖  17:88   List item link and description must be separated with a dash                remark-lint:awesome-list-item
  ✖  25:51   List item link and description must be separated with a dash                remark-lint:awesome-list-item
  ✖  26:86   List item link and description must be separated with a dash                remark-lint:awesome-list-item
  ✖  43:49   List item description must end with proper punctuation                      remark-lint:awesome-list-item
  ✖  44:110  List item description must end with proper punctuation                      remark-lint:awesome-list-item
  ✖  45:108  List item description must end with proper punctuation                      remark-lint:awesome-list-item
  ✖  47:107  List item description must end with proper punctuation                      remark-lint:awesome-list-item
  ✖  48:122  List item description must end with proper punctuation                      remark-lint:awesome-list-item
  ✖  49:145  List item description must end with proper punctuation                      remark-lint:awesome-list-item
  ✖  50:97   List item description must end with proper punctuation                      remark-lint:awesome-list-item
  ✖  51:108  List item description must end with proper punctuation                      remark-lint:awesome-list-item
  ✖  53:141  List item description must end with proper punctuation                      remark-lint:awesome-list-item
  ✖  54:125  List item description must end with proper punctuation                      remark-lint:awesome-list-item
  ✖  55:177  List item description must end with proper punctuation                      remark-lint:awesome-list-item
  ✖  56:102  List item description must end with proper punctuation                      remark-lint:awesome-list-item

  20 errors

@ibnesayeed See #7.


Cross-build to Scala 2.13

Upstream requirement: https://github.com/eed3si9n/scalajson/pull/3.

Blocked by https://github.com/non/jawn/issues/97

Actually the jawn situation is complicated, given it's broken it's binary api in 0.11 and we've established the sbt ecosystem on jawn 0.10...

jawn 0.12.1 is out for Scala 2.13.0-M3 but, as I said, sbt 1 is stuck on jawn 0.10.


Fix/arch key + update README + disable serviceMonitoring

1. README update
1.1. put master branch in the curl operations - make it ready for GA.
1.2. remove IBMBlockCSI yaml from the readme, and instead asking the user to create it, we just move it as a file that the user curl and download from github. the same as the operator yaml.
1.3. Use -l csi to get the pods
2. fix kubernetes.io/arch
instead of beta.kubernetes.io/arch which is deprecated.
3. Disable the ServiceMonitor
Disable the creation of ServiceMonitoring since we don't have time to test it for v1. Created epic in jira for later version

@27149chen  - please review again and approve

@27149chen  please review again - i fixed your comment and also resolved conflict from changes you did in develop, so please check it again.

Great - merging to develop


Compile time

I just tried the aws-sdk-ec2 in my workspace. rusoto_ec2 was always the second to last crate and took forever (66s) to build. aws-sdk-ec2 seems to be even worse.
>  cargo +nightly build -Z timings

aws-sdk-ec2 v0.0.8-alpha | 95.8s
rusoto_ec2 v0.46.0 | 66.5s

Is compile time something you are going to consider in the future?

yeah this is definitely on our radar. Thanks for flagging it.
Other folks: please 👍🏻  this issue to prioritize compile times if they become an issue for you.

You could add a demo example that depends on all SDKs for compile times measurement.

We kind of get that right now through the CI in smithy-rs where one of the actions compiles every single SDK as a single Cargo workspace. For example, the cargo test AWS SDK step, which currently shows 14 minutes for all of the supported services. This data gets muddied every time we add a new service though. It would be nice to have it split out by service so that we could, for example, compare EC2's compile time across builds. It would also be nice to track with cargo llvm-lines for a measurement that is less impacted by differences in the CI hosts.

This is also my biggest complain in using the SDK. One idea that a team member here had is that EC2 functionality could be opted-into using Cargo features. Often you will be using only a few calls from the API, but you have to compile all of it.
Another idea would be to use dynamically dispatched trait objects instead of trait bound generics. I don't know how much that wouldn't help, but I mention it because Rust for Rustaceans mentions faster compile times as a benefit of using dynamic dispatch.

I've investigated this a teeny bit and at least part of the problem seems to stem from the fact that EC2 has a significant number (118) of paginated APIs. The paginators return impl tokio_stream::Stream which means we're doing a good amount of monomorphization. We'll investigate feature gating pagination or switching to dynamic dispatch.

For my purposes I actually switched off of using this SDK to instead using a thin wrapper around the AWS CLI and shaved off many many minutes of build time, especially when you consider the separate rebuilds involved for e.g. tests/clippy/release.
Obviously this it not an option or ideal in most cases where the SDK is used, I'm simply sharing the difference it made for me.
I would love to use the SDK instead but for my light purposes it wasn't worth the cost of many minutes of build time.
I sincerely believe that it would be a great idea to get in touch with the Rust compiler optimization team about the compile times incurred by these crates because I think it would serve as a great case study and benchmark on the impact of e.g. monomorphization and optimizing that would benefit the broader Rust ecosystem.

We'll investigate feature gating pagination or switching to dynamic dispatch

Did anything come of these investigations?
For a short term fix I am considering forking the generated ec2 crate and removing all the code that is unused by my project.

I think we were all expecting at least some progress on this before this crate went 1.0. :-(
Any plans to event start addressing this issue?

i attempted stripping out the bits I didnt need but I've given up because of how entangled the generated code is.

I think what is needed is for the code generation to include a bunch of feature gates on EC2 API functionality so that we can enable the things we need in Cargo.toml.

Compile times of an async cli application went from 5 to 15 seconds just by referencing aws-sdk-ec2, and calling 2 different areas of the functionality.
I also was expecting some progress on this before Stable release, please address this asap.

Just to chime in here on the build times, I'm looking into build failures (well, more like Jenkins dying during builds) and noticed a couple of things that seem to line up with the behaviour here in more details.
I've been tinkering with more verbose compiler logging/tracing for each stage and noticed some pretty interesting results for the AWS SDKs in particular that are miles more expensive that any other crate in shotover. The important results here are as follows.
[2024-01-09T01:45:01.991Z]    Compiling aws-sdk-ec2 v1.3.0
[2024-01-09T01:45:01.991Z] time:   0.001; rss:   41MB ->   44MB (   +3MB)	parse_crate
[2024-01-09T01:45:01.991Z] time:   0.000; rss:   44MB ->   49MB (   +4MB)	crate_injection
[2024-01-09T01:45:11.935Z] time:   9.141; rss:   49MB -> 1016MB ( +967MB)	expand_crate
[2024-01-09T01:45:11.935Z] time:   9.142; rss:   49MB -> 1016MB ( +967MB)	macro_expand_crate
[2024-01-09T01:45:11.935Z] time:   0.258; rss: 1016MB -> 1015MB (   -1MB)	AST_validation
[2024-01-09T01:45:11.935Z] time:   0.027; rss: 1015MB -> 1018MB (   +4MB)	finalize_imports
[2024-01-09T01:45:11.935Z] time:   0.108; rss: 1018MB -> 1018MB (   +0MB)	compute_effective_visibilities
[2024-01-09T01:45:11.935Z] time:   0.116; rss: 1018MB -> 1019MB (   +1MB)	finalize_macro_resolutions
[2024-01-09T01:45:15.203Z] time:   3.400; rss: 1019MB -> 1282MB ( +264MB)	late_resolve_crate
[2024-01-09T01:45:15.203Z] time:   0.163; rss: 1282MB -> 1284MB (   +1MB)	resolve_check_unused
[2024-01-09T01:45:15.459Z] time:   0.297; rss: 1284MB -> 1284MB (   +0MB)	resolve_postprocess
[2024-01-09T01:45:15.459Z] time:   4.117; rss: 1015MB -> 1284MB ( +269MB)	resolve_crate
[2024-01-09T01:45:15.714Z] time:   0.163; rss: 1284MB -> 1282MB (   -2MB)	write_dep_info
[2024-01-09T01:45:15.970Z] time:   0.165; rss: 1282MB -> 1282MB (   +0MB)	complete_gated_feature_checking
[2024-01-09T01:45:24.052Z] time:   0.497; rss: 1940MB -> 1901MB (  -39MB)	drop_ast
[2024-01-09T01:45:24.052Z] time:   8.104; rss: 1282MB -> 1746MB ( +464MB)	looking_for_derive_registrar
[2024-01-09T01:45:24.979Z] time:   9.303; rss: 1282MB -> 1743MB ( +461MB)	misc_checking_1
[2024-01-09T01:45:27.493Z] time:   2.087; rss: 1743MB -> 1823MB (  +80MB)	type_collecting
[2024-01-09T01:45:27.493Z] time:   0.490; rss: 1823MB -> 1819MB (   -4MB)	coherence_checking
[2024-01-09T01:45:42.328Z] time:  13.207; rss: 1819MB -> 1953MB ( +135MB)	wf_checking
[2024-01-09T01:46:00.366Z] time:   4.718; rss:  813MB -> 1577MB ( +764MB)	codegen_to_LLVM_IR
[2024-01-09T01:46:00.366Z] time: 109.242; rss: 1286MB -> 1577MB ( +291MB)	LLVM_passes
[2024-01-09T01:46:00.366Z] time: 117.170; rss:  637MB -> 1577MB ( +940MB)	codegen_crate
[2024-01-09T01:46:00.366Z] time:   0.187; rss: 1577MB ->  939MB ( -638MB)	free_global_ctxt
[2024-01-09T01:46:00.366Z] time:   0.133; rss:  939MB ->  944MB (   +5MB)	link_rlib
[2024-01-09T01:46:00.366Z] time:   0.141; rss:  939MB ->  944MB (   +5MB)	link_binary
[2024-01-09T01:46:00.366Z] time:   0.142; rss:  939MB ->  937MB (   -2MB)	link_crate
[2024-01-09T01:46:00.366Z] time:   0.143; rss:  939MB ->  937MB (   -2MB)	link
[2024-01-09T01:46:00.366Z] time: 132.034; rss:   27MB ->  143MB ( +116MB)	total
[2024-01-09T01:46:00.366Z]    Compiling aws-sdk-iam v1.3.0
[2024-01-09T01:46:00.366Z] time:   0.001; rss:   39MB ->   43MB (   +3MB)	parse_crate
[2024-01-09T01:46:01.293Z] time:  36.337; rss: 1743MB -> 2523MB ( +780MB)	type_check_crate
[2024-01-09T01:46:02.657Z] time:   2.174; rss:   43MB ->  289MB ( +247MB)	expand_crate
[2024-01-09T01:46:02.658Z] time:   2.175; rss:   43MB ->  289MB ( +247MB)	macro_expand_crate
[2024-01-09T01:46:02.658Z] time:   0.059; rss:  289MB ->  289MB (   +0MB)	AST_validation
[2024-01-09T01:46:02.658Z] time:   0.006; rss:  289MB ->  290MB (   +0MB)	finalize_imports
[2024-01-09T01:46:02.658Z] time:   0.015; rss:  290MB ->  290MB (   +0MB)	compute_effective_visibilities
[2024-01-09T01:46:02.658Z] time:   0.021; rss:  290MB ->  291MB (   +1MB)	finalize_macro_resolutions
[2024-01-09T01:46:03.219Z] time:   0.649; rss:  291MB ->  361MB (  +71MB)	late_resolve_crate
[2024-01-09T01:46:03.219Z] time:   0.041; rss:  361MB ->  362MB (   +0MB)	resolve_check_unused
[2024-01-09T01:46:03.475Z] time:   0.081; rss:  362MB ->  362MB (   +0MB)	resolve_postprocess
[2024-01-09T01:46:03.475Z] time:   0.816; rss:  289MB ->  362MB (  +72MB)	resolve_crate
[2024-01-09T01:46:03.475Z] time:   0.044; rss:  362MB ->  362MB (   +0MB)	write_dep_info
[2024-01-09T01:46:03.475Z] time:   0.046; rss:  362MB ->  362MB (   +0MB)	complete_gated_feature_checking
[2024-01-09T01:46:04.839Z] time:   0.107; rss:  493MB ->  494MB (   +0MB)	drop_ast
[2024-01-09T01:46:05.095Z] time:   1.554; rss:  362MB ->  460MB (  +98MB)	looking_for_derive_registrar
[2024-01-09T01:46:05.351Z] time:   1.807; rss:  362MB ->  463MB ( +101MB)	misc_checking_1
[2024-01-09T01:46:05.607Z] time:   0.386; rss:  463MB ->  502MB (  +39MB)	type_collecting
[2024-01-09T01:46:05.862Z] time:   0.113; rss:  502MB ->  515MB (  +13MB)	coherence_checking
[2024-01-09T01:46:09.129Z] time:   2.793; rss:  515MB ->  599MB (  +84MB)	wf_checking
[2024-01-09T01:46:13.294Z] time:   7.677; rss:  463MB ->  686MB ( +223MB)	type_check_crate
[2024-01-09T01:46:19.830Z] time:   6.148; rss:  686MB ->  886MB ( +200MB)	MIR_borrow_checking
[2024-01-09T01:46:21.736Z] time:   2.683; rss:  886MB ->  956MB (  +71MB)	MIR_effect_checking
[2024-01-09T01:46:23.102Z] time:   0.354; rss:  962MB ->  967MB (   +5MB)	module_lints
[2024-01-09T01:46:23.102Z] time:   0.355; rss:  962MB ->  967MB (   +5MB)	lint_checking
[2024-01-09T01:46:23.358Z] time:   0.362; rss:  967MB ->  968MB (   +1MB)	privacy_checking_modules
[2024-01-09T01:46:23.358Z] time:   1.166; rss:  957MB ->  968MB (  +12MB)	misc_checking_3
[2024-01-09T01:46:27.526Z] time:   4.251; rss:  968MB -> 1090MB ( +122MB)	generate_crate_metadata
[2024-01-09T01:46:27.526Z] time:   0.017; rss: 1090MB -> 1090MB (   +0MB)	monomorphization_collector_root_collections
[2024-01-09T01:46:29.418Z] time:   1.944; rss: 1090MB -> 1146MB (  +55MB)	monomorphization_collector_graph_walk
[2024-01-09T01:46:31.307Z] time:   1.705; rss: 1146MB -> 1182MB (  +36MB)	partition_and_assert_distinct_symbols
[2024-01-09T01:46:31.563Z] time:  30.141; rss: 2523MB -> 3476MB ( +953MB)	MIR_borrow_checking
[2024-01-09T01:46:43.763Z] time:  12.219; rss: 3476MB -> 3761MB ( +285MB)	MIR_effect_checking
[2024-01-09T01:46:50.315Z] time:   1.548; rss: 3776MB -> 3782MB (   +6MB)	module_lints
[2024-01-09T01:46:50.315Z] time:   1.549; rss: 3776MB -> 3782MB (   +6MB)	lint_checking
[2024-01-09T01:46:50.877Z] time:   1.663; rss: 3782MB -> 3797MB (  +15MB)	privacy_checking_modules
[2024-01-09T01:46:50.877Z] time:   5.817; rss: 3761MB -> 3797MB (  +36MB)	misc_checking_3
[2024-01-09T01:47:12.764Z] time:  21.787; rss: 3797MB -> 4225MB ( +428MB)	generate_crate_metadata
[2024-01-09T01:47:12.764Z] time:   0.107; rss: 4225MB -> 4225MB (   +0MB)	monomorphization_collector_root_collections
[2024-01-09T01:47:22.707Z] time:   8.825; rss: 4225MB -> 4436MB ( +211MB)	monomorphization_collector_graph_walk
[2024-01-09T01:47:54.736Z] time:  28.788; rss: 4436MB -> 4534MB (  +98MB)	partition_and_assert_distinct_symbols

Crate Highlights
I'd like to note that I'm aware that RSS based metrics for memory usage aren't the most accurate, but they are decent indicators. To that end, there are some seriously big numbers here.
AWS IAM
Most notably in the IAM crate, with type checking and borrow checking. The first round is ok, but the second round of it is what concerns me the the most.
time:  36.337; rss: 1743MB -> 2523MB ( +780MB)	type_check_crate
time:   6.148; rss:  686MB ->  886MB ( +200MB)	MIR_borrow_checking
time:   7.677; rss:  463MB ->  686MB ( +223MB)	type_check_crate
time:  30.141; rss: 2523MB -> 3476MB ( +953MB)	MIR_borrow_checking

This crate doesn't even finish compiling, and by the time the stage after partition_and_assert_distinct_symbols (whatever it is, since it hasn't been logged yet) has been underway, the
memory suage goes from 4436MB to 7900MB+.

AWS EC2
Similar behaviour in the EC2 crate as well, but it seems to be more centred around translation to IR and codegen. It seems the EC2 crate has a smaller footprint in terms of checkable usage
for contexts passed around than IAM though.
[2024-01-09T01:45:11.935Z] time:   9.141; rss:   49MB -> 1016MB ( +967MB)	expand_crate
[2024-01-09T01:45:11.935Z] time:   9.142; rss:   49MB -> 1016MB ( +967MB)	macro_expand_crate
[2024-01-09T01:46:00.366Z] time:   4.718; rss:  813MB -> 1577MB ( +764MB)	codegen_to_LLVM_IR
[2024-01-09T01:46:00.366Z] time: 117.170; rss:  637MB -> 1577MB ( +940MB)	codegen_crate

Crate Structure Considerations
Immediately, this rings a bell to me as poorly structured code that relies on insane amounts of context passed around, or transitive overuse of contexts between function calls. Another avenue
is async lifetimes dominating the context transfers on mutable objects.
Looking at the IAM crate in particular, first thing is that the crate seems match that, from what I can see it relies on async state passed around through invocations of various kinds. Within
those methods, transitive state can be constructed or copied to be used in other invocations for the IAM API. The bit that also adds to that by the example, everything is within an async context
and most operations are build from the client context itself.
#[::tokio::main]
async fn main() -> Result<(), iam::Error> {
    let config = aws_config::load_from_env().await;
    let client = aws_sdk_iam::Client::new(&config);
	// ... make some calls with the client
    Ok(())
}

I feel like this presents a nightmare situation for the borrow checker with production-level usage of this crate. Could be wrong, but the compilation data seems to indicate something in this realm.
I haven't looked too far into the EC2 crate characteristics though

Immediately, this rings a bell to me as poorly structured code ...

Well said.  I have run out of memory on 16GB laptop just trying to compile something that used one EC2 API call.  It's unacceptable.
AWS Developers, thank you for the efforts to make this usable.  However, your team really needs to address this even if it means releasing v2.0.0 of the crates with an entirely different approach.

I totally agree—especially about memory. The compile time itself I expect to be improved by the upcoming rustc parallel frontend, but memory is very difficult to work around and can make compiling the Rust SDK in CI prohibitively expensive. We're prioritizing investigating this and will keep folks posted.
We're also very cognizant of the fact that without decisive action, this issue is only going to get worse as more and more APIs are added to these existing crates.
I think, however, a lot of this comes from a very fundamental limitation: EC2 currently has 615 operations. They each need their own serializers and deserializers, inner types, etc.—unavoidably an absurdly large amount of code. There are definitely things we can do to shrink how much code we generate; I expect that these will yield improvements in the low double digit percentages. This is helpful, but I think ultimately an EC2 that needs 12GB of memory and compiles in 1m30s is not that much better than 16GB compiling in 2m assuming the rest of your build only needs 1GB and 10s.
We're concurrently investigating ways that we can allow customers to compile only parts of the EC2 crate. These are unfortunately also limited because it was recently discovered that features do not scale well on crates.io, which would prevent us from doing things like feature-per-operation.
Another last ditch item that we're floating is a way to generate ad-hoc SDKs with only a subset of operations included. This is obviously non-ideal for a number of reasons, but we agree that's it's incredibly frustrating to run out of memory on your build because you want to make one HTTP request to start an instance.
In any case, bear with us, this is among our top priorities to improve for this quarter.

It seems there is a big emphasis on EC2 here but I am curious if the other SDKs are being included as well. I am experimenting with aws-sdk-s3 and aws-sdk-sqs in an established project and they are considerably longer to build than the other 500 dependencies we have.

that's a good point—I updated the note. EC2 is the biggest, but there are a lot that are close behind.

These are unfortunately also limited because it was recently discovered that features do not scale well on crates.io, which would prevent us from doing things like feature-per-operation.

What do you mean by "does not scale well on crates.io"?
Is there a source for this that you could link to? I couldn't find anything after about 2 mins or so of searching.
web-sys has over 1500 different features and I haven't noticed anything as a user. I never use more than a couple dozen feature flags, however.
Or do you mean that having hundreds of feature flags in a single crate causes problem for crates.io's internal processes?

There was a Rust blog post on the feature scalability issue. Although, 23,000 is quite a bit larger than 600 😄

We are aware of a couple of crates that also have legitimate reasons for having more than 300 features, and we have granted them appropriate exceptions to this rule, but we would like to ask everyone to be mindful of these limitations of our current systems.

The limitation can be relaxed for a project. aws-sdk-ec2 should also belong to "have legitimate reasons" (I'm not sure, maybe @Turbo87 can comment this?), and 600 is also much smaller than 1500 of web-sys.

We also invite everyone to participate in finding solutions to the above problems.

Therefore, I think it's still worth considering the approach, because IMHO it's quite straightforward and improves the situation a lot immediately.
Maybe not at the first resort, but can give it a try if other ways don't work or require too much effort.

I thought it worth mentioning that I've ported our usage of aws-sdk-ec2 to shell out to the aws CLI as a workaround until the compile time issues are fixed: https://github.com/shotover/aws-throwaway/pull/41

I think, however, a lot of this comes from a very fundamental limitation: EC2 (edit: and other large crates like S3, SQS, IAM etc.) currently have hundreds of operations (EC2 has 615 as of today). They each need their own serializers and deserializers, inner types, etc.—unavoidably an absurdly large amount of code. There are definitely things we can do to shrink how much code we generate; I expect that these will yield improvements in the low double digit percentages. This is helpful, but I think ultimately an EC2 that needs 12GB of memory and compiles in 1m30s is not that much better than 16GB compiling in 2m assuming the rest of your build only needs 1GB and 10s.

I was having a read of the generated code in aws-sdk-ec2 to get a feeling for whether and how much doing more things dynamically might help. E.g. what if the various request/response types exposed a reflection API (read/write key/values by name paired with embedding the smithy models themselves into the crate), and then there was only one implementation of serialization/deserialization, pagination, etc.? I suppose the reflection code itself could end up costing as much to compile... but maybe not. Is this something that's been considered?

We're concurrently investigating ways that we can allow customers to compile only parts of the EC2 crate. These are unfortunately also limited because it was recently discovered that features do not scale well on crates.io, which would prevent us from doing things like feature-per-operation.

What about features for groups of operations? Is there any kind of internal grouping of operations in the underlying model?
Or otherwise by "commonness of use"? From quickly skimming the operation list, it looked to me that most of the operations I would never need because they cover some use case that I think of as obscure. I wonder how big the subset is of "stuff that most people use a lot"? I.e. would it be useful to have features for:

default: loads of people will need these
obscure: very special people use these, but most people won't
arcane-dark-magic: there are three people who use these operations, and AWS knows them by name.

:man_shrugging:
I'd be keen to know if there are particular avenues that are currently favored / considered most promising by the AWS team.
Thanks!
:bow:

The Parallel Rustc Working Groups is working on releasing a new parallel compiler front-end this year.
I tested it out on EC2 v1.42.0 and it's a big improvement over the current front-end. I ran the following compiles on my work laptop: a 2021 16" M1 MacBook Pro 32GB.

Current front-end cargo build -p aws-sdk-ec2 --timings: 148.6s (2m 28.6s)
Parallel front-end RUSTFLAGS="-Z threads=8" cargo +nightly build -p aws-sdk-ec2  --timings: 85.9s (1m 25.9s)

It looks like they still have many issues to solve before it can be the default, though.

Parallel frontend will also not help with memory usage — I'd expect it to make it worse.
For a really meaningful improvement, I think a solution is still needed that allows doing less work, not just trying to do the same amount of work faster. E.g. splitting it up into separate crates, feature flags, etc.
If Rust had better support for dynamic linking (e.g. via the proposed crABI and #[export] RFCs) then I'd be pushing for a solution that allows using pre-built binaries...

Update: For our team, here us how we solved aws-sdk compile times:
We wrote a rust cli designed build a custom version of aws-sdk-*
The cli program does this:

clones smithy-rs into temporary dir
strips down the json definition files to just the servicesand apis qe use
Calls the smithy-rs build process to generate the rust code
Copies that outputted rust code into a public github repo in our account
Tags it with the aws-sdk version

Finally we updated our project's Cargo.toml to refer to our github version
of aws-sdk-*
This works great because smithy-rs is powered from json definitions, and
there were no additional changes needed.
Once using our own github version of the aws-sdk-* crates, fresh compile
times for our project went from minutes down to seconds, recompile times
went from 20 seconds down to 8.
Using the mold linker brought this down to 1.5 seconds.
For us it was worth it because nearly tripling our compile times was
affecting the development flow, esp in web dev where you need to iterate
more quickly.
Keeping updated with upstream is as simple as running a cli command and
updating the version tags in our project's cargo.toml
We considered committing the generated crates to our project directory, but
there were two main reasons we elected to use a separate repo instead:

cargo clippy DOES NOT like generated aws-sdk code
It was still 10mb of source code with only four sdk apis enabled.

Looking forward to not needing to do this, but it does work well for now.
On Sat, May 18, 2024, 00:40 Jeff Parsons @.***> wrote:

Parallel frontend will also not help with memory usage — I'd expect it to
make it worse.
For a really meaningful improvement, I think a solution is still needed
that allows doing less work, not just trying to do the same amount of
work faster. E.g. splitting it up into separate crates, feature flags, etc.
If Rust had better support for dynamic linking (e.g. via the proposed
crABI and #[export] RFCs) then I'd be pushing for a solution that allows
using pre-built binaries...
—
Reply to this email directly, view it on GitHub
https://github.com/awslabs/aws-sdk-rust/issues/113#issuecomment-2118637540,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AABDF3W3ECM4XMHBBV2TMSTZC3LVJAVCNFSM46X6O4FKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMJRHA3DGNZVGQYA
.
You are receiving this because you commented.Message ID:
@.***>

@gahooa Would your team be willing to open source your implementation?

@vultix yes we would.  We are working on updating it to account for recent changes in the aws crates (service definitions moved to another repo) which prevented us from upgrading.

Could someone from AWS please provide some kind of update on your thinking about this issue? Things like:

Is anyone actively working on this?
Are there avenues you are currently exploring/favoring?
Are there options discussed here that you've already ruled out?
Is there anything from the Rust/Cargo side that might help?
Is there anything from users of this crate that might help?
Do you even see it as an issue worth solving?

Is anyone actively working on this?

Not actively no.

Are there avenues you are currently exploring/favoring?

We want to spend some time in 2025 doing some investigations. Where those will lead we don't know. Possibly it provides data back to other teams (cargo, rustc, etc). Possibly it leads to recommending use of particular tools for development if you fit a certain usage scenario (e.g. cranelift). It may go nowhere.

Are there options discussed here that you've already ruled out?

We can't break backwards compatibility and there are certain avenues we don't plan on pursuing (e.g. feature based enablement of operations).

Is there anything from the Rust/Cargo side that might help?

See above.

Is there anything from users of this crate that might help?

Knowing if compile times only becomes a concern for particular services or if you pull in more than N services, etc. Knowing if this is a development workflow or release workflow issue or both. Of course if someone has taken the time to profile things already we'd be interested in that data.

Do you even see it as an issue worth solving?

We believe it is an issue worth our time. I hesitate to say "solving" because I'm not sure how much of a difference we will be able to make. I expect if any difference it will be multiple small things over a longer period that help alleviate some of the compile time issues but I doubt we fully solve it (e.g. EC2 is simply a massive API and by extension a large amount of Rust code generated to compile).

Thanks for your reply, @aajtodd. :sparkling_heart:
One thing in particular jumped out at me:

We can't break backwards compatibility and there are certain avenues we don't plan on pursuing (e.g. feature based enablement of operations).

What about breaking the crate into many "sub-service" crates which are then rolled up as re-exports into the aws-sdk-ec2 crate? This could be done without breaking backward compatibility (aws-sdk-ec2 would still contain all the same items at exactly the same paths) but users that are significantly affected by compile times and are willing to deal with a bit of inconvenience to improve it could include, e.g., just aws-sdk-ec2-ami.
Would this be possible, or perhaps too hard if the service API definitions used for codegen don't currently have any meaningful separation beneath the level of "EC2"?

Is there anything from users of this crate that might help?

Knowing if compile times only becomes a concern for particular services or if you pull in more than N services, etc. Knowing if this is a development workflow or release workflow issue or both. Of course if someone has taken the time to profile things already we'd be interested in that data.

I maintain many CLI tools that pull in lots of AWS SDK crates one way or another. aws-sdk-ec2 is generally the real killer — if it's in the crate graph, it's by far the biggest crate being compiled even in projects that compile thousands, and it often takes over half the total compilation time. It's also the only crate I've come across that tends to OOM rustc even on fairly capable workstation hardware. So having some way to avoid the compiler ever seeing the vast majority of what's currently in aws-sdk-ec2 would be a huge win, even if it meant that I had to carefully pick the "sub-crates" I actually need.


Overriding cache

Hi,
I would like an option to override the currently existing cache.
This is especially useful for building docker images where the original cache might useless if the base image is updated.
Thanks

This is crazy

Why's that?

I desperately need this as well. CI is failing and there seems to be no way to cache bust

Cache hit occurred on the primary key Linux-maven-2d15385c799c9a1e31be36551973654b2bb55900a19be80d07dcf3e716f5daa9, not saving cache.

Not saving?
That's why my build incrementality does not work. I hoped I can accumulate files in the directory with ever build.

As a workaround I use key with randomized suffux and restore-key without it.
That way cache is restored using the restore-key which matches all previously saved caches and according to the spec we get the latest one: "If there are multiple partial matches for a restore key, the action returns the most recently created cache." - https://docs.github.com/en/free-pro-team@latest/actions/guides/caching-dependencies-to-speed-up-workflows#matching-a-cache-key
redisson$ git diff c5d170f3bec0fb38152d0dd22634cd2a03e7fb70~1 c5d170f3bec0fb38152d0dd22634cd2a03e7fb70
diff --git a/.github/workflows/maven.yml b/.github/workflows/maven.yml
index 6472f5313..2e0f96427 100644
--- a/.github/workflows/maven.yml
+++ b/.github/workflows/maven.yml
@@ -13,13 +13,16 @@ jobs:
       uses: actions/setup-java@v1
       with:
         java-version: 1.8
+    - name: Generate random cache key to ensure the cach directories are saved after build
+      id: random-cache-key
+      run: head /dev/random -c 32 > random-cache-key
     - name: Setup caching directories for local Maven repo and for DB of successfuly built hashversioned modules
       uses: actions/cache@v2
       with:
         path: |
           ~/.m2/repository
           ~/successful-hashvers
-        key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
+        key: ${{ runner.os }}-maven-${{ hashFiles('random-cache-key') }}
         restore-keys: |
           ${{ runner.os }}-maven-
     - name: Determine affected Maven projects

https://github.com/avodonosov/redisson/commit/c5d170f3bec0fb38152d0dd22634cd2a03e7fb70
So, every build restores the latest cache and repackages it as a new cache entry. It can probably result in redundant storage use, unless the implementation inherit old data intelligently, al-la OverlayFS. On the other hand, old caches will be garbage collected by github in 7 days, so maybe not an issue at all.
If this is how updating the cache is intended to be done, it would make sense to add this to documentation.
If cache updating it not intended at all, then this is big miss and the action does not deserve to be called cache.
Updatable caches can be very useful for build tools like Gradle and Bazel that have build cache functionality that allows to skip tasks whose inputs hasn't changed. Some people reported 10 times build speedup: https://redfin.engineering/we-switched-from-maven-to-bazel-and-builds-got-10x-faster-b265a7845854
See also: https://about.gitlab.com/blog/2020/09/01/using-bazel-to-speed-up-gitlab-ci-builds/
I am exploring something similar for Maven: https://github.com/avodonosov/hashver-maven-plugin

Very elegant, @avodonosov thank you

Here's my solution: https://github.com/actions/cache/pull/498#issuecomment-753804797

Here's my solution: https://github.com/actions/cache/pull/498#issuecomment-753804797

This shouldn't be marked stale - it's waiting on #498

Following @avodonosov, a cool workaround is leveraging ${{ github.sha }}:
- name: Cache
  uses: actions/cache@v3
  with:
    path: myfile
      key: ${{ runner.OS }}-${{ github.repository }}-myfile-${{ github.sha }}
      restore-keys: |
        ${{ runner.OS }}-${{ github.repository }}-myfile-

You can programmatically delete cache entries: https://github.blog/changelog/2022-06-27-list-and-delete-caches-in-your-actions-workflows/ and you can also delete them from the website...

:sob:


How to change the font size of Katex math symbol and backtick text?

Hello there, thank you for the beautiful template. I want to ask if it's possible to change/reduce the font size of the Katex math symbols and backtick text (e.g. text)? It seems like on the website (see the screenshots), the math symbols and backtick texts are always larger than the regular text, it would be much better if their sizes can be reduced.
I have searched online and didn't get much useful information. There is a post talking about changing the size of MathJax symbols, but didn't find anything about Katex symbols. And here is a post in which the math symbol looks in normal size, so I think it's possible to do so but have no idea how to do it. Do you have any suggestions? Any information is greatly appreciated.
screenshot1 (the Katex math symbol is larger)

screenshot2 (the backtick text is larger)

@xiangli2pro Hi there, the CSS identifiers you are looking for are code and Katex. You can override them as following in your custom CSS.
code {
}
.katex {
}

You would need to adjust the font-size of these according to your liking. However, note that the Katex uses a different font than the theme, therefore, the characters might still look slightly different.
Hope this helps. Going to close this issue. Feel free to reopen it if you need additional information.


Network Files Access

To Reproduce

Go to 'a folder on a network'
Click on 'any folder'
Folders all appear as "empty"'
Seems like there's no access to files on the network

Screenshots
See attached screenshot
Desktop

OS: Windows 10 Pro 2004 [build: 19041.264]

Just wondering, what does "closed" status mean?
Does that indicate that the problem has been resolved or will not be done?
Thanks.

As we mentioned, it is a duplicate of #642

@mlagardenhose "Closed" means that we are not going to be tracking the specific issue mentioned here since we are tracking it in #642.

got it. thanks.
bear with me as I am new to this platform.
great work, keep it up! can't wait for official release version
On Thu, Jun 11, 2020 at 10:51 AM Yair Aichenbaum<EMAIL_ADDRESS>wrote:

@mlagardenhose https://github.com/mlagardenhose "Closed" means that we
are not going to be tracking the specific issue mentioned here since we are
already tracking it in #642
https://github.com/files-community/files-uwp/issues/642.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/files-community/files-uwp/issues/982#issuecomment-642710093,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AP5FZM7KSG2XNXM7RQGGPRLRWDVNLANCNFSM4N2POVUQ
.

--
www.jcndesign.com http://jcndesign.com/


Add option to collect cluster, datacenter and datastore metrics

What does this PR do?
Add ability to collect cluster metrics.
This also fixes getting metrics for datastore and datacenter, which was not actually working
It won't collect this metrics by default, since getting them from vCenter considerably increase the collection time
Motivation
What inspired you to submit this pull request?
Review checklist

[ ] PR has a meaningful title or PR has the no-changelog label attached
[ ] Feature or bugfix has tests
[ ] Git history is clean
[ ] If PR impacts documentation, docs team has been notified or an issue has been opened on the documentation repo

Additional Notes
Anything else we should know when reviewing?

Codecov Report

Merging #2655 into master will decrease coverage by 1.26%.
The diff coverage is 61.36%.

@@            Coverage Diff             @@
##           master    #2655      +/-   ##
==========================================
- Coverage   83.67%   82.41%   -1.27%
==========================================
  Files         598       17     -581
  Lines       35264     1359   -33905
  Branches     4262      163    -4099
==========================================
- Hits        29507     1120   -28387
+ Misses       4516      203    -4313
+ Partials     1241       36    -1205


"Fetching Before Navigation" example does not work

Version
2.5.3
Reproduction link
https://github.com/vuejs/vue-router/issues/1144
Steps to reproduce
The previously closed issue #1144 was opened because the documentation is incorrect. The message from maintainers on that issue states that the method used on the "Fetching Before Navigation" example for the vue-router docs does not work: https://router.vuejs.org/en/advanced/data-fetching.html
What is expected?
this.post to be set
What is actually happening?
this.post is null (see issue #1144)

The example does work: http://jsfiddle.net/onve61xt/
But I understand this is confusing because you shouldn't see an error. See #1499
I never said the example doesn't work....

Next time, make sure to include a reproduction, please


tree-sitter: no longer broken on aarch64

Motivation for this change
The latest tree-sitter seems to build fine on aarch64.
Draft while I finish building/testing.
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x0] Fits CONTRIBUTING.md.

Closing, since #118408 was merged


General Issue: Remove CRAN comments file

Background Information
remove cran comments file following release
Definition of Done
No response

Hi @tarunsamanta2k20 I would think to delete both.


Algolia search using objective c client

ASRemoteIndex index or ASQuery query is returning invalid jason. Can anyone please help me out.
ASRemoteIndex *index = [apiClient getIndex:@"YourIndexName"];
[index search:[ASQuery queryWithFullTextQuery:@"jim"]
success:^(ASRemoteIndex *index, ASQuery *query, NSDictionary *answer) {
// answer object contains a "hits" attribute that contains all results
// each result contains your attributes and a _highlightResult attribute that contains highlighted version of your attributes
} failure:nil];
// setting some query parameters
ASQuery* query = [ASQuery queryWithFullTextQuery:@"jim"];
query.hitsPerPage = 5;
[index search:query success:^(ASRemoteIndex *index, ASQuery *query, NSDictionary *answer) {
// answer object contains a "hits" attribute that contains all results
// each result contains your attributes and a _highlightResult attribute that contains highlighted version of your attributes
} failure:nil];
Thanks in advance

Hi @subhojit123,
Could you give me the error returned by the api client?
Best regards


Usage of std collection even when BGFX_CONFIG_USE_TINYSTL == 1

In bgfx_p.h is used std::list without custom allocator. As result, global operator new is called.
https://github.com/bkaradzic/bgfx/blob/master/src/bgfx_p.h#L2473
Sadly it makes little hole in my memory management. Is it intended?

Not intended but I know about this problem. I was planning to remove need for list but still didn't get around it.


why Azure file share SDK is being degraded over time?

How come, that the old package works better than the new one ?
In particular speaking of [DEPRECATED] "WindowsAzure.Storage" Version="9.3.3"
or "Microsoft.Azure.Storage.File" Version="11.2.2"
(question #1 - how long can I keep using it until I'm forced to use the "NEW" "Azure.Storage.Files.Shares" Version="12.5.0" ????)
and as for the main question - Your page  clearly states that:

...Azure SDKs that we believe will make the libraries easier to learn and integrate into your software.

Which in the case "Azure.Storage.Files.Shares" Version="12.5.0" - is completely opposite.
I'm as developer am being "forced" to use this library, without any knowledge for how long the old one will keep working,
and I would dare saying - one which is worse, and more difficult to use and integrate than the OLD one.
I'm speaking in particular about the File Upload API, which is now forcing me to write code that requires me to split files into chunks and upload them piece by peace instead of just pointing the SDK to the file, and asking it to UPLOAD IT!
(that's at least how it perfectly worked in the OLD  "WindowsAzure.Storage" package)
so the question #2  are You planning to provide a proper API in the SDK that would allow users to seamlessly upload files bigger than 4MB ? (or for that matter at least samples !?!?!)
And lastly - (I do not have the data yet, to back this statement up) - but it feels like it's also slower !?!?!?
#3. Whys is the performance worse ??

Thank you for your feedback.  Tagging and routing to the team best able to assist.  Please be aware that due to the holidays, responses may be delayed.

@martys thank you for the feedback.
Have you looked at this sample? It shows how you can easily create a file share and upload a file to it without having to manually split the file into blocks.
Would you be able to share more about the scenario where you are seeing slowness?

Yes of course. Following this sample is exactly what got me into the error - that request size is too big, and that I must split my file into chunks .
to be more exact : Azure.Storage.Files.Shares Azure.RequestFailedException: The request body is too large and exceeds the maximum permissible limit.

@martys thanks for the quick reply. Let me take a quick look. The Upload method you are using should use the partitioned uploaded that would split the file into multiple chunks.

Can you try replacing the line:
 file.UploadRange(
        new HttpRange(0, stream.Length),
        stream);

with
file.Upload(stream);

@AlexGhiondea  Thanks. Will test it out tomorrow, and will measure the speed difference also.
Any word on - for how long the old deprecated package can be used ?

@martys great! Let me know if that works and also what you perf number show!
You should be able to continue to use the deprecated package, however we strongly recommend that you consider upgrading to our latest version of those libraries.

Interestingly - my feelings were deceptive... the old package is actually noticeably slower than the new one
And thank You for a very quick response and guidance. I did jump to conclusions too quickly :(


Listar Link

[ LOW LEVEL ]

When any special char is added on the "Obs textbox" the BD crash.
Similarweb doesnt reconise links after TLD (Top Level Domain), remove everything from it, please.

Issue Resolved: [LOW LEVEL]
We've successfully resolved the reported problems:
Database Crashes with Special Characters: We've implemented measures to handle special characters in the "Obs" textbox, preventing crashes and enhancing security.
Incorrect Link Recognition by Similarweb: We've adjusted the link extraction process to ensure accurate link processing by Similarweb.