RequestModelFactory should look at Config.title attribute

Description
This a PR to fix part of the issues from https://github.com/doccano/doccano/issues/1307, which caused failures when trying to create Custom REST Request models.
Type of change
Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?
Created a Custom REST Request.

Thanks!


flutter screenshot should work on multiple views

Currently the screenshotting function of CLI (flutter screenshot), which calls Shell::OnServiceProtocolScreenshot and Shell::OnServiceProtocolScreenshotSKP, assumes that Flutter has a single view. We should make it able to handle multiple screenshots.
As I'm working on multi-view, I'm planning to keep the current behavior of targeting the implicit view (view 0) for now, but we'll need to make it work with other views.

@dkwingsmt is this still something we are planning to implement?

Closing this as stale with no response. @dkwingsmt feel free to re-open if this you think this work will happen.

@christopherfujino I never planned to work on it since it involves multiple teams (such as CLI), and apparently it was a low priority even when I was still on multiview. I mostly opened it because naturally flutter screenshot should work on multiple views now that we support multiple views. I suggest keeping it open.


M3-611 Make a Credit Card Payment

Purpose
Users need a means by which they can make payment using a stored credit card.

Coverage increased (+0.02%) to 27.656% when pulling 441ea274c02648a5a7bada89ae911266a40501db on rjstires:M3-611 into 7f1b918a357f1e0f38c238d3171cb3b0c8172716 on linode:develop.


Codebase: Upgradeable Contracts: Unify/Fix Storage Location

in upgradeable contracts storage is mostly handled by structs per upgradeable contract.
storage location of these is fixed using a hash stored in a constant *_STORAGE_LOCATION.
currently, the calculation of these storage location is inconsistent as the underlying naming pattern has not yet been properly defined.
step 1: define naming pattern
step 2: calculated resulting storage locations for all contracts
step 3: fix contracts to use above storage locations in a way that storage locations can be pulled throug a simple grep from all contracts (to support simple checks that there are not collisions)
example:
keccak256(abi.encode(uint256(keccak256("gif-next.contracts.component.Component.sol")) - 1)) & ~bytes32(uint256(0xff));
what is needed: how to create simple and unique name strings (simpler and more generic than "gif-next.contracts.component.Component.sol", eg "etherisc.gif.Component")

Name string must contain version (major, minor, patch) of a contract. Major version is release version
<EMAIL_ADDRESS>or<EMAIL_ADDRESS>

storage address variable format is: _STORAGE_LOCATION_V<release.storage-version>


Support connecting via SOCKS proxy (Tor)

Orbot, Android incarnation of Tor, acts as SOCKS proxy. Applications willing to use Tor need to implement SOCKS protocol.
Although Orbot provides HTTP proxy too, from my experience it is much less reliable than SOCKS.

#1040


Fixing up bat/abcipp-rebase-master for v0.6.1

This PR is fixing up bat/abcipp-rebase-master such that the end-to-end tests will pass with the Tendermint version as specified in CI for ABCI++ (which is Tendermint commit 29e5fbcc648510e4763bd0af0b461aed92c21f30).
(this PR should target #1151 which is bat/abcipp-rebase-master but rebased for v0.6.1)

skip test_node_connectivity using tests/skip-tendermint-8840 feature flag
[ ] skip test_genesis_validators using tests/skip-tendermint-8840 feature flag (this test isn't in the bat/abcipp-rebase-master branch - so this feature flag should be applied to that test during merge, or elsewhere)
use the specific commits of the dependencies ibc-rs, tendermint-rs and tower-abci that work

Replacing this PR with https://github.com/anoma/anoma/pull/1179


coordinate systems

Hello, this is a great job, but I have a confusion. I want to know that is the point cloud in the world coordinate system or in the LiDAR coordinate system?

Hi, @123myx, the point cloud is in the world coordinate system.
You could take a look at the Argoverse API github for more information.
Cheers.


Limit the depth of the debug output console's log view

The debug console's log appender continuously pushes to a TextFlow control and older entries never get deleted, which means that memory usage for that functionality could get unreasonable after a while.

The console output appender now appends logs into a circular buffer rather than directly into the TextFlowArea. Size of the buffer is exposed as a user preference.
TextFlowArea is now refreshed periodically (every 500 ms, only if needed) instead of synchronously on the append() method.
This avoids the issue of logging throttling down global app performances, as inserting into the TextFlow observableList becomes costly as the list grow, as well as fixes the reported issue of having an ever growing text area.


Add JSON support

This commits adds JSON support by adding Jsmn
(http://zserge.com/jsmn.html), a minimalistic JSON parser.
Signed-off-by: Drasko DRASKOVIC<EMAIL_ADDRESS>

@foogod thanks! Should be fixed now.

On the other hand - I did no change in Jsmn code. Maybe this can be introduced as a git submodule?

Yep, I just tested the version with submodule, and it works. Would you prefer PR with submodule version?

Thanks for this @drasko. I added a few minor comments inline.
The git submodule would be preferable, please.
Also, I think there's a solid chance we'll end up with more than one JSON library in the long term. Could you please name the example something like "json_jsmn_simple"?

@projectgus thanks for the review. I will close this PR and open new one with fixes and submodule approach.

Great! You don't need to close this PR, you can squash your commits and force push a new version of the PR on top of this one (as described here: http://eli.thegreenplace.net/2014/02/19/squashing-github-pull-requests-into-a-single-commit/ )
(Or alternatively just keep adding commits, and github's fancy web interface will let me squash them into one commit when I merge the PR.)

Fixed and squashed.

This looks good to me except you've somehow picked up two of my commits into the PR when you rebased - not sure how?
@foogod, are you OK with the rest of these changes? I can cherry-pick it into master by hand if need be.

Sorry I've been sorta sporadic lately..  Looks fine to me.  Feel free to merge however you think is best..

Thanks again for this, and sorry for the delay. I've cherry-picked it as e7607ffc.


Future: Support workspace

Support everything needed for the change to workspace tokens.
https://api.slack.com/docs/working-with-workspace-tokens
In general, these new Slack apps cannot do the following:

Impersonate users — your app can't post as as team member, nor award reacji on behalf of a user. Your app is your app and only your app. (Some impersonation options will be made available at a later date.)
Contain a bot user — workspace tokens make bot tokens and users obsolete; your app is a bot if you want it to be but you don't need a bot user feature to provide that functionality.
Connect to the RTM API — it's just not possible or practical with this permission model. Please use the Events API for near-real-time delivery instead.
Incoming webhooks aren't part of the new Slack app model. Apps should use chat.postMessage with the new channel & conversation selection model in the Permissions API instead.
Sign in with Slack is not yet available for new Slack apps.

Some functionality previously made possible through impersonating the authenticating user, like creating and inviting users to channels, is no longer possible with this model.

Quoting from that same page:

Using workspace tokens isn't recommended unless you've already built an app using them. New workspace apps may no longer be created.

— Sounds like they've deprecated workspace apps already? 🤔

yeah, closing.


nixos/prometheus-exporters/artifactory: init at 1.9.0

Motivation for this change
Adds a Prometheus exporter to scrape metrics from the API of JFrog Artifactory instances.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[ ] NixOS
[ ] macOS
[x] other Linux distributions

[x] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

This is a semi-automatic executed nixpkgs-review with nixpkgs-review-checks extension. It is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch).
If you have any questions or problems please reach out to SuperSandro2000 on IRC.
Result of nixpkgs-review pr 111364 run on x86_64-darwin 1

1 package failed to build and are new build failure:

prometheus-artifactory-exporter: log was empty

This is a semi-automatic executed nixpkgs-review with nixpkgs-review-checks extension. It is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch).
If you have any questions or problems please reach out to SuperSandro2000 on IRC.
Result of nixpkgs-review pr 111364 run on x86_64-darwin 1

1 package failed to build and are new build failure:

prometheus-artifactory-exporter: log was empty

hash mismatch in fixed-output derivation '/nix/store/h850l0r7ylmv6mq8xk5vqd2cx4khf9x5-artifactory_exporter-1.9.0-go-modules':                                                                 wanted: sha256:18piysgiw62qhbv4zlr4lyl872h7xmpz2cxskchvfh73rpcz9izj                                                                                got:    sha256:1594bpfwhbjgayf4aacs7rfjxm4cnqz8iak8kpm1xzsm1cx1il17                                                                                cannot build derivation '/nix/store/sz330gjfvn335r72a3v11p1k262469gk-artifactory_exporter-1.9.0.drv': 1 dependencies couldn't be built                                                      cannot build derivation '/nix/store/i81w4hnbw9kczapnx11hmbx1hngfbplc-env.drv': 1 dependencies couldn't be built

hash mismatch in fixed-output derivation '/nix/store/h850l0r7ylmv6mq8xk5vqd2cx4khf9x5-artifactory_exporter-1.9.0-go-modules':                                                                 wanted: sha256:18piysgiw62qhbv4zlr4lyl872h7xmpz2cxskchvfh73rpcz9izj                                                                                got:    sha256:1594bpfwhbjgayf4aacs7rfjxm4cnqz8iak8kpm1xzsm1cx1il17                                                                                cannot build derivation '/nix/store/sz330gjfvn335r72a3v11p1k262469gk-artifactory_exporter-1.9.0.drv': 1 dependencies couldn't be built                                                      cannot build derivation '/nix/store/i81w4hnbw9kczapnx11hmbx1hngfbplc-env.drv': 1 dependencies couldn't be built

This is a semi-automatic executed nixpkgs-review with nixpkgs-review-checks extension. It is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch).
If you have any questions or problems please reach out to SuperSandro2000 on IRC.
Result of nixpkgs-review pr 111364 run on x86_64-darwin 1

1 package built:

prometheus-artifactory-exporter

This is a semi-automatic executed nixpkgs-review with nixpkgs-review-checks extension. It is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch).
If you have any questions or problems please reach out to SuperSandro2000 on IRC.
Result of nixpkgs-review pr 111364 run on x86_64-darwin 1

1 package built:

prometheus-artifactory-exporter

This is a semi-automatic executed nixpkgs-review with nixpkgs-review-checks extension. It is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch).
If you have any questions or problems please reach out to SuperSandro2000 on IRC.
Result of nixpkgs-review pr 111364 run on x86_64-linux 1

1 package blacklisted:

tests.nixos-functions.nixos-test

1 package built:

prometheus-artifactory-exporter

This is a semi-automatic executed nixpkgs-review with nixpkgs-review-checks extension. It is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch).
If you have any questions or problems please reach out to SuperSandro2000 on IRC.
Result of nixpkgs-review pr 111364 run on x86_64-linux 1

1 package blacklisted:

tests.nixos-functions.nixos-test

1 package built:

prometheus-artifactory-exporter

Squashed - let me know if you need anything else.

@ofborg test tests.nixos-functions.nixos-test

@GrahamcOfBorg test prometheus-exporters.artifactory

CC @WilliButz @kalbasit if any of you has time to review this. Thanks!


Make default placement constraint values be valid stringified JSON.

Values should be a JSON-stringified array of constraints (which are expressed as arrays of strings).
Example: "hostname:MAX_PER:1" becomes "[[\"hostname\", \"MAX_PER\", \"1\"]]"
Empty strings become JSON-stringified empty arrays, "" -> "[]".
Chat with Andrew:
Murilo Pereira [12:14 PM]
for values that used to default to `"hostname:MAX_PER:1"`, does it matter
whether the JSON equivalent has the `1` as a string or number? e.g.
`"[[\"hostname\", \"MAX_PER\", \"1\"]]"` for string

Andrew Hoskins [12:21 PM]
Hey! Make it a string

Murilo Pereira [12:21 PM]
cool. and empty defaults just `""` right?

Andrew Hoskins [12:26 PM]
Empty state should actually be `"[]"`

[12:26]
Because the meme type is <…>+json, so it expects a valid JSON value

One of the things I was struggling with (after finding out about the need for the propagating mustache triple bracket on friday) was the need for double-escaping quotes in config.json files, e.g. https://github.com/mesosphere/dcos-commons/pull/1903/files#diff-56550f345e72e9ddaffbc774bb87a7a9R124
Another was missing test config files with hard-coded constraints like https://github.com/mesosphere/dcos-commons/pull/1903/files#diff-9c0a8da33f4e191b89245b7045401adfL4
The other things was that making [] be the default value instead caused the constraint parser to treat it as a empty constraint instead of an empty list of constraints:
testcase com.mesosphere.sdk.elastic.scheduler.ServiceTest >
  testSpec: java.io.IOException: Invalid number of entries in rule. Expected 2 or 3, got 0: []
    at com.mesosphere.sdk.offer.evaluate.placement.MarathonConstraintParser.parseRow(MarathonConstraintParser.java:106)

which is addressed by https://github.com/mesosphere/dcos-commons/pull/1903/commits/b8662cfffe13ca8cf585f7b86a9cb8675bf2c017
I just noticed https://github.com/mesosphere/dcos-commons/blob/INFINITY-2620-placement-constraint-defaults/sdk/scheduler/src/test/java/com/mesosphere/sdk/offer/evaluate/placement/MarathonConstraintParserTest.java#L296-L299 so it's possible we don't need to double-escape double quotes in config.json files and use single quotes instead, I'll push a commit trying that after the checks for the current code pass.

To recap:
config.json files need stringified JSON to double-escape double quotes
"placement_constraint": {
  "type": "string",
  "description": "The Marathon-style placement constraint string controlling node placement",
  "default": "[[\\\"hostname\\\", \\\"UNIQUE\\\"]]",
  "media": {
    "type": "application/x-zone-constraints+json"
  }
}

marathon.json.mustache files have triple-brace mustache templates
"PLACEMENT_CONSTRAINT": "{{{nodes.placement_constraint}}}",

expands into:
"PLACEMENT_CONSTRAINT": "[[\"hostname\", \"UNIQUE\"]]",

svc.yml files need the single quote around the triple braces
placement: '{{{PLACEMENT_CONSTRAINT}}}'

expands into:
placement: '[["hostname", "UNIQUE"]]'

Maybe we could make it possible to just use a regular array value in config.json in the future, but for now this should work.

@benclarkwood https://jira.mesosphere.com/browse/INFINITY-2769


src in the project error

I have been importing the project to eclipse and I did. But there are many errors in >src such as
inside info.guardianproject.onionkit.ui , com.google.zxing.intergration.android and many else below of it

Please view the readme and build instructions for how to retrive all of the git submodules properly.
Also, you can take a look at our next-gen chat client called Zom, which is now here: https://github.com/zom/zom-android and uses Android Studio and Gradle.


[CLI]: tox fails to collect test with import

Describe the bug

when trying to run tox pytest it fails on import wandb

import wandb

.tox/py311/lib/python3.11/site-packages/_pytest/python.py:617: in _importtestmodule
    mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
.tox/py311/lib/python3.11/site-packages/_pytest/pathlib.py:564: in import_path
    importlib.import_module(module_name)
/usr/local/Cellar/python@3.11/3.11.4/Frameworks/Python.framework/Versions/3.11/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1204: in _gcd_import
    ???
<frozen importlib._bootstrap>:1176: in _find_and_load
    ???
<frozen importlib._bootstrap>:1147: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:690: in _load_unlocked
    ???
.tox/py311/lib/python3.11/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
test/mlflow_tracker_test.py:8: in <module>
    from af_experiment_tracking.mlflow_tracker import MLFlowExperimentRun
src/af_experiment_tracking/__init__.py:3: in <module>
    from .wandb_tracker import WandbTracker
src/af_experiment_tracking/wandb_tracker.py:4: in <module>
    import wandb
.tox/py311/lib/python3.11/site-packages/wandb/__init__.py:26: in <module>
    from wandb import sdk as wandb_sdk
.tox/py311/lib/python3.11/site-packages/wandb/sdk/__init__.py:4: in <module>
    from .artifacts.local_artifact import Artifact  # noqa: F401
.tox/py311/lib/python3.11/site-packages/wandb/sdk/artifacts/__init__.py:1: in <module>
    from wandb.sdk.artifacts.local_artifact import Artifact as LocalArtifact
.tox/py311/lib/python3.11/site-packages/wandb/sdk/artifacts/local_artifact.py:33: in <module>
    from wandb.sdk.artifacts.artifact_manifests.artifact_manifest_v1 import (
.tox/py311/lib/python3.11/site-packages/wandb/sdk/artifacts/artifact_manifests/artifact_manifest_v1.py:6: in <module>
    from wandb.sdk.artifacts.storage_policies.wandb_storage_policy import WandbStoragePolicy
.tox/py311/lib/python3.11/site-packages/wandb/sdk/artifacts/storage_policies/wandb_storage_policy.py:11: in <module>
    from wandb.apis import InternalApi
.tox/py311/lib/python3.11/site-packages/wandb/apis/__init__.py:42: in <module>
    from .internal import Api as InternalApi  # noqa
.tox/py311/lib/python3.11/site-packages/wandb/apis/internal.py:3: in <module>
    from wandb.sdk.internal.internal_api import Api as InternalApi
.tox/py311/lib/python3.11/site-packages/wandb/sdk/internal/internal_api.py:33: in <module>
    from wandb_gql import Client, gql
.tox/py311/lib/python3.11/site-packages/wandb/vendor/gql-0.2.0/wandb_gql/__init__.py:1: in <module>
    from .gql import gql
.tox/py311/lib/python3.11/site-packages/wandb/vendor/gql-0.2.0/wandb_gql/gql.py:1: in <module>
    from wandb_graphql.language.parser import parse
.tox/py311/lib/python3.11/site-packages/wandb/vendor/graphql-core-1.1/wandb_graphql/__init__.py:43: in <module>
    from .graphql import (
.tox/py311/lib/python3.11/site-packages/wandb/vendor/graphql-core-1.1/wandb_graphql/graphql.py:3: in <module>
    from .language.parser import parse
.tox/py311/lib/python3.11/site-packages/wandb/vendor/graphql-core-1.1/wandb_graphql/language/parser.py:3: in <module>
    from .lexer import Lexer, TokenKind, get_token_desc, get_token_kind_desc
E     File "/Users/erankrakovsky/Projects/utils/af-experiment-tracking/.tox/py311/lib/python3.11/site-packages/wandb/vendor/graphql-core-1.1/wandb_graphql/language/lexer.py", line 224
E       """Reads a number token from the source file, either a float
E       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E   SyntaxError: invalid escape sequence '\.'

Additional Files
No response
Environment
WandB version:
0.15.4
OS:
MacOS 13.4 (22F66)
Python version:
3.7, 3.8, 3.9, 3.10, 3.11
Versions of relevant libraries:
Additional Context
in lexer.py in line 224 add r before the docstring:
def read_number(source, start, first_code):
    r"""Reads a number token from the source file, either a float
    or an int depending on whether a decimal point appears.

    Int:   -?(0|[1-9][0-9]*)
    Float: -?(0|[1-9][0-9]*)(\.[0-9]+)?((E|e)(+|-)?[0-9]+)?"""

tox version 4.0.0

Hey @ekrako, I was able to properly import wandb and pass a test using tox 4.0.0 without experiencing the error. I recommend creating a new environment with just the relevant packages and versions and retesting. Please let me know if this works on your end.

Uma Krishnaswamy commented:
Hi @ekrako, since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!


feat (Issue #65: aws-alb-lambda construct)

Issue 65, if available:
Description of changes:
Architecture, Props & Properties of the construct submitted for review
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

AWS CodeBuild CI Report

CodeBuild project: codebuildgithubautobuildPro-fkVQbXRiQi6A
Commit ID: f412ec3a5bacf989f57050fb40e19ef328b957cf
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: codebuildgithubautobuildPro-fkVQbXRiQi6A
Commit ID: 7091c6d9c104ae6e0c51eb8403251a21c3126c19
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: codebuildgithubautobuildPro-fkVQbXRiQi6A
Commit ID: f72018315cbf32d0fe7135f7aa6afdc7d74ea594
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository


Update read_str

Update version and adds @dimfeld branch

Current master seems to solve the problem better, with no impact on performance and no extra deps

Current master seems to solve the problem better, with no impact on performance and no extra deps


Uploaded file variable expected "Promise" object, but "false".

currently, I'm using express to use apollo-upload-server middleware. but It works quite strange. the file variable in resolver has "false" value not a "promise".
I've checked with my own middleware to check how apolloUploadExpress(/* ... */) middleware set a variables object. and It successfully assigned promise but It changes from promise to false when it passed to resolver function.
right below code snippet is my own middleware code:
app.use("/graphql",
    bodyParser.json(),
    apolloUploadExpress(),
    (request, _, next) => {
        console.log(request.body.variables.thumbnail); // this prints 'Promise { ... }'
        next();
    },
    graphqlExpress({
        schema: require("./schema"),
    }),
);

and this is resolver code:
{
    push: async (_, args) => {
        // the 'thumbnail' variable accepted as file.
        const { thumbnail } = args;

        // so we could expect that right below code snippet will prints about "Promise" object.
        // but just prints "false".
        console.log(thumbnail);
    },
};

Are both the server and client packages at the latest alpha versions?

@jaydenseric
yes.
        "apollo-upload-client": "^7.0.0-alpha.1",
        "apollo-upload-server": "^4.0.0-alpha.1",

Solved.
I didn't specify Upload: GraphQLUpload item on resolvers object when I make executable schema. yeah, I Know It's too pathetic mistake to tell. by the way, thanks for helping me out.

Glad it was an easy fix 😅


[WIP]: Add sparse marlin AQT layout

Work in progress for #549
WIP

[x] Wait for #733 to be merged
[ ] Figure out the problems with the shapes
[ ] Get torch.compile to work
[ ] Check if it works with llama3

so you should have perms to trigger CI, keep in mind that CI wont run if you have merge conflicts like now

@jcaip - Thank you for the comments! I am going to continue working on this today and will take care of them.
I think I have most things hooked together. There seems to be a weird bug somewhere but I will grind through it.
Once we merge this, I’d be happy to add it to SAM and run the benchmarks!
Also, feel free to let me know if there is anything that I can help out on for the PTC poster. I managed to finish mine earlier this week and so I have a bit more time 🙂

so you should have perms to trigger CI, keep in mind that CI wont run if you have merge conflicts like now

Awesome! I am going to resolve them and try it out once I am at the computer

Hi, I did a bit more work on this PR this last few days and, if possible, I would like to ask for another preliminary review on it. It is not yet finished because I am still having a bug somewhere that is causing llama2 to output some weird text (I am using llama2 to validate everything E2E). Will be working on it the next couple of days:

I added a file in root of the project called wip_test_llama.py just to make it easier for you in case you want to check it. This file will be removed before the PR is merged.
The main difference from last time is that instead of using the code from the original repo I went with the code that they have in the nm-vllm repo. Three reasons for it: the code was easier to follow, had better APIs and their kernel implementation also supports fp8 making it fairly straightforward to add support for it in the future if we want to.
There are two parts of the code of which I am not confident if it is correct. To make it easier for you to look for them, I added a comment above the code section with the text NOTE(reviewers):

In here, I am not sure if I should transpose it and/or if it is even correct how I did it. It was done to match the in_features, out_features format that was in the original repo and because I was having some trouble with the shapes 😅
In here I am not sure if I am handling the reshape of the input tensor correctly.

Let me know if you find something that needs refactoring!
Aside from that, I have a question:

There is a section of code that is using numpy. I tried converting it to torch but it was throwing the error Promotion for uint16, uint32, uint64 types is not supported, attempted to promote UInt32 and Int. Is there a way for me to write this part of the code in native torch that would work? If not, then we will probably need to add numpy as a dependency to torchao before this gets merged. How should I proceed on this?

tagging for visibility: @msaroufim @jcaip

Thanks for the update @Diogo-V! I 'll take a more thorough pass through later tonight

Hi @jcaip, just saw that you did a commit getting torch.compile to work. Thank you!
Later tonight I will be working on getting the failing tests to pass. Is there anything else that you feel like is still missing/needs refactoring and I could take a stab at?

@Diogo-V Thanks for your help, and good job landing the op!
I took a look at your code and I do think there might be an issue in the way we hook up the marlin gemm op. I noticed for that changing the batch dimension (32, 16, 14096) for the input breaks the test, which shouldn't be the case. The compile test is also failing for me.
I don't know exactly what's where the problem is, but the general gist is that we have XW implemented for marlin op but actually for linear we calculate xW', which is why I think you need to transpose int_data and scales in from_float. Somewhere along the lines I think we're passing in the wrong dimensions, but need to do more debugging today.
Could you open a PR with just the marlin op (without the affine quantized tensor implementation) for now? You can git squash all your commits and then remove the AQT files with: https://stackoverflow.com/questions/12481639/remove-file-from-latest-commit. We can merge that in first, since I'll be on PTO next week. cc @jerryzh168 @msaroufim
I think the compile debugging will be a bit more involved, so don't want you to get blocked on that. Will update if I find the bug.

Yeah totally! Will open the PR in a bit
I am not super acquainted with the compile process (have been learning it the past few weeks). Do you have any resources that would be good for me to take a look at so that I can help out more and get across this issues in future PRs?
This weekend, I will take another stab at this and try to get the tests to pass.

Hey @Diogo-V

I am not super acquainted with the compile process (have been learning it the past few weeks). Do you have any resources that would be good for me to take a look at so that I can help out more and get across this issues in future PRs?

https://pytorch.org/tutorials/intermediate/torch_compile_tutorial.html is a good tutorial. I would actually recommend not debugging with compile, until you first fix the batch size=32 eager test case, since it'll be a lot easier to be able to debug in eager.

This weekend, I will take another stab at this and try to get the tests to pass. I would also be happy to pass this work off to someone else if you feel like it would be too tight of a schedule for the poster.

Thanks, that would be awesome! It would definitely help to have another person debug this. Let me try to explain a bit more about what I think is going on here:
On CUDA, tensors are represented as just blobs of data, and we keep track of the strides / offsets / dimensions of the tensor separately in metadata. This is nice because we can do something like a transpose without needing to modify our existing blog of data, we just change how we access it via the strides / offsets.
The Marlin kernel also takes in these m.k.n values, likely to make the kernel implementation easier. I think somewhere along the line we are flubbing a transpose somewhere, where we are switching up our mkn dimensions (because transpose), but still passing in the contiguous data of the non-transposed tensor. I think this mismatch is the cause of our error.
This is especially compounded because for 2:4 spares support, the tensor cores only support the first element being sparse really, which we can deal with by using transpose properties to move the second element sparse.

But note how we are now returning a transposed matrix, which means it is not contiguous, hence why it is in column major format. I think this row/column major format mismatch, or something related is the cause of our compile / batch_size=n bug. To make sure, we can check that both the marlin output and the reference output share the same strides in our unit test:  assert sparse_result.stride() == reference_result.stride()
Just a tip, feel free to do whatever you feel is best: I would gradually work up from your test_marlin_24 test here, by writing a nn.Linear layer that outputs the same values as your reference mm implementation.
Basically my thinking is that we go from mm unit test -> nn.Linear unit test -> e2e test. Let me know if you have any questions, I'll try to answer them the best I can.
As far as deadlines - just FYI: 9/6 is the deadline for our torchAO announcement, with the 9/16 is the deadline for making updates to our PTC poster.

https://pytorch.org/tutorials/intermediate/torch_compile_tutorial.html is a good tutorial. I would actually recommend not debugging with compile, until you first fix the batch size=32 eager test case, since it'll be a lot easier to be able to debug in eager.

Thank you very much! I will start off by tackling the batch size issue before moving onto the compile test as you suggested.

On CUDA, tensors are represented as just blobs of data (...) unit test: assert sparse_result.stride() == reference_result.stride()

Yep, I agree with you on this. Thanks for sharing your insights about the current problem! I think I have a better idea of how to debug this after reading your explanation 🙂

Just a tip, feel free to do whatever you feel is best: I would gradually work up from your test_marlin_24 test here, by writing a nn.Linear layer that outputs the same values as your reference mm implementation.
Basically my thinking is that we go from mm unit test -> nn.Linear unit test -> e2e test. Let me know if you have any questions, I'll try to answer them the best I can.

I think that suggestion is pretty good. I will follow that flow to debug this issue.

As far as deadlines - just FYI: 9/6 is the deadline for our torchAO announcement, with the 9/16 is the deadline for making updates to our PTC poster.

Thanks for sharing the deadlines. I was getting a bit concerned because I know that this is a time sensitive task and I don't want to risk missing an important date. I will try to surface updates and blocks as often as I can.

Hi, this last couple of days I have been trying some things and wanted to give an update / surface some of the findings to help anyone else that might be also tackling this:
Update
I was able to fix a bug in how the batch_size was being handled and I seem to have been able to replicate the issue that we are facing on a smaller example. Just for debug purposes, I added a test test_marlin_24_debug in test_ops.py file. It has notes explaining which section on the larger part of the code it is trying to simulate.
Observation 1
I went back to the Sparse Marlin repo to find out how they handle stuff in a Linear layer.
In their tests, they create a tensor w with shapes (21504, 4096) and then transpose it into (4096, 21504). In the next couple of lines, they create a Linear layer and then overwrite it's weight just before packing:
w = torch.randn((21504, 4096))
w = w.t()  # Is now (4096, 21504)
...
linear = nn.Linear(4096, 21504)  # Has weight tensor of shape (21504, 4096)
linear.weight.data = w  # Is now of shape (4096, 21504), the equivalent of transposing the linear's weight
...
# q has shape (128, 43008), where 128 = 4096 / 2 / 16 and 43008 = 21504 * 16 / 8
q = pack(linear, scales)

After packing, the shapes are a transformation of the original transposed tensor. This seems to indicate that the transpose inside the from_plain method might be necessary.
Observation 2
In the Sparse Marlin repo, I also tried to get the output metadata and, even though, in their documentation, it is mentioned that the output is in column-major format, the strides are (21504, 1) which seems to indicate that it is in row-major format and when calling is_contiguous() on the output tensor, the result is also True. Is there an instance that I am not aware of where it is possible for a tensor to be in column-major format and have the above mentioned strides?
Observation 3
Still in the Sparse Marlin repo, I adapted their replace_linear function to replace the linear layers with the implementation of Layer_2_4 from their repo as well as perform all the changes and computations that they have in their test examples to get it to work. I used a TinyLlama model and the results were not great:

I made sure to check that the weights from my adapted code matched and generated the same weights from the tests.
Considerations
I am still not sure where we might be failing but I am going to continue on this. Does anyone have any more insights or ideas as to what might be causing this issue?

@Diogo-V thanks for the update, I feel the suggested order of mm unit test -> nn.Linear unit test -> e2e test. makes sense, I'm wondering if you can run https://github.com/neuralmagic/nm-vllm/blob/9daca33a6fdc429802f448e1ea71630c996c9740/vllm/model_executor/layers/quantization/marlin.py#L232 and print the intermediate results and compare with the implementation in torchao to narrow down the problem?

@Diogo-V thanks for the update, I feel the suggested order of mm unit test -> nn.Linear unit test -> e2e test. makes sense, I'm wondering if you can run https://github.com/neuralmagic/nm-vllm/blob/9daca33a6fdc429802f448e1ea71630c996c9740/vllm/model_executor/layers/quantization/marlin.py#L232 and print the intermediate results and compare with the implementation in torchao to narrow down the problem?

Yeah absolutely! That seems a good thing to try. Will let you know of the results once I have them

Hi @jerryzh168, so I believe I have been able to fix the problem that we were having in the code and now the model outputs reasonable responses and seems to be correct:

After the previous debug that I did, I was able to narrow down the issue to the function inject_24. The bug was happening because applying that function to a transposed matrix and then transposing the output, does not yield the same results as applying the function to a non-transposed matrix:
x = torch.randn(128, 128)
y = copy.deepcopy(x)

x_24 = inject_24(x, *x.shape)

y_t = y.t()
y_t_24 = inject_24(y_t, *y_t.shape)
y_24 = y_t_24.t()

assert torch.equal(x_24, y_24)  # This will raise an expection

With this, we might just be missing getting the torch.compile to work without using the @torch._dynamo.disable decorators. I will take a stab at it during the weekend.
Let me know if think there is anything that might need to be refactored. I would be more than happy to do so.

That's great @Diogo-V, glad to see the progress here, only thing is I'm thinking of refactoring some of the dispatch code in affine quantized tensor to allow people to define their layout and quantized linear dispatch in separate files for better code structure, but it's not ready yet, will let you know when it's ready.

@Diogo-V Nice catch! That definitely seems like the source of the issue. I've been on PTO the last week, but getting back into this now.
cc @jerryzh168 do you know when you expect that to be ready by? Just asking because I have some BSR + int8 work that I want to land and not sure if I should hold off on that as well. Thanks for your help guiding diogo too while I've been out.

@jcaip yeah no problem, the refactor PR is here: https://github.com/pytorch/ao/pull/783 feel free to stamp

@jcaip - Awesome, glad you are back and hope you had a great PTO :)
To give a bit on an update on this: I had a busier weekend than I was expecting but still managed to take a look at what is currently happening. When running the example that compiles the forward pass from the file wip_test_llama2.py, an exception was raised on this line with the message saying that the weight_tensor was a FakeTensor and did not have the property layout_type.
That was as far as I got but I am planning on taking another look at it tomorrow evening.
Let me know if you have any insights into what might be happening here.

@Diogo-V the error might be related to version of pytorch, can you try with pytorch nightly version? you can first pip uninstall torch and then reinstall with https://pytorch.org/get-started/locally/

@jerryzh168 - After some debugging, I was able to fix most of the issues that were happening when torch.compile was used. There seems to be just one last problem that, once fixed, will let us run using torch.compile.
The issue is on this line and the raised exception is RuntimeError: .numpy() is not supported for tensor subclasses.. This seems to be happening because a FakeTensor is being passed here and it does not support numpy.
However, a similar action is also done in this line but no errors are thrown which left me a bit confused.
So, I have the below questions:

Is there a function and/or approach that I need to use to get dynamo to work with this section of numpy code?
Do you have any insights/tips on how to approach this?

@jerryzh168 - After some debugging, I was able to fix most of the issues that were happening when torch.compile was used. There seems to be just one last problem that, once fixed, will let us run using torch.compile.
The issue is on this line and the raised exception is RuntimeError: .numpy() is not supported for tensor subclasses.. This seems to be happening because a FakeTensor is being passed here and it does not support numpy.
However, a similar action is also done in this line but no errors are thrown which left me a bit confused.
So, I have the below questions:

Is there a function and/or approach that I need to use to get dynamo to work with this section of numpy code?
Do you have any insights/tips on how to approach this?

this is an error on tensor subclasses, I'm wondering why q_packed.cpu() is a tensor subclass? if it's because of FakeTensor I feel we just have to workaround this by not using numpy there. I'd suggest to rewrite _from_marlin_weights with native pytorch APIs instead of using numpy.

Is there a function and/or approach that I need to use to get dynamo to work with this section of numpy code?

it seems numpy with pytorch native tensor works: https://pytorch.org/blog/compiling-numpy-code/ the problem is just tensor subclass + numpy

@jerryzh168 - I was able to solve the issue and so, I think the PR is done now. Will mark it for review.

I had to add the below decorator to the test test_quant_sparse_marlin_layout_compile
@pytest.mark.skipif(not TORCH_VERSION_AT_LEAST_2_5, reason="Needs PyTorch 2.5")

because it was raising this exception (you can observe it on the CI):

I had previous gotten around this issue by installing the nightly version as you had suggested. Is this the correct thing to do or should I try something else?

@jerryzh168 - Just addressed your review and I have a question:
You suggested adding a test in the file test_sparse_api.py that would be similar to those there. Wouldn't that be repeating the tests implemented in test_marlin.py?
Was this the kind of test that you were expecting to add on that file or did you have something else on your mind?
    def test_sparse_marlin(self):
        input = torch.rand((256, 256)).half().cuda()
        model = (
            nn.Sequential(
                nn.Linear(256, 1024),
                nn.Linear(1024, 256),
            )
            .half()
            .cuda()
        )

        apply_fake_sparsity(model)
        model_copy = copy.deepcopy(model)

        # Quantized
        quantize_(model_copy.bfloat16(), int4_weight_only())
        dense_result = model_copy(input.bfloat16()).half()

        # Sparse + quantized
        quantize_(model, int4_weight_only(layout_type=MarlinSparseLayoutType()))
        sparse_result = model(input)

        assert torch.allclose(dense_result, sparse_result, atol=3e-1), "Results are not close"

Let me know if there is anything else that needs to be updated!

I tried runnning this with SAM but was running into a CUDA graph error, however with the 24_sparse_hf_example, I'm seeing:
Tokens/second: 134.497 for 2:4 sparse marlin
Tokens/second: 111.182 for our int4 tinygemm version
Tokens/second: ~80 for 2:4 sparse
Tokens/second: ~75 for compile baseline

🚀 🚀 🚀  cc @Diogo-V
I'll write this up and update the README @jerryzh168 for BE day tomorrow. It would be best if we could hook up into @HDCharles LLaMa benchmarks for consistency.

I had previous gotten around this issue by installing the nightly version as you had suggested. Is this the correct thing to do or should I try something else?

only test this in nightly sounds good if that's the issue

@jerryzh168 - Just addressed your review and I have a question:
You suggested adding a test in the file test_sparse_api.py that would be similar to those there. Wouldn't that be repeating the tests implemented in test_marlin.py?
Was this the kind of test that you were expecting to add on that file or did you have something else on your mind?
    def test_sparse_marlin(self):
        input = torch.rand((256, 256)).half().cuda()
        model = (
            nn.Sequential(
                nn.Linear(256, 1024),
                nn.Linear(1024, 256),
            )
            .half()
            .cuda()
        )

        apply_fake_sparsity(model)
        model_copy = copy.deepcopy(model)

        # Quantized
        quantize_(model_copy.bfloat16(), int4_weight_only())
        dense_result = model_copy(input.bfloat16()).half()

        # Sparse + quantized
        quantize_(model, int4_weight_only(layout_type=MarlinSparseLayoutType()))
        sparse_result = model(input)

        assert torch.allclose(dense_result, sparse_result, atol=3e-1), "Results are not close"

Let me know if there is anything else that needs to be updated!

yeah looks good to me

@jerryzh168 - Should be good to go now :)

Perfect, let me add my benchmarking updates to this branch so it's easier for Andrew to cherry-pick the whole thing.
BTW I'm seeing 226 tok/s for LLama3 - a 25% increase from 180 tok/s for our tiny gemm int4 kernel.
The perplexity is bad, we'll have to use the neuralmagic checkpoint for a fair comparison, but the weights are in a different format than the LLama3 checkpoint and I don't want to block this on that.

@jcaip - Yeah, I also noticed that sometimes the model's outputs were not super great. My simple sanity check was comparing against a quantize_(fake_sparsity(model), int4_wo(layout=Marlin...)) version and they were equal most of the time.
Let me know if there is anything else that I might be of help to get this across today and feel free to ping me on discord (Diogo-V on #cuda-mode) if you run into any issues!

Yes, for context, I am seeing the same text generation output on the hf example with the neuralmagic 2:4 sparse model, when I run quantize_(model, int4_wo()) and quantize_(model, int4_wo(layout=SparseMarlinLayout())) so I don't think it's an implementation issue. I'm guessing we'll may need to also use GPTQ as well for the best accuracy numbers.
But I'm very pleased with the speedup you've achieved (+25%), thats a great result :)

I wouldn't mind taking a look at adding support for GPTQ with Sparse Marlin if you feel like it would be a good thing to work on afterwards. I would have some time for it after the 21st of September.


add a unit test in sds service that trigger racing condition and fix

https://github.com/istio/istio/issues/14714
without adding read-lock in receiveThread, the added unit test fail with below error
Write at 0x00c4205d0d70 by goroutine 52:
  istio.io/istio/security/pkg/nodeagent/sds.(*sdsservice).StreamSecrets()
      /usr/local/google/home/quanlin/go/src/istio.io/istio/security/pkg/nodeagent/sds/sdsservice.go:192 +0x147e
  istio.io/istio/vendor/github.com/envoyproxy/go-control-plane/envoy/service/discovery/v2._SecretDiscoveryService_StreamSecrets_Handler()
      /usr/local/google/home/quanlin/go/src/istio.io/istio/vendor/github.com/envoyproxy/go-control-plane/envoy/service/discovery/v2/sds.pb.go:175 +0xd2
  istio.io/istio/vendor/google.golang.org/grpc.(*Server).processStreamingRPC()
      /usr/local/google/home/quanlin/go/src/istio.io/istio/vendor/google.golang.org/grpc/server.go:1176 +0x157c
  istio.io/istio/vendor/google.golang.org/grpc.(*Server).handleStream()
      /usr/local/google/home/quanlin/go/src/istio.io/istio/vendor/google.golang.org/grpc/server.go:1256 +0x13f2
  istio.io/istio/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1()
      /usr/local/google/home/quanlin/go/src/istio.io/istio/vendor/google.golang.org/grpc/server.go:691 +0xac

Previous read at 0x00c4205d0d70 by goroutine 59:
  runtime.convT2Estring()
      /usr/local/go/src/runtime/iface.go:346 +0x0
  istio.io/istio/security/pkg/nodeagent/sds.receiveThread()
      /usr/local/google/home/quanlin/go/src/istio.io/istio/security/pkg/nodeagent/sds/sdsservice.go:499 +0x13a

Goroutine 52 (running) created at:
  istio.io/istio/vendor/google.golang.org/grpc.(*Server).serveStreams.func1()
      /usr/local/google/home/quanlin/go/src/istio.io/istio/vendor/google.golang.org/grpc/server.go:689 +0xb8
  istio.io/istio/vendor/google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
      /usr/local/google/home/quanlin/go/src/istio.io/istio/vendor/google.golang.org/grpc/internal/transport/http2_server.go:421 +0x14d5
  istio.io/istio/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
      /usr/local/google/home/quanlin/go/src/istio.io/istio/vendor/google.golang.org/grpc/internal/transport/http2_server.go:461 +0x81e
  istio.io/istio/vendor/google.golang.org/grpc.(*Server).serveStreams()
      /usr/local/google/home/quanlin/go/src/istio.io/istio/vendor/google.golang.org/grpc/server.go:687 +0x16e
  istio.io/istio/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1()
      /usr/local/google/home/quanlin/go/src/istio.io/istio/vendor/google.golang.org/grpc/server.go:649 +0x50

Goroutine 59 (running) created at:
  istio.io/istio/security/pkg/nodeagent/sds.(*sdsservice).StreamSecrets()
      /usr/local/google/home/quanlin/go/src/istio.io/istio/security/pkg/nodeagent/sds/sdsservice.go:154 +0x193
  istio.io/istio/vendor/github.com/envoyproxy/go-control-plane/envoy/service/discovery/v2._SecretDiscoveryService_StreamSecrets_Handler()
      /usr/local/google/home/quanlin/go/src/istio.io/istio/vendor/github.com/envoyproxy/go-control-plane/envoy/service/discovery/v2/sds.pb.go:175 +0xd2
  istio.io/istio/vendor/google.golang.org/grpc.(*Server).processStreamingRPC()
      /usr/local/google/home/quanlin/go/src/istio.io/istio/vendor/google.golang.org/grpc/server.go:1176 +0x157c
  istio.io/istio/vendor/google.golang.org/grpc.(*Server).handleStream()
      /usr/local/google/home/quanlin/go/src/istio.io/istio/vendor/google.golang.org/grpc/server.go:1256 +0x13f2
  istio.io/istio/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1()
      /usr/local/google/home/quanlin/go/src/istio.io/istio/vendor/google.golang.org/grpc/server.go:691 +0xac

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: quanjielin
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

security/OWNERS [quanjielin]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@quanjielin: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/e2e-bookInfoTests-v1alpha3.sh
91533744ddf1e79782b7767de5b021ae06a162bc
link
/test e2e-bookInfoTests-envoyv2-v1alpha3-master

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test e2e-bookInfoTests-envoyv2-v1alpha3-master

@quanjielin: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/e2e-bookInfoTests-v1alpha3.sh
91533744ddf1e79782b7767de5b021ae06a162bc
link
/test e2e-bookInfoTests-envoyv2-v1alpha3-master

prow/integ-security-k8s-presubmit-tests.sh
91533744ddf1e79782b7767de5b021ae06a162bc
link
/test integ-security-k8s-presubmit-tests-master

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: quanjielin, wattli
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

security/OWNERS [quanjielin,wattli]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

New changes are detected. LGTM label has been removed.

@quanjielin: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/integ-security-k8s-presubmit-tests.sh
91533744ddf1e79782b7767de5b021ae06a162bc
link
/test integ-security-k8s-presubmit-tests-master

prow/integ-telemetry-local-presubmit-tests.sh
8b654865d5692a6c026f911a551290b622a26533
link
/test integ-telemetry-local-presubmit-tests-master

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@quanjielin: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/integ-security-k8s-presubmit-tests.sh
91533744ddf1e79782b7767de5b021ae06a162bc
link
/test integ-security-k8s-presubmit-tests-master

prow/integ-telemetry-local-presubmit-tests.sh
8b654865d5692a6c026f911a551290b622a26533
link
/test integ-telemetry-local-presubmit-tests-master

prow/integ-mixer-k8s-presubmit-tests.sh
8b654865d5692a6c026f911a551290b622a26533
link
/test integ-mixer-k8s-presubmit-tests-master

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@quanjielin: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/integ-security-k8s-presubmit-tests.sh
91533744ddf1e79782b7767de5b021ae06a162bc
link
/test integ-security-k8s-presubmit-tests-master

prow/integ-telemetry-local-presubmit-tests.sh
8b654865d5692a6c026f911a551290b622a26533
link
/test integ-telemetry-local-presubmit-tests-master

prow/integ-mixer-k8s-presubmit-tests.sh
8b654865d5692a6c026f911a551290b622a26533
link
/test integ-mixer-k8s-presubmit-tests-master

prow/integ-framework-local-presubmit-tests.sh
8b654865d5692a6c026f911a551290b622a26533
link
/test integ-framework-local-presubmit-tests-master

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test integ-framework-local-presubmit-tests-master

@quanjielin: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/integ-security-k8s-presubmit-tests.sh
91533744ddf1e79782b7767de5b021ae06a162bc
link
/test integ-security-k8s-presubmit-tests-master

prow/integ-telemetry-local-presubmit-tests.sh
8b654865d5692a6c026f911a551290b622a26533
link
/test integ-telemetry-local-presubmit-tests-master

prow/integ-mixer-k8s-presubmit-tests.sh
8b654865d5692a6c026f911a551290b622a26533
link
/test integ-mixer-k8s-presubmit-tests-master

prow/integ-framework-local-presubmit-tests.sh
8b654865d5692a6c026f911a551290b622a26533
link
/test integ-framework-local-presubmit-tests-master

prow/integ-galley-k8s-presubmit-tests.sh
8b654865d5692a6c026f911a551290b622a26533
link
/test integ-galley-k8s-presubmit-tests-master

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test integ-galley-k8s-presubmit-tests-master


New event codes with sysmon 3.2 release 1/4/2016

Suggested update for eventcode.csv:
EventCode,EventDescription
1,"Process Create"
2,"File Create Time"
3,"Network Connect"
4,"Sysmon Start"
5,"Process Terminate"
6,"Driver Load"
7,"Image Load"
8,"Create Remote Thread"
9,"Raw Access Read"
255,"Error"

Thanks for reporting. Traveling the next few weeks but will have a look as soon as I can.

Updated and checked in to master. Version 3.2.1 of this TA is forthcoming.

Thanks!


Add a helper function to BenchmarkConfigSpec to redirect flags.

Make it easier to access. Remove the need for multiple lambdas.

LGTM


Record recordset updates

Error handling for recordset
Common navigation bar
Recordset login/logout
URL encoding/decoding fixes for record and recordset
Put URI encoding to common util file.

I'm testing this with the record-recordset branch you pushed for me in chaise, not the fork which is the source of this pull-request...  the specific tests I could think of seem to work now.


Added protocol-interactiveselenium.jar dependency.

Added LoginHandler22.class as example

where's the rest of the handlers?

@raghavbharadwaj

great work now we can incrementally update it. Thanks @raghavbharadwaj cc @sujen1412

Thanks Professor. I think MultiplePattern33.java (team 33) and LoginHandler36.java (team 36) are kind of well written and we could further refactor them. Can you take a look at them?

sure I will take a look @raghavbharadwaj


Draw chart and save it in jpg

Hi , Thanks for make this project .
How can i draw the all candlestick with Ichimoku or any indicator in jpg image ? this is possible with this library ? or i have to use another ? if another, pls share some think.
Best Regards.

This package does not do that


Is cert-manager with Let's Encrypt issuer affected by the TLS-ALPN-01 revocation?

Let's Encrypt 2022.01.25 Issue with TLS-ALPN-01 Validation Method,

All active certificates that were issued and validated with the TLS-ALPN-01 challenge before 00:48 UTC on 26 January 2022 when our fix was deployed are considered mis-issued. In compliance with the Let’s Encrypt CP, we have 5-days to revoke and will begin to revoke certificates at 16:00 UTC on 28 January 2022.

I don't understand how to check whether cert-manager with Let's Encrypt issuer certificates are affected or not.  Do the cert-manager devs or community know?

I found the 0.16 Release Notes mention,

Unsupported challenge types (TLS-ALPN-01, TLS-SNI-01, TLS-SNI-02 and others) are not generated by cert-manager any longer

ACME Orders and Challenges makes it look like only HTTP01 and DNS01 challenges are supported.

One can list their Challenges,
$ kubectl get challenges --context product360-prod --all-namespaces --output json | jq '.items[] | {"name": .metadata.name, "type": .spec.type}'
{
  "name": "example-com-glz7c-3107669515-2566867772",
  "type": "HTTP-01"
}

ACME Orders and Challenges makes it look like only HTTP01 and DNS01 challenges are supported.

Seconded. And just to make it 100% clear. The only way you can be affected is if you use TLS-ALPN-01 which is TLS-based (aka HTTPS typically port 443). This challenge is mostly used in cases where port 80 is inaccessible during the challenge.
The Let's encrypt docs has a description of the different challenge types, and the pro's and cons for TLS-ALPN-01 are pretty telling for why they only estimate that 1% of the issued certificates are affected:

Pros:

It works if port 80 is unavailable to you.
It can be performed purely at the TLS layer.

Cons:

It’s not supported by Apache, Nginx, or Certbot, and probably won’t be soon.
Like HTTP-01, if you have multiple servers they need to all answer with the same content.
This method cannot be used to validate wildcard domains.

I'll close this out since we're past the expiration.


Example, using table caption, table number & cross-ref?

This is a great filter!
It would be very helpful if you included in the README or elsewhere a complete quarto example, including a table caption, table number, and ability to cross-reference that works with quarto.
For example, a LaTeX table I'd like to include in a Quarto doc originally (in .Rnw) looked like below. I'm not sure how to use your extension for this in .qmd.
\begin{table}[htb]
\renewcommand{\arraystretch}{1.6}
\caption{Multivariate test statistics as functions of the eigenvalues $\lambda_i$ solving $\det{\mat{H} - \lambda \mat{E}}=0$
or eigenvalues $\rho_i$ solving  $\detbracket{\mat{H} - \rho (\mat{H}+\mat{E})}=0$.
}\label{tab:criteria}
\begin{center}
\begin{tabular}{|l|l|l|l|}
  \hline
  % after \\: \hline or \cline{col1-col2} \cline{col3-col4} ...
  Criterion & Formula &  Partial $\eta^2$   \\
  \hline
  Wilks's $\Lambda$ & $\Lambda = \prod^s_i \frac{1}{1+\lambda_i}$ &  $\eta^2 = 1-\Lambda^{1/s}$   \\
  Pillai trace & $V = \sum^s_i \frac{\lambda_i}{1+\lambda_i}$ &  $\eta^2 = \frac{V}{s} $   \\
  Hotelling-Lawley trace & $H = \sum^s_i \lambda_i$ & $\eta^2 = \frac{H}{H+s}$   \\
  Roy maximum root & $R = \lambda_1$  &  $ \eta^2 = \frac{\lambda_1}{1+\lambda_1}  \\
  \hline
\end{tabular}
\end{center}
\end{table}

The table must be valid LaTeX and must be parsable by pandoc. To check the latter you can use quarto pandoc --from=latex and pass the table as input. If you add --mathml to that command, then you'll also receive warnings about equations that pandoc had trouble parsing.
To fix the given case above, just add the missing $ after the last formula in the table. The \mat and \detbracket commands are not standard LaTeX, the definitions of those macros must be part of the table.

I tried to use this in my Quarto book document, but I get no output for the table.
I'm building an HTML version

Installed taleb/parse-latex using quarto
Put in my _quarto.ylm: a top-level global filter (as your example suggests)

filters:
  - parse-latex

Put the following in one chapter .qmd file

Does the `parse-latex` filter work?

```{=latex}
\begin{center}
\begin{tabular}{|l|l|l|l|}
  \hline
  % after \\: \hline or \cline{col1-col2} \cline{col3-col4} ...
  Criterion & Formula &  Partial $\eta^2$   \\
  \hline
  Wilks's $\Lambda$ & $\Lambda = \prod^s_i \frac{1}{1+\lambda_i}$ &  $\eta^2 = 1-\Lambda^{1/s}$   \\
  Pillai trace & $V = \sum^s_i \frac{\lambda_i}{1+\lambda_i}$ &  $\eta^2 = \frac{V}{s} $   \\
  Hotelling-Lawley trace & $H = \sum^s_i \lambda_i$ & $\eta^2 = \frac{H}{H+s}$   \\
  Roy maximum root & $R = \lambda_1$  &  $ \eta^2 = \frac{\lambda_1}{1+\lambda_1}  \\
  \hline
\end{tabular}
\end{center}
```

What I get in the HTML output is shown below. There is now output from parse-latex

Caption / label
Even if I can get this to work, it leaves me with the question of how to provide a table caption, label, and make it so I can refer to @tbl-stats in the text.
In quarto, I would just use a chunk like
```{r}
#| label: tbl-stats
#| tbl-cap: Multivariate tests statistics
 ...
```

I understand. The table must be fixed, as described in my previous comment. Most important is to add the missing $.

As for the remaining issues: I don't know, honestly. It would be a good addition for the docs, and I'd be most happy to merge a PR that adds this info.

that pesky $!  This now works:
Does the `parse-latex` filter work?

```{=latex}
\begin{center}
\begin{tabular}{|l|l|l|l|}
  \hline
  % after \\: \hline or \cline{col1-col2} \cline{col3-col4} ...
  Criterion & Formula &  Partial $\eta^2$   \\
  \hline
  Wilks's $\Lambda$ & $\Lambda = \prod^s_i \frac{1}{1+\lambda_i}$ &  $\eta^2 = 1-\Lambda^{1/s}$   \\
  Pillai trace & $V = \sum^s_i \frac{\lambda_i}{1+\lambda_i}$ &  $\eta^2 = \frac{V}{s} $   \\
  Hotelling-Lawley trace & $H = \sum^s_i \lambda_i$ & $\eta^2 = \frac{H}{H+s}$   \\
  Roy maximum root & $R = \lambda_1$  &  $ \eta^2 = \frac{\lambda_1}{1+\lambda_1}$  \\
  \hline
\end{tabular}
\end{center}
```

And produces what I want:

** Caption / label**
However, I still can't figure out how to do this because it is a Latex, not an R chunk, so I can't use the Quarto comments like
#' label: tab-stats
#' tbl-caption: Test statistics for multivariate tests
#' ...

It's your extension, so I thought you would have an idea how to do that.

I started this quarto-dev discussion:
https://github.com/quarto-dev/quarto-cli/discussions/10432

OK: Here's the answer. I hope you will use it in your README or other examples.
To use parse-latex in a Quarto document embed the {=latex} chunk in a ::: div
::: {#tbl-mstats}

```{=latex}
\begin{center}
\begin{tabular}{|l|l|l|l|}
  \hline
  % after \\: \hline or \cline{col1-col2} \cline{col3-col4} ...
  Criterion & Formula &  Partial $\eta^2$   \\
  \hline
  Wilks's $\Lambda$ & $\Lambda = \prod^s_i \frac{1}{1+\lambda_i}$ &  $\eta^2 = 1-\Lambda^{1/s}$   \\
  Pillai trace & $V = \sum^s_i \frac{\lambda_i}{1+\lambda_i}$ &  $\eta^2 = \frac{V}{s} $   \\
  Hotelling-Lawley trace & $H = \sum^s_i \lambda_i$ & $\eta^2 = \frac{H}{H+s}$   \\
  Roy maximum root & $R = \lambda_1$  &  $ \eta^2 = \frac{\lambda_1}{1+\lambda_1}$  \\
  \hline
\end{tabular}
\end{center}
```
Test statistics for multivariate tests combine the size of dimensions of $\mathbf{H}\mathbf{E}^{-1}$ into a single measure.
:::

Hi! I have a question on parsing tables, and the example above shows my issue, so I will hijack this issue if it's OK.
In @friendly example where a screenshot of the table is shown, there are three hline in the latex code, but only one horizontal line appears. In some cases, I will get horizontal lines between each row, which I think is more a question of the markdown "flavor". But what I'm guessing is that the filter simply ignore \hline when converting to markdown, is that correct?


Request: Save to Same Snapshot

Instead of saving many snapshots, generating confusing clutter, give option to save to same snapshot. Could be global, or just a different keystroke.
Auto-save would also be nice.

It does auto save, but it's denounced at 30 seconds


Running PySPLIT without HYSPLIT

Hi everyone,
I am new in using PySPLIT. I am wondering if it is possible to run PySPLIT without having the HYSPLIT installed. I would like to do a baktrajectory analysis over the big size data on a server and its not possible to get the HYSPLIT installed on it.
Thanks,
Hooman

Hi H0omaN, I would like to do the same thing as you, if you have a solution please let me know, thank you very much.

Hi H0omaN, I would like to do the same thing as you, if you have a solution please let me know, thank you very much.

I have successfully run it under linux.
Firstly, you should download the HYSPLIT FOR LINUX (https://www.ready.noaa.gov/ready2-bin/getlinuxtrial.pl).
Then, decompressing it.
At last, Installing the pysplit in a conda virtual environment(https://github.com/mscross/pysplit#installing-in-a-conda-virtual-environment).
You can find the hyts_std in /user/hysplit/exec/. load it
Hope this is helpful for all of you who are confused

@mscross It says in bulk_trajgen_example.py:

"""
Trajectory Generation
PySPLIT includes a method that calls HYSPLIT to generate trajectories
launched from a single location over as many months and years as desired in
a single comprehensive batch.

Requirements

HYSPLIT installation

Unlike the majority of PySPLIT, this feature requires a local installation
of HYSPLIT, which can be acquired from the NOAA ARL
here<http://ready.arl.noaa.gov/HYSPLIT.php>_.

The location of the `hyts_std` executable must be known.

Meteorology
~~~~~~~~~~~

Is there any way to run PySPLIT AND still be able to use the generate_bulktraj() function without a local HYSPLIT installation?
or is this something I can help you develop?

Is there any way to run PySPLIT AND still be able to use the generate_bulktraj() function without a local HYSPLIT installation?
or is this something I can help you develop?

Pysplit is just a wrapper for hysplit. Running pysplit without hysplit would mean re-inventing hysplit.


Allow spaces in tagged user's names

Codecov Report

Merging #67 (a47d0cc) into master (0afb360) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##            master       #67   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files           43        43
  Lines          933       933
  Branches       128       128
=========================================
  Hits           933       933

Impacted Files
Coverage Δ

services/bot/src/commands/admins/create.ts
100.00% <100.00%> (ø)

services/bot/src/commands/admins/remove.ts
100.00% <100.00%> (ø)

services/bot/src/commands/queue/addPerson.ts
100.00% <100.00%> (ø)

services/bot/src/commands/queue/removePerson.ts
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0afb360...a47d0cc. Read the comment docs.


Splitgate Unable To Load (Freezes On Launch)

Game
Name: Splitgate (Free on steam)
Settings: Default
Error when: On launch immediately the pop up freezes
System Info:
GTX 4090
7950x
OS: Windows 11
DXVK
Latest: 2.1
Logs:
PortalWars-Win64-Shipping_d3d9.log
PortalWars-Win64-Shipping_d3d11.log
PortalWars-Win64-Shipping_dxgi.log

I quickly launched this on Windows with AMD and dxvk 2.1 and didn't experience the same issue.
A few things to try to see if any of them help.

Try https://github.com/doitsujin/dxvk/wiki/Windows#nvidia-gpus
At Splitgate\PortalWars\Binaries\Win64 make a file called dxvk.conf and instert dxgi.nvapiHack = False
Make sure GPU driver is up to date.
Try latest master build https://github.com/doitsujin/dxvk/actions/runs/4626931879

I will try this and report back, thanks!
On Thu, Apr 6, 2023, 12:36 PM Blisto91 @.***> wrote:

I quickly launched this on Windows with AMD and dxvk 2.1 and didn't
experience the same issue.
A few things to try to see if any of them help.

Try https://github.com/doitsujin/dxvk/wiki/Windows#nvidia-gpus
At Splitgate\PortalWars\Binaries\Win64 make a file called dxvk.conf
and instert dxgi.nvapiHack = False
Make sure GPU driver is up to date.
Try latest master build
https://github.com/doitsujin/dxvk/actions/runs/4626931879

—
Reply to this email directly, view it on GitHub
https://github.com/doitsujin/dxvk/issues/3240#issuecomment-1499403749,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AL3XFJQOKNPTNIKXGBFPGSLW735JDANCNFSM6AAAAAAUTFIRQI
.
You are receiving this because you authored the thread.Message ID:
@.***>

@RitzDaCat Friendly ping

Same issue unfortunately. Let me know if there's any other configurations I
can test.
On Sat, May 6, 2023, 11:27 AM Blisto91 @.***> wrote:

@RitzDaCat https://github.com/RitzDaCat Friendly ping
—
Reply to this email directly, view it on GitHub
https://github.com/doitsujin/dxvk/issues/3240#issuecomment-1537176267,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AL3XFJTAL47FQ4GXPEROI7TXEZ3YNANCNFSM6AAAAAAUTFIRQI
.
You are receiving this because you were mentioned.Message ID:
@.***>

Howdy,
Was just able to login without a black screen, hoped in a custom and
appears to be working as intended!
On Tue, Sep 5, 2023 at 3:47 AM Blisto91 @.***> wrote:

@RitzDaCat https://github.com/RitzDaCat Could you retest this dxvk 2.3
and latest driver?
—
Reply to this email directly, view it on GitHub
https://github.com/doitsujin/dxvk/issues/3240#issuecomment-1706200796,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AL3XFJRIXXMFZQS2B7MWY5LXY3RK7ANCNFSM6AAAAAAUTFIRQI
.
You are receiving this because you were mentioned.Message ID:
@.***>


add fast_finish

Coverage remained the same at 93.951% when pulling 96fe5d7a2cbc45633ea9fff2fb1e83dcaf053400 on chylli:chylli/fast_finish into e1785b99106358945b7ba876893d227ae5ee37ee on binary-com:master.


query - neuraltranslation.py

I was trying to execute neural_translation.py but it is generating the below error

@ilblackdragon Is this an error on our end?

Yes....it is working in tensorflow 7

Closing due to inactivity. Feel free to open a new github issue if the problem still persists in recent versions. If you chose to do so, please include a small repro snippet. Thanks!


Screenshot of two emails

Please paste the output of the following command here: sudo edit-chroot -all

Please describe your issue: Screenshot of two emails.

If known, describe the steps to reproduce the issue:
At https://github.com/dnschneid/crouton/issues/5039 without e-mailing with the images, same issue observed.

At https://github.com/dnschneid/crouton/issues/5047, slideshow of two emails.
https://github.com/dnschneid/crouton/assets/154830140/dd85f770-cb25-4fdc-82df-9e0c7146e49b


Fixed issue with wrong active value for widths smalller than smallest breakpoint

Hey,
I found one issue with your library concerning screen widths below smallest breakpoint.
In situation when my smallest breakpoint was set to 480px:
$breakpoints: (phone: 480px, tablet: 768px, tablet-large: 906px, desktop: 1024px, desktop-large: 1200px);
if i resize the window to a width smaller than 480px from anything larger (let's say >=desktop)  the CSS attributes on document.body:after (default element) are being cleared. The JS library is still keeping the old copy of CSS attributes in the memory and reports that my current breakpoint is still 'desktop'.
Please have a look and let me know what you think.
Cheers,
Tomasz

Thanks! 🎉


Fix #50 #51 #52 print button works. print modal new layout. success notif shows on print

print button in search now shows print modal on click. Print modal has new layout with qr code and select all button. Created success notification that shows on print.

Unable to get Codacy to run due to overload. Merging for now


Add protection from invalid responses #42

Fixes #42 We're assuming data is always an array (because we're using filter and map). We should validate that assumption and if the response does not meet that expectation, throw a helpful error message.

My best guess to what's happening is SuaceLabs is rate limiting this api (429 Too Many Requests)

In this case, we could retry, or we might want to log the status code. Thoughts?

the rate limit could be solved by this PR https://github.com/TestArmada/guacamole/pull/41, and it's out in<EMAIL_ADDRESS>referred in https://github.com/TestArmada/guacamole/issues/40.

@archlichking Mind merging and publishing when you have a chance? I don't have rights.


feat(Weaviate): Enforce capitalized index name

Add get_valid_index_name method to ensure index name is capitalized
Update index_name input with info about capitalization requirement
Implement capitalization check before creating vector store

Ref: https://forum.weaviate.io/t/feature-or-bug-first-letter-of-class-name-gets-capitalized/1203

closes #3742


Rules not being matched before terminals

Grammar using LALR
?start: ANY_STRING "and" ANY_STRING
         | ANY_STRING

    ANY_STRING: /.+/

How can it be ensured that the pattern ANY_STRING "and" ANY_STRING is matched to form a tree, instead of just parsing to a simple string.
For example parsing "s and s" through this grammar simply returns the original input "s and s" but the following tree is the intended output
start
    s
    s

Is this the correct behaviour?

Solved using common.ESCAPED_STRING


Insert Timestamp Prompt Format

"Insert > Insert Timestamp" prompts for format if default is empty.
Moved LineEntryDialog to PluginCore.Control.
Closes #929.

This PR is currently under a discussion in the related issues thread.


blocking assignment must be used for a clock divider

Let me propose this again.  It gets longer, but I hope this let you understand.

We have the following rule;

Do not use #delay in synthesizable design modules.

This rule does not work with a divided clock.
The following code demonstrates the issue.
module tb;
    logic clk = 0;
    always
        #5ns clk = ~clk;

    logic rst_n;
    logic [3:0] d;
    always_ff @(posedge clk, negedge rst_n)
        if (~rst_n)
            d <= '0;
        else
            d <= 4'(d + 4'h1);

    // clock with nonblocking assignment
    logic clk_div_nb;
    always_ff @(posedge clk, negedge rst_n)
        if (~rst_n)
            clk_div_nb <= '0;
        else
            clk_div_nb <= ~clk_div_nb;

    logic [3:0] q_nb_ng;
    always_ff @(posedge clk_div_nb)
        q_nb_ng <= d;   // not work by race condition

    // clock with blocking assignment
    logic clk_div_b;
    always_ff @(posedge clk, negedge rst_n)
        if (~rst_n)
            clk_div_b = '0;
        else
            clk_div_b = ~clk_div_b;

    logic [3:0] q_b_ok;
    always_ff @(posedge clk_div_b)
        q_b_ok  <= d;   // works

    initial begin
        rst_n = 1'b0;
        #10ns
        rst_n = 1'b1;
        // $dumpfile("div_clock.vcd");
        // $dumpvars;
        #100ns
        $stop;
    end
endmodule

q_nb_ng and q_b_ok are D-Flip-flops clocked by divided clocks.  They capture the output value of d.

d is clocked by clk.
q_nb_ng is clocked by clk_div_nb which is divided by using non-blocking assignment (conforming our current rule).
q_b_ok is clocked by clk_div_b which is divided by using blocking assignment (as I am proposing this pull-request).

The following is the simulation result of the sample code above.

q_b_ok works as we expect, but q_nb_ok does not.  For example at the timing on the yellow line they have to capture value 0b0010 as q_b_ok does.
The following shows how they works on the yellow line.

previous time slot
1st active region
1st NBA region
2nd active region
2nd NBA region

clk
L
L->H
H
H
H

d
0010
0010
0010->0011
0011
0011

clk_div_nb
L
L
L->H
H
H

q_nb_ng
0001
0001
0001
0001
0001->0011

clk_div_b
L
L->H
H
H
H

q_b_ok
0000
0000
0000->0010
0010
0010

clk_div_nb (non-blocking) is asserted at the first NBA region.
The state goes back to the second active region and at the 2nd NBA region q_nb_ng captures value (0b0011) of d updated at the first NBA region of this time slot.

This is not the behavior expected.  (cf. LRM 2017 Figure 4-1—Event scheduling regions)

Plan B: Use #delay for all FFs (when we have a divided clock)
Many engineers are choosing this solution.  But we know this degrades simulation performance.
Plan C: Use #delay for FFs clocked by divided clock and capture signals derived from FFs clocked by the original clock.
This may be better performance than plan B, but very easy to cause bugs.

I should create a branch after fetching updated upstream.  Sorry.

I tried to send the pull request of the fix on https://github.com/lowRISC/style-guides/pull/44/commits/957948ec27a580aa4f12a371c8dd6291e8b732ac.
But something wrong... Sorry.

I was pointed to this conversation by a colleague. My thinking about this is along these lines:
The general rules (for flop-based designs) are:
D1. State element updates are edge triggered on a clock and assigns must be NB
D2. Combinational logic are always blocking.
D3. Do not use delays (recommendation for cleanliness and avoidance of simulation-synthesis mismatch)
With these rules in place, I believe you should create all your clocks using blocking assigns.
This forces all clock events into the active region and all state updates into the NB region, guaranteed to be after all clocks update.
I believe this is what you're proposing, i.e. to add a style guide statement that all clocks should be generated using blocking assigns. I think that perhaps the wording needs to be clearer and the reason made explicit.

@rich-ho, thank you for you comment.

I believe this is what you're proposing,

Yes.

In my proposal always_ff for a clock divider is an exception to non-blocking assign.
In your proposal always_ff for a clock divider is an exception to "State element".

In either case we need an example code of clock divider for readers.

I'm sorry I left this PR for a long time.
I've push update, but something wrong.

This branch cannot be rebased due to conflicts
Only those with write access to this repository can merge pull requests.

I did "git rebase". Is it the cause?
Sorry for my mess.

I'm sorry I left this PR for a long time.
I've push update, but something wrong.

This branch cannot be rebased due to conflicts
Only those with write access to this repository can merge pull requests.

I did "git rebase". Is it the cause?
Sorry for my mess.

Ok, I think I've fixed the PR by removing the merge commits. @hirooih do you want to go through and fix the merge conflicts?

@marnovandermaas
Thanks.
It's been long time since I sent this PR.  Give me some time to recall.

Over the past four years, I have gained experience with git. Now I could make it a clean PR.

I moved this topic to the end of the section since it is not the main topic of the section.
I added an example of generating a normal clock in addition to the example of the divided clock.

Don't ask me why non-blocking assignment is used in clk <= 1'b0;.  I've been using this style from the example in RTL Modeling with SystemVerilog For Simulation and Synthesis: Using SystemVerilog for ASIC and FPGA Design.

We have the following rule;

Do not use #delay in synthesizable design modules.

This rule does not work with a divided clock.

To be clear (since this seems to be accidental misdirection), the rule is about synthesizable design modules, and it will not be changed. Delay statements only apply to simulation models, which is what you investigated and provided guidance for.

@a-will
Synthesizable design module often includes clock dividers.
But the following code does not work.
    // clock with nonblocking assignment
    logic clk_div_nb;
    always_ff @(posedge clk, negedge rst_n)
        if (~rst_n)
            clk_div_nb <= '0;
        else
            clk_div_nb <= ~clk_div_nb;

We can solve this issue by using #delay only for this case.  (The Plan C in my first comment)
This PR proposes  to use blocking assignment only for clock generation.
    // clock with blocking assignment
    logic clk_div_b;
    always_ff @(posedge clk, negedge rst_n)
        if (~rst_n)
            clk_div_b = '0;
        else
            clk_div_b = ~clk_div_b;

The following timing chart using Wavedrom would be easier to understand the problem.

I've added the following comments by reading your feedback. Thank you.
// only for test bench code
...
// for both synthesizable and test bench code


[WIP] Update Advanced Topics Doc

Description of the change:
Update info in advanced topics doc.
Motivation for the change:

@theishshah Can you also remove the TODO's from this doc as you address them in this PR.
e.g the doc still has my note:
// TODO: Update the main.go code in these sections to use the init() func to register the scheme instead of doing it in main().

@theishshah progress on this? I'd like to get this in before defaulting to the new project layout.


feat(injector): add component to shouldInclude

Edit: ~I think I have a better idea that solves the same issue: Let's include the filename where the prop/component symnol is declared. That way we can leverage includeUnusedProps and filter by filename in shouldInclude. In the end we'll probably filter out any prop type that's not declared in the same file as the component.~ It's still needed because includeUnusedProps causes usedProps to be empty.
Two goals are achieved with this change:

includeUnusedProps: true + shouldInclude as a ignore list doesn't scale very well if you extend very large prop types (e.g. HTML Attributes).
It seemed like it bailed out early if includeUnusedProps was true even though we might have included it later via shouldInclude
we want to move from custom @document directives to internal allow lists. Having magic markers in the jsdoc means that library users also see these magic markers.

I've separated this PR into test and implementation in case there's a better implementation (or even existing options) to achieve the same goal.

Thanks, the failing build seems to be yarn generating different hashes on Windows and Linux


yt-dlp does not allow to delete a cancelled file from the disk

Checklist

[X] I'm asking a question and not reporting a bug or requesting a feature
[X] I've looked through the README
[X] I've verified that I'm running yt-dlp version 2022.06.29 (update instructions) or later (specify commit)
[X] I've searched the bugtracker for similar questions including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue

Please make sure the question is worded well enough to be understood
I run a python file that downloads videos from YouTube. When I call DownloadCancelled, yt-dlp in progress_hooks it stops downloading, but won't let me remove the .part file from disk until I close the python script. Is there any way to delete a .part file without stopping the python script? (For example close the yt-dlp instance or something like that
Provide verbose output that clearly demonstrates the problem

[ ] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[ ] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output
No response

Sounds like a bug (either in yt-dlp or in your code). Provide a minimum reproducible example and it's verbose log

Sounds like a bug (either in yt-dlp or in your code). Provide a minimum reproducible example and it's verbose log

[debug] Encodings: locale cp1251, fs utf-8, pref cp1251, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.06.29 [9d339c4]
[debug] Python 3.9.13 (CPython 64bit) - Windows-10-10.0.19044-SP0
[debug] Checking exe version: "./api/ffmpeg" -bsfs
[debug] Checking exe version: "./api/ffprobe" -bsfs
[debug] exe versions: ffmpeg n5.0.1-5-g240d82f26e-20220614 (setts), ffprobe n5.0.1-5-g240d82f26e-20220614
[debug] Optional libraries: Cryptodome-3.15.0, brotli-1.0.9, certifi-2022.06.15, mutagen-1.45.1, sqlite3-2.6.0, websockets-10.3
[debug] Proxy map: {}
[debug] [youtube] Extracting URL: https://www.youtube.com/watch?v=1D5cxbDbKMc
[youtube] 1D5cxbDbKMc: Downloading webpage
[youtube] 1D5cxbDbKMc: Downloading android player API JSON
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, codec:vp9.2, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), acodec, lang, proto, filesize, fs_approx, tbr, vbr, abr, asr, vext, aext, hasaud, id
[info] 1D5cxbDbKMc: Downloading 1 format(s): 140
[debug] Invoking http downloader on "https://rr6---sn-01oxu-u5nd.googlevideo.com/videoplayback?expire=1657498832&ei=cBjLYsTTIMWF0u8P3_GdwAk&ip=<IP_ADDRESS>&id=o-AHvpD2_x4HZ9Jph80QNG-3gs_gv_NeiTGoURGNk3DlUv&itag=140&source=youtube&requiressl=yes&mh=wU&mm=31%2C29&mn=sn-01oxu-u5nd%2Csn-n8v7kn7z&ms=au%2Crdu&mv=m&mvi=6&pl=23&initcwndbps=402500&vprv=1&mime=audio%2Fmp4&gir=yes&clen=179601207&dur=11097.489&lmt=1641719888326317&mt=1657476784&fvip=5&keepalive=yes&fexp=24001373%2C24007246&c=ANDROID&txp=4531432&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIgGOdQJ8dPxe_eSm7f6gvcTeiZsxc-zFhz_m3rB6pc7psCIQCGfbCBKWPX5CvDMn1HOnDXrRKnuWj0dg8HI6_2EhgI_g%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRQIhAMu7NrDuXnx6J9ymIlQoqNP82n062QqDTwkDSmcuuXb1AiBJ1105HR_sPHYEg0t0IhtuWuKrDxgwbZdOtBvZUyuMvA%3D%3D"
[download] Resuming download at byte 68598612
[debug] File locking is not supported. Proceeding without locking
[download] Destination: temp\audios\Top 50 Most Popular Songs by NCS _ No Copyright.m4a
[download]  38.8% of 171.28MiB at  135.12KiB/s ETA 13:14[info] The download was cancelled

I don't see the example code, only logs

I don't see the example code, only logs

I don't see the example code, only logs

import os, yt_dlp

def progress_bar(data):
    if video not in db:
        # This is a db check. If video not in the database, I raise an exception
        raise DownloadCancelled

    print(data.get('_percent_str'))

video_title = 'Egzod & Maestro Chives - Royalty'

yt_dlp.YoutubeDl(options) as yt:
    yt.download_video(url, progress_hooks: [progress_bar])
    try:
        os.remove(video_title + '.mp4')
    except OSError as e:
        print(e)

The simpler the better 👍 Let me test it

The example is totally wrong!

But I was able to figure out what you are asking

The example is totally wrong!

I apologize that it is not a 100% finished sample. I didn't think you would run it, so I wrote it for show

But I was able to figure out what you are asking

Thank you. Most importantly, is this a bug?

Yes, yt-dlp isnt closing the file handle. Will fix asap

Yes, yt-dlp isnt closing the file handle. Will fix asap

I see, and where can I keep track of the fixes?

It's done

It's done

It's done

I updated the library, but the error did not disappear.

Either wait for next release, or update to master branch

https://github.com/yt-dlp/yt-dlp#with-pip

https://github.com/yt-dlp/yt-dlp#with-pip

Thank you so much for your promptness. Everything is running exelent.