[STOR-539]Allow to label consumergroup_lag and consumergroup_lag_sum metrics by…

… owner
Summary
This PR aims to allow clients to TAG consumergroup_lag and consumergroup_lag_sum metrics.

Improving the kafka_Exporter code
Adding "first" unit tests for the customLabels code (avoid keep adding tech debt)
Adding cache logic for keeping cosumergroups names--> owner tags in
memory
Removing Docker upload inherited from the original kafka_exporter repo

Limitations:

In order to do so CONSUMERGROUP_LAG_CUSTOM_LABELS environment variable must exist.
The format expected is a dictionary , if the json schema is incorrect the owner tagging will be skipped (warning message on starting)
If repeated "startWith" are present with different owners the first match will be the chosen one.
This happens because we want to be efficient in the hash lookup so we create a hash with all the startWith as key and the owner as values.

Test:
I ran it in a DEV cluster with real metrics, we can see in the logs the correct load when the ENV is set:
{"level":"info","msg":"Env CONSUMERGROUP_LAG_CUSTOM_LABELS is set, loading custom owner label for consumergroups","source":"kafka_exporter.go:726","time":"2020-11-24T07:18:26Z"}
We have set for start_with="p10n-" the owner:testp10 and for  start_with="related" the owner: testrelated
$ curl localhost:9308/metrics|grep lag|grep owner
kafka_consumergroup_lag_sum{consumergroup="related-items-loader-us-KApp",owner="testrelated",topic="recsys-related-items-adamic_adar-yapocl-v1"} 0
kafka_consumergroup_lag_sum{consumergroup="related-items-loader-us-KApp",owner="testrelated",topic="recsys-related-items-bm25-similarity-segundamanomx-v1"} 0
kafka_consumergroup_lag_sum{consumergroup="related-items-loader-us-KApp",owner="testrelated",topic="recsys-related-items-bm25-similarity-yapocl-v1"} 0
kafka_consumergroup_lag_sum{consumergroup="p10n-content-consumer-f3d18403-3524-408e-aca7-88f258f20b56",owner="testp10",topic="privacy-consent-subito"} 79204
kafka_consumergroup_lag_sum{consumergroup="p10n-content-consumer-fa397240-f05b-48cf-8e34-4c798547aae3",owner="testp10",topic="privacy-consent-subito"} 55084
kafka_consumergroup_lag_sum{consumergroup="p10n-content-consumer-fff84547-75a5-49e2-9d35-f7a009cc25b8",owner="testp10",topic="privacy-consent-subito"} 62386

@alxgarcia , @marcossantiago , @claudio-benfatto can you please review it? thanks!


Can a translation dataflow be integrated into the build process?

I would lice to be able to 'tx push' automatically when building the original language and 'tx pull' automatically when building the translation.
Is there a way to extend the builder script or do I need to create my own builder (or perhaps another hook to trigger these processes on build)?
Thanks
Chris

So you would like to execute a command when the build of your documentation on readthedocs.org starts?
I see the problem with this is that the builds will become non deterministic. What this means is that the result of the built might differ for the very same version depending on whether the built happened before or after some new translations where pushed to transifex.
Maybe could you write a bit more about how your current workflow for changes to the english version vs changes to the translations is?

Hello and thanks for responding!
Our workflow is:
Gitlab (english) -> RTD (english) -> Transifex (french) -> (tx pull, git add and push) -> RTD (english+french)
Gitlab... We upload the docs (would be nice if we had drag and drop but not there yet) and add a bit of mark-up for headings, etc.  Gitlab handles triggering a build using a webhook when something is committed.  We actually use both GitLab and Redmine (with the Redmine git hosting plugin) so I may change this step so that we manage the content within Redmine but this makes no difference to the workflow itself.
RTD... Used to display, search, bookmark, etc
Transifex...  to translate to other languages.  I've written a python script that creates .tx/config based on the .rst files that exist in a particular project.  After running the script a 'tx push' sends the files to Transifex.
RTD... back to RTD to view the translated content.
The bits that I still need to automate are the generation of the config file, pushing to Transifex and grabbing the translated docs from Transifex...  The steps are pretty straight forward... pull from transifex, add/push to the git repo and then trigger a RTD rebuild
If we added config options for target translations (of just used the languages of the linked projects then we can easily automate getting everything to Transifex by running the script to generate the .tx/config and then pushing the docs.  Perhaps using a new button in RTD to update/initiate Transifex translations by creating the cofig and pushing to Transifex.
Perhaps then we should add another button to retrieve translations from Transifex, that would do the 'tx pull', 'git add' and 'git push' to the source repo.  The rebuild could then be handled by a webhook in GitLab (or triggered directly after the translation updates have been pushed)
I think that you are right re getting this in the right order... The key will be finding just the right point to initiate the rebuild based on something becoming available (perhaps post review) within Transifex.  Perhaps the Github/Trtansifex hook can be modified for this to trigger an update through gitlab or RTD directly from Transifex.

Thanks for detailing the whole process.
I'm not convinced yet that readthedocs is the right place for it to happen. The purpose for readthedocs is to host your documentation, which includes the building of the docs as it is required to display them somehow. Also readthedocs can be used to host your translated documentation, however it doesn't impose any structure on you how you make the translation happen. Therefore I wouldn't like to see more automation for external services in the build process of readthedocs.
IMO a valid alternative would be a build server like what TravisCI is offering to implement this step. I think it's already possible to make TravisCI do exactly what you want. Probably the after_success documented here: http://docs.travis-ci.com/user/customizing-the-build/ would make a good point to start a script which handles your needs for tx pull and git push

@carukc What's your opinion on using Travis for this problem?

Hi,  Thank you for responding and apologies for not replying sooner myself.  TravisCI looks pretty nice.  I'm giving it a try now to see if I can integrate with GitLab too.  I still think that I need a way to manually initiate the pull/build process though (a button somewhere)... There will be lots of translation work being done and I would not want to initiate a rebuild every time someone translates some part of a document.
Transifex supports webhooks but the docs only say that it fires 'When the translations are done'.  I'm not sure whether this means when a whole doc has been translated or when a paragraph has been translated.
I'll keep you posted as I test all this out.

@carukc hi, is this still needed for you? Travis has a rebuild button (or another name, I don't remember)

I really would like to have transifex integration, see #3329 for more there, if we ever add this to our roadmap. Closing this due to lack of activity.


Allow further attach options for debugging.

Changed the attach script launching of the debug session, to use the attach configuration settings.
Allowing for more settings to be passed to the launch of the debug session.
Kept the added attach parameters to be true by default. So current usability isn't affected, but can be better modied from settings. Not many attach params but more can be added if needed.

Closes #23

Hi,
This is a great idea and thanks for taking time to look into a solution for this as well!
I just have a few notes:

The ue-python.debug.port config needs to be deprecated before it's removed. So users has a chance to update before it potentially breaks.
"showOutput" doesn't seem to be a valid python debug argument?
For the description of the justMyCode config, we can probably copy the offical docs: "Restricts debugging to user-written code only. Set to false to also enable debugging of standard library functions."
The return value of await utils.findFreePort(startPort, 101) needs to be assigned to attachConfig.port.

Otherwise nicely done, looks like a really clean solution!
Feel free to look at this if you want to, otherwise I can probably take a look at this at some point during this week.

Amended for the recommended changes.
Had to refactor slightly again due to typechecking not liking the if (!attachOptions) null check.
Another adjustment could possibly move the strictPort config option to attach as well, as don't think it is used anywhere else (correct me if I'm wrong) but is up to you. Would just require a seperation of the strictPort and other config options after loading.

All looks good, thank you for this contribution!
StrictPort is currently used in the modules/remote-handler.ts file as well. When deciding on which port to use for communication with Unreal.

I've currently credited you as "F-Dudley" in the CHANGELOG.md. But just let me know if you have another preferred name


Change the way the Doctrine tracing middleware is registered in DBAL >=3.x

Fixes #805. The problem that was reported is that the MiddlewarePass compiler pass in DoctrineBundle adds a new call to setMiddlewares(), which means that all the middlewares added before get lost. Technically speaking, if you load SentryBundle after DoctrineBundle, it works as expected, because our code appends our middleware to the existing list instead of replacing it entirely. However, as noted by @fridtjof, a better integration approach is to tag our middleware as doctrine.middleware and let the Doctrine bundle do all the rest of the work to properly configure the connection.

Do note that this raises the required doctrine-bundle version to 2.6 (as opposed to the current ^2.5 listed in require-dev), as their tagged middleware support was only introduced then: https://github.com/doctrine/DoctrineBundle/releases/tag/2.6.0, https://github.com/doctrine/DoctrineBundle/pull/1472

Do note that this raises the required doctrine-bundle version to 2.6

Nice catch. Unfortunately, I realized that these changes are not compatible with version 1.12 of the bundle, which we still support. It seems that we declare the compatibility with that version mainly because of Symfony 3.4, but since we nowadays require at least version 4.4 of the framework, I think we can just drop the support for the the older 1.x...

do we have any E2E test to check if all of this is working?

No, we don't. There are some E2E tests for other stuffs, but nothing specifically tailored at asserting that the tracing integrations work as expected. It's something we could definitely improve in the future though...


restore command is failed

Command failed: \extensions\patrickschiefer.al-nuget-helper-0.2.5\tools\windows\paket.exe install
Paket failed with
-> Error in paket.dependencies line 9
PackageName can't contain ':'
Genarated paket.dependencies:
strategy: min
lowest_matching: true
source https://dynamicssmb2.pkgs.visualstudio.com/DynamicsBCPublicFeeds/_packaging/MSSymbols/nuget/v3/index.json
source https://dynamicssmb2.pkgs.visualstudio.com/DynamicsBCPublicFeeds/_packaging/AppSourceSymbols/nuget/v3/index.json
nuget Microsoft.Application.symbols >= <IP_ADDRESS>
nuget Microsoft.Platform.symbols >= <IP_ADDRESS>
nuget Microsoft.SystemApplication.symbols.63ca2fa4-4f03-4f2b-a480-172fef340d3f >= <IP_ADDRESS>
nuget Microsoft.Exclude_APIV2.symbols.10cb69d9-bc8a-4d27-970a-9e110e9db2a5 >= <IP_ADDRESS>
nuget InsightWorks.DynamicShip:IntegratedShipping.symbols.9c55319d-f3b9-441b-9956-0a0a04decd4b >= <IP_ADDRESS>

Seems like the Normalize function does not remove ":". I will have a look at it.
Does your appname iclude ":"?

Rule for removing characters is defined as
removing chars other than „a-zA-Z0-9_-“,

if you want to be compatible with MS.

Does your appname iclude ":"?
Yes. The AppName is "Dynamic Ship: Integrated Shipping".
Unfortunately, this is not my app I just need to use it in my customization.

There was a problem in the normalization code, it should be solved in 0.2.6


 Next time beer is on me.

LGTM

landed as 18a2e4fbfc44216c31bbcdf7705ca87c53e6f1fa


Landing page

Spec
figma: https://www.figma.com/file/zpD45j8nqDB2XfA6m2QskO/Cofacts-website?node-id=2760%3A463

@MrOrz Both css and js have been revised, please check if there are any problems.


Is Marshmellow<3 required to function?

I've started messing around with spinning up BLESS internally, but after a few iterations I got stuck on the following error popping up when I ran the lambda:
{
  "errorMessage": "__init__() got an unexpected keyword argument 'strict'",
  "errorType": "TypeError",
  "stackTrace": [
    "  File \"/var/task/bless_lambda_user.py\", line 68, in lambda_handler_user\n    schema = BlessUserSchema(strict=True)\n"
  ]
}

Anyway after a bunch of random Google-fu and stumbling through code. I ran across this comment by Jeremy Stott:

So (like any trained security professional) I simply blindly followed suggestions from random internet comments to try and get what I hope to be a critical security control for my org to work. Low and behold it works! s/'marshmallow',/'marshmallow<3',/ got things back to a happy state. Here's my current setup.py
import os

from setuptools import setup, find_packages

ROOT = os.path.realpath(os.path.join(os.path.dirname(__file__)))

about = {}
with open(os.path.join(ROOT, "bless", "__about__.py")) as f:
    exec(f.read(), about)

setup(
    name=about["__title__"],
    version=about["__version__"],
    author=about["__author__"],
    author_email=about["__email__"],
    url=about["__uri__"],
    description=about["__summary__"],
    license=about["__license__"],
    packages=find_packages(exclude=["test*"]),
    install_requires=[
        'boto3',
        'cryptography',
        'ipaddress',
        'marshmallow<3',
        'kmsauth'
    ],
    extras_require={
        'tests': [
            'coverage',
            'flake8',
            'pyflakes',
            'pytest',
            'pytest-mock'
        ]
    }
)

I've just started to familiarize myself with Bless, so I feel like I'm pretty strongly lacking in context for the full implications of this change here. Is this workaround of pinning to an earlier version of marshmellow the correct way forward?

Ah, interesting. Looks like there's already a PR that would similarly solve my particular issue already in the works: https://github.com/Netflix/bless/pull/99/files

Same here, thanks for this, it works

@benjamin-hering I'm the guy that posed the question, glad that it helped.
And yes, This PR fixes completely the issue, if we could fast-track the approval it would be great, also because as it stands the Lambda is broken and will not work at all without that fix.

So (like any trained security professional) I simply blindly followed suggestions from random internet comments to try and get what I hope to be a critical security control for my org to work.

😂 haha so relatable.


Networking: Add support for IPv6 configuration

Currently our runtime lacks support for IPv6. We need to enhance the runtime, virtcontainers as well as our agent protocol and agent to support discovery and configuration of IPv6.

Kata issue is: https://github.com/kata-containers/runtime/issues/147.


pluck: fix signature

LGTM. not sure how we got it that wrong. probably because it was done in a wicked rush.


fix(highlights): Correct DapUINormalFloat to DapUIFloatNormal

It looks like in https://github.com/rcarriga/nvim-dap-ui/commit/d7a9f032e7e45b37d778ef83e3d412566ba97cb5, the use of DapUIFloatNormal was introduced, but two different names were used (DapUIFloatNormal and DapUINormalFloat).
This fix is correcting the names to both be DapUIFloatNormal as it seems to follow the general naming convention used

Thanks for the PR :smile:


add  cRonstrue, parses a cron expression and outputs a human readable…

… description of the cron schedule

hello @alseambusher

Thanks @hasanbasri1993. This is now merged.


chore(deps): update terraform terraform-ibm-modules/cos/ibm to v8.10.1

This PR contains the following updates:

Package
Type
Update
Change

terraform-ibm-modules/cos/ibm (source)
module
patch
8.10.0 -> 8.10.1

Release Notes

terraform-ibm-modules/terraform-ibm-cos (terraform-ibm-modules/cos/ibm)
v8.10.1
Compare Source
Bug Fixes

deps: update required_provider to latest for the deployable architecture solution (#690) (214b30b)

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

/run pipeline

:tada: This issue has been resolved in version 1.24.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Verify test package name when generating tests

I've noticed that when generating tests, they will generate as invalid if the *_test.go file uses the _test suffix in its package name (i.e.package *_test).

I believe the convention is to save the *_test suffix for tests only, so this is working as intended, unless you have a concrete example of when this functionality is needed.

I've heard it to referred to as external testing, but this page covers the functionality. I'm sure there's more info available

Test files that declare a package with the suffix "_test" will be compiled as a separate package, and then linked and run with the main test binary.

There is an issue on functionality documentation here

I see, this is used for test-only packages right? If you want to send a PR to fix this, I'll be happy to approve.

Yes it is, I'll see if I can come up with a simple way to solve this.


prop type touched is invalid; it must be a function, usually from React.PropTypes

"redux-form": "^6.0.0-rc.3"
Trying out the canonical text input example and I'm getting this error, I'm not sure what is going wrong here...
Component code:
import React, {PropTypes} from 'react';
import TextField from 'material-ui/TextField';

const renderTextField = ({ input: { label }, name, type, touched, error }) => (
  <TextField
    id={name}
    type={type}
    hintText={label}
    floatingLabelText={label}
    errorText={touched && error}
  />
);

renderTextField.propTypes = {
  error: PropTypes.string,
  input: PropTypes.object,
  name: PropTypes.string,
  touched: PropTypes.boolean,
  type: PropTypes.string,
};

export default renderTextField;

Same error occurs with a regular (non-material-ui) input.

touched was under input in 6RC3. I recommend upgrading to 6RC4, where it is now under meta.
e.g.
const renderTextField = ({ input, name, type, { meta: { touched, error } } }) =>


all dialogs throw error

Throws error on opening of any dialogs
Uncaught Error
function showOpenDialog(allowMultipleSelection, chooseDirectories, title, initialPath, fileTypes, callback) {
// FIXME
throw new Error();
}

Which branch are you using?  If you're on the bramble branch, this should work.

I'm going to close this, since we do this in bramble here: https://github.com/humphd/brackets/blob/bramble/src/filesystem/impls/filer/FilerFileSystem.js#L21-L27.  Feel free to file more bugs if you get stuck.


MudNumericField: Use native html input number type

Description
Render MudNumericField as <input type=number> instead of <input type=text>.

Pattern now defaults to null as opposed to [0-9,.\-]
Setting a Pattern, Format or Culture will render the input using type=text, allowing the desired format to be enforced.

Resolves #10191
How Has This Been Tested?
Existing unit tests as well as an additional test to confirm rendered html
Reviewed all examples on the doc page
Android Screenshot

iPhone Screenshot

Type of Changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Documentation (fix or improvement to the website or code docs)

Checklist

[x] The PR is submitted to the correct branch (dev).
[x] My code follows the code style of this project.
[x] I've added relevant tests.

Make sure you test in mobile too as I believe rendering on most mobile OS is different for type=number

Will try to get an iPhone test done later today

@Anu6is I just tried removing the pattern but still does not seem to work. I am not sure if your change resolves the issue. Take a look at my new comment: https://github.com/MudBlazor/MudBlazor/issues/10191#issuecomment-2459971059

@Anu6is I just tried removing the pattern but still does not seem to work. I am not sure if your change resolves the issue. Take a look at my new comment: #10191 (comment)

Based on your comment setting the input type to number works, which is what this change does.

This one is hard, as nobody from core members have iphone to test

@mikes-gh Has a Macbook

@mikes-gh Has a MacBook

Is it a Safari test you're looking for?
I always use Edge but I have Safari as part of the macOS ofc.

Resolves the linked issue (created by me). Also no issues on Android (Google Pixel 8a) so I would love to see this in a future release 😍

@henon does this PR also suffers from https://github.com/MudBlazor/MudBlazor/issues/10191#issuecomment-2468790508?

@ScarletKuro is this a valid use of parameter state? Especially as I didn't address the "bad" implementation in the base class

@ScarletKuro is this a valid use of parameter state? Especially as I didn't address the "bad" implementation in the base class

Technically, you can use it for the change handler. Practically, you would usually want to fully utilize ParameterState and remove the logic in the getter/setter, and write via SetValueAsync  and read from _cultureInfo.Value everywhere instead of the Parameter

@ScarletKuro Why is this not merged, yet. When will it be merged?

@ScarletKuro will you take a second look at this one or shall I merge it?

@ScarletKuro will you take a second look at this one or shall I merge it?

Well, the code looks okay, but I can't test it as I don't have an iPhone. I'm slightly not a big fan of using the parameter state just for _cultureHasValue = x.Value is not null. I wonder if this would be sufficient?"
protected override void OnParametersSet()
{
    base.OnParametersSet();
    _cultureHasValue = Culture is not null;
}


Added Saplings to age mech/tags

Saplings added to MaterialTag.age tags/mechs.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Restructure notification logic

Making all these separate notifications is not the most efficient. Many of them loop through summoners, with simple checks being done.
To me it would make sense to accumulate data, or create a different structure for adding embeds.
Now we do:

loop notification
loop summoners
check conditions
add build embed..

And since almost all embed context is the same (summoner, queuename...) we can add this in one template as well. So it'd be something like:

loop some notification type (e.g. summoners)
check condition of all attached/subscribed notifications
add build embed

We could then remove the need for creating a class for each new notification (if it is one of the "trivial" existing types).
So, instead of "registering" the team notification checkers:
https://github.com/freshcoders/discorki/blob/87656c209df6e9ec3668f5299cc70c51f3d78efd/src/main/java/com/alistats/discorki/tasks/CheckMatchFinishedTask.java#L31
We would just run some wrapper loops.

Ill update on this later, to see if it actually makes sense to make the effort


Implement _check

within calculator.js file:

[ x ] complete _check function definition
[ x ] refactor add, subtract, multiply, and divide function by calling the _check function
[ x ] all calculator.test.js tests are passing

Thanks for the help  and the great experience!


feat: implement "compress" and lzw writer

at this point the "compress" utility works as required(except for the force flag), integration test is missing, which needs to be added and few code cleanups i guess :)

@jgarzik now compress is ready to be used completely :)

This unfortunately adds several new compile warnings, which is unwanted.  posixutils should build warning-free.

i'm really sorry for that @jgarzik, will make sure such mistakes dont happen next time


decodeList discards error messages from list items

When decoding a list, if any of the items in the list fail to decode, then the failure message from Decoder. decodeList is always "[A]List[A]" and does not include any of the failure messages from the individual items.  This makes trouble shooting difficult.
The relevant line is:
https://github.com/circe/circe/blob/master/modules/core/shared/src/main/scala/io/circe/Decoder.scala#L664
The same applies to decodeSet and decodeVector.

Could you please provide an example, @davidbkemp? I'm curious of the code demonstrating the issue that you've raised here.

Actually, Travis helped me out.
You're talking about this scenario, @davidbkemp?
import io.circe._
import parser._
import syntax._

scala> val ints = parse("[1,2]")
ints: Either[io.circe.ParsingFailure,io.circe.Json] =
Right([
  1,
  2
])

scala> Decoder.decodeList[Int].decodeJson(ints.right.get)
res0: io.circe.Decoder.Result[List[Int]] = Right(List(1, 2))

scala> val strings = parse("[\"a\"]")
strings: Either[io.circe.ParsingFailure,io.circe.Json] =
Right([
  "a"
])

scala> Decoder.decodeList[Int].decodeJson(strings.right.get)
res2: io.circe.Decoder.Result[List[Int]] = Left(DecodingFailure([A]List[A], List(DownArray)))

Per this issue, you'd prefer a better DecodingFailure "message" (if that's the right word) than [A]List[A]?

Yes, it would be nice if the "message" included the decoding failure message for attempting to decode the string as an integer. The current message "[A]List[A]" doesn't tell you much, especially if the actual failure occurs deep inside the attempt to decode one of the A's.

@cponomaryov Thanks for fixing this.
Looking forward to a new release with this fix.

Resolved in #649.


readme, contribute guide?

icc/gcc/build info/### code static analysis### (https://scan.coverity.com/github), code guide, contribute guide, etc...?

README is in progress in pr-basic-readme branch.


Linting errors - 'Unexpected token' after updating @nuxt/eslint

@nuxt/eslint module version 0.4.0 produces weird errors when encountering type keyword:

Reverting back to 0.3.13. Opening this issue to track the problem.
Reported in project repo - https://github.com/nuxt/eslint/discussions/444

Reported in upstream repo: https://github.com/nuxt/eslint/issues/492

The issue was solved by adding typescript dependency into package.json


Set-AzsLocationInformation function missing in AzureStack.Infra.psm1

Set-AzsLocationInformation function missing in AzureStack.Infra.psm1 Copy the old function from older version does not work...

PR #276 is for this issue.

Closing since #276 is now merged


[Stress Testing] Deployment of stress tests -- file sizes too large

Error: create: failed to create: Secret "sh.helm.release.v1.stress-py-eh.v1" is invalid: data: Too long: must have at most 1048576 bytes
Without a .helmignore file, deploying stress tests will drag in all files that are on the same level as chart.yaml (i.e log files etc.).

@LibbaLawrence Aside from following the layout recommendations (https://github.com/Azure/azure-sdk-tools/blob/main/tools/stress-cluster/chaos/README.md#layout) I'm not if there's a super great generic solution for this. Some options:

We could try to create some sort of reverse helm ignore allowlist
We could do some pre-calculation in the deploy stress script that looks if extra files are being included that shouldn't be and warns you or gives you a prompt option to add them to the .helmignore file.

I think I like option 2 the most. Thoughts @richardpark-msft @ckairen?


Documentation for expression-bodied members in C# 7

Documentation for expression-bodied members in C# 7
Summary
Documents expression body definitions, and also mentions expression body definitions in each supported member topic that did not mention it.
Also revised the top-level constructor topic somewhat.
Fixes #1789
Suggested Reviewers
@BillWagner

Thanks, @mairaw. I've incorporated your corrections.

@BillWagner, I've addressed your comments and also changed destructor -> finalizer throughout the doc set.

Wow. Thanks for checking everywhere for the destructor -> finalizer change @rpetrusha

Thanks, @mairaw. I've changed "cs" to "csharp" and replaced "do not" with "don't". I'll merge when this publishes.


Mouse position as Vector2

Before, there were only two functions int get_mouse_x and int get_mouse_y but now you can do Vector2 get_mouse to get Vector2(get_mouse_x(), get_mouse_y())

Wow, very impressive. This could be used for many things, such as button clicking. Thanks.


sciond API: add path policy support to path requests

sciond path requests should have a path policy field that sciond considers.

We should probably close this due to age/lack of context. This will come up again eventually, but we can create a new issue then.
@lukedirtwalker: what do you think?

Yeah this is old and not clearly specified. So let's close and create a new issue once we have the details specified.


Proximity search

Define distance between two terms with AND

See https://platform.newseye.eu/en and syntax « "vote femme" ~10 »

also https://github.com/newseye for implementation details

This is actually Lucene standard syntax automatically supported by ES, magic! :-)
https://www.camille-ulb-kbr.be/?query="vote+femme"~10


How can we re-train the frozen generator on a custom dataset?

Hi!
Thanks for putting out the great work!
I am interested in training the frozen generator on a custom dataset. Could you please guide me, or kindly share the training code?
Would sincerely appreciate any help.
Thanks!

Hi,
Thank you for your interest in our work!
 This project is mostly focused on training without data, but if you tell me roughly how much data you have (10? 100? 5000 images?), maybe I could point you at projects that are a good fit?

Thanks for the prompt reply!
I have ~1200 images.

With 1200 you could try training a StyleGAN-ADA model. Just make sure you either train with the Tensorflow version of ADA or with the Rosinality pytorch implementation. The conversion script we have in this repo (and in the Colab notebook) doesn't really work with models from the official StyleGAN-ADA pytorch version. You might also have better luck if you fine tune from an existing model rather than train from scratch.
I think this youtube channel has some guides on training with new datasets: https://www.youtube.com/channel/UCaZuPdmZ380SFUMKHVsv_AA

Hi,
I trained StyleGAN-ADA on my dataset (resolution 256x256), using the Tensorflow version of ADA.
However, unfortunately, the convert_weight.py script is not working for me. I get the following error:
Traceback (most recent call last):
File "/content/gdrive/MyDrive/StyleGAN_NADA/stylegan_nada/convert_weight.py", line 247, in
state_dict = fill_statedict(state_dict, g_ema.vars, size, n_mlp)
File "/content/gdrive/MyDrive/StyleGAN_NADA/stylegan_nada/convert_weight.py", line 143, in fill_statedict
update(state_dict, convert_dense(vars, f"G_mapping/Dense{i}", f"style.{i + 1}"))
File "/content/gdrive/MyDrive/StyleGAN_NADA/stylegan_nada/convert_weight.py", line 100, in update
raise ValueError(f"Shape mismatch: {v.shape} vs {state_dict[k].shape}")
ValueError: Shape mismatch: torch.Size([512, 512]) vs torch.Size([256, 256])
Any clues for resolving this would be really appreciated. Thanks!

Hi,
Could you try running the convert script with --channel_multiplier 1? (instead of the default which is 2).

Thanks a lot for the help!
Setting --channel_multiplier 1 in the convert_weight.py script resolved the above error.
But now I get the following error when I try to train a text-guided StyleGAN-NADA model, using the converted weights as my frozen generator.
Loading base models...
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-4-d9c9711b741e> in <module>()
     69
     70 print("Loading base models...")
---> 71 net = ZSSGAN(args)
     72 print("Models loaded! Starting training...")
     73

2 frames
/content/gdrive/MyDrive/ML/Code/StyleGAN_NADA/stylegan_nada/ZSSGAN/model/ZSSGAN.py in __init__(self, args)
    154
    155         # Set up frozen (source) generator
--> 156         self.generator_frozen = SG2Generator(args.frozen_gen_ckpt, img_size=args.size).to(self.device)
    157         self.generator_frozen.freeze_layers()
    158         self.generator_frozen.eval()

/content/gdrive/MyDrive/ML/Code/StyleGAN_NADA/stylegan_nada/ZSSGAN/model/ZSSGAN.py in __init__(self, checkpoint_path, latent_size, map_layers, img_size, channel_multiplier, device)
     29         checkpoint = torch.load(checkpoint_path, map_location=device)
     30
---> 31         self.generator.load_state_dict(checkpoint["g_ema"], strict=True)
     32
     33         with torch.no_grad():

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in load_state_dict(self, state_dict, strict)
   1481         if len(error_msgs) > 0:
   1482             raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
-> 1483                                self.__class__.__name__, "\n\t".join(error_msgs)))
   1484         return _IncompatibleKeys(missing_keys, unexpected_keys)
   1485

RuntimeError: Error(s) in loading state_dict for Generator:
	Missing key(s) in state_dict: "style.3.weight", "style.3.bias", "style.4.weight", "style.4.bias", "style.5.weight", "style.5.bias", "style.6.weight", "style.6.bias", "style.7.weight", "style.7.bias", "style.8.weight", "style.8.bias".
	size mismatch for convs.6.conv.weight: copying a param with shape torch.Size([1, 256, 512, 3, 3]) from checkpoint, the shape in current model is torch.Size([1, 512, 512, 3, 3]).
	size mismatch for convs.6.activate.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([512]).
	size mismatch for convs.7.conv.weight: copying a param with shape torch.Size([1, 256, 256, 3, 3]) from checkpoint, the shape in current model is torch.Size([1, 512, 512, 3, 3]).
	size mismatch for convs.7.conv.modulation.weight: copying a param with shape torch.Size([256, 512]) from checkpoint, the shape in current model is torch.Size([512, 512]).
	size mismatch for convs.7.conv.modulation.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([512]).
	size mismatch for convs.7.activate.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([512]).
	size mismatch for convs.8.conv.weight: copying a param with shape torch.Size([1, 128, 256, 3, 3]) from checkpoint, the shape in current model is torch.Size([1, 256, 512, 3, 3]).
	size mismatch for convs.8.conv.modulation.weight: copying a param with shape torch.Size([256, 512]) from checkpoint, the shape in current model is torch.Size([512, 512]).
	size mismatch for convs.8.conv.modulation.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([512]).
	size mismatch for convs.8.activate.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([256]).
	size mismatch for convs.9.conv.weight: copying a param with shape torch.Size([1, 128, 128, 3, 3]) from checkpoint, the shape in current model is torch.Size([1, 256, 256, 3, 3]).
	size mismatch for convs.9.conv.modulation.weight: copying a param with shape torch.Size([128, 512]) from checkpoint, the shape in current model is torch.Size([256, 512]).
	size mismatch for convs.9.conv.modulation.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([256]).
	size mismatch for convs.9.activate.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([256]).
	size mismatch for convs.10.conv.weight: copying a param with shape torch.Size([1, 64, 128, 3, 3]) from checkpoint, the shape in current model is torch.Size([1, 128, 256, 3, 3]).
	size mismatch for convs.10.conv.modulation.weight: copying a param with shape torch.Size([128, 512]) from checkpoint, the shape in current model is torch.Size([256, 512]).
	size mismatch for convs.10.conv.modulation.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([256]).
	size mismatch for convs.10.activate.bias: copying a param with shape torch.Size([64]) from checkpoint, the shape in current model is torch.Size([128]).
	size mismatch for convs.11.conv.weight: copying a param with shape torch.Size([1, 64, 64, 3, 3]) from checkpoint, the shape in current model is torch.Size([1, 128, 128, 3, 3]).
	size mismatch for convs.11.conv.modulation.weight: copying a param with shape torch.Size([64, 512]) from checkpoint, the shape in current model is torch.Size([128, 512]).
	size mismatch for convs.11.conv.modulation.bias: copying a param with shape torch.Size([64]) from checkpoint, the shape in current model is torch.Size([128]).
	size mismatch for convs.11.activate.bias: copying a param with shape torch.Size([64]) from checkpoint, the shape in current model is torch.Size([128]).
	size mismatch for to_rgbs.3.conv.weight: copying a param with shape torch.Size([1, 3, 256, 1, 1]) from checkpoint, the shape in current model is torch.Size([1, 3, 512, 1, 1]).
	size mismatch for to_rgbs.3.conv.modulation.weight: copying a param with shape torch.Size([256, 512]) from checkpoint, the shape in current model is torch.Size([512, 512]).
	size mismatch for to_rgbs.3.conv.modulation.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([512]).
	size mismatch for to_rgbs.4.conv.weight: copying a param with shape torch.Size([1, 3, 128, 1, 1]) from checkpoint, the shape in current model is torch.Size([1, 3, 256, 1, 1]).
	size mismatch for to_rgbs.4.conv.modulation.weight: copying a param with shape torch.Size([128, 512]) from checkpoint, the shape in current model is torch.Size([256, 512]).
	size mismatch for to_rgbs.4.conv.modulation.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([256]).
	size mismatch for to_rgbs.5.conv.weight: copying a param with shape torch.Size([1, 3, 64, 1, 1]) from checkpoint, the shape in current model is torch.Size([1, 3, 128, 1, 1]).
	size mismatch for to_rgbs.5.conv.modulation.weight: copying a param with shape torch.Size([64, 512]) from checkpoint, the shape in current model is torch.Size([128, 512]).
	size mismatch for to_rgbs.5.conv.modulation.bias: copying a param with shape torch.Size([64]) from checkpoint, the shape in current model is torch.Size([128]).

It's the same channel_multiplier issue. However, I see that I was not currently passing the channel_multiplier argument to the generator constructor.
I just pushed an update that fixes this. Just grab the new version add the same --channel_multiplier 1 argument when you call train.py

Thanks
I pulled the changes, and passed the --channel_multiplier 1 argument when calling train.py.
Getting the following error:
Loading base models...
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-5-f7063a26894a> in <module>()
     70
     71 print("Loading base models...")
---> 72 net = ZSSGAN(args)
     73 print("Models loaded! Starting training...")
     74

2 frames
/content/gdrive/MyDrive/ML/Code/StyleGAN_NADA/stylegan_nada/ZSSGAN/model/ZSSGAN.py in __init__(self, args)
    154
    155         # Set up frozen (source) generator
--> 156         self.generator_frozen = SG2Generator(args.frozen_gen_ckpt, img_size=args.size, channel_multiplier=args.channel_multiplier).to(self.device)
    157         self.generator_frozen.freeze_layers()
    158         self.generator_frozen.eval()

/content/gdrive/MyDrive/ML/Code/StyleGAN_NADA/stylegan_nada/ZSSGAN/model/ZSSGAN.py in __init__(self, checkpoint_path, latent_size, map_layers, img_size, channel_multiplier, device)
     29         checkpoint = torch.load(checkpoint_path, map_location=device)
     30
---> 31         self.generator.load_state_dict(checkpoint["g_ema"], strict=True)
     32
     33         with torch.no_grad():

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in load_state_dict(self, state_dict, strict)
   1481         if len(error_msgs) > 0:
   1482             raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
-> 1483                                self.__class__.__name__, "\n\t".join(error_msgs)))
   1484         return _IncompatibleKeys(missing_keys, unexpected_keys)
   1485

RuntimeError: Error(s) in loading state_dict for Generator:
	Missing key(s) in state_dict: "style.3.weight", "style.3.bias", "style.4.weight", "style.4.bias", "style.5.weight", "style.5.bias", "style.6.weight", "style.6.bias", "style.7.weight", "style.7.bias", "style.8.weight", "style.8.bias".

Looks like your model is also using 2 mapping network layers instead of the standard 8.
Can you try modifying line 22 of ZSSGAN/model/ZSSGAN.py so that the default value for map_layers is 2 and not 8?
Basically change it so it reads:
def __init__(self, checkpoint_path, latent_size=512, map_layers=2, img_size=256, channel_multiplier=2, device='cuda:0'):
If that works for you, I'll see about adding the option as a command line argument so you don't have to play with the code itself.

I set the value of map_layers to 2 and it works now!
Thanks a lot for all the help!

No problem :) Keep in mind that if you later want to load this model with rosinality's code in order to generate more images, you'll need to similarly set channel_multiplier and the number of mapping layers (I believe his code calls it n_mlp).
If that's it, I'll close the issue for now. Feel free to re-open it or start a new issue if you need additional help!

@rinongal
Hi, thank for great release.
I face the error in my custom dataset.
After add the arg "--channel_multiplier 1" when I call train.py.
The error still happened.
I have no ideal how to fix it.
Initializing networks...
Traceback (most recent call last):
  File "train.py", line 161, in <module>
    train(args)
  File "train.py", line 58, in train
    net = ZSSGAN(args)
  File "/mnt/StyleGAN-nada-main/ZSSGAN/model/ZSSGAN.py", line 275, in __init__
    self.generator_frozen = SG2Generator(args.frozen_gen_ckpt, img_size=args.size).to(self.device)
  File "/mnt/StyleGAN-nada-main/ZSSGAN/model/ZSSGAN.py", line 136, in __init__
    self.generator.load_state_dict(checkpoint["g_ema"], strict=True)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 1052, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for Generator:
        size mismatch for convs.6.conv.weight: copying a param with shape torch.Size([1, 256, 512, 3, 3]) from checkpoint, the shape in current model is torch.Size([1, 512, 512, 3, 3]).
        size mismatch for convs.6.activate.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([512]).
        size mismatch for convs.7.conv.weight: copying a param with shape torch.Size([1, 256, 256, 3, 3]) from checkpoint, the shape in current model is torch.Size([1, 512, 512, 3, 3]).
        size mismatch for convs.7.conv.modulation.weight: copying a param with shape torch.Size([256, 512]) from checkpoint, the shape in current model is torch.Size([512, 512]).
        size mismatch for convs.7.conv.modulation.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([512]).
        size mismatch for convs.7.activate.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([512]).
        size mismatch for convs.8.conv.weight: copying a param with shape torch.Size([1, 128, 256, 3, 3]) from checkpoint, the shape in current model is torch.Size([1, 256, 512, 3, 3]).
        size mismatch for convs.8.conv.modulation.weight: copying a param with shape torch.Size([256, 512]) from checkpoint, the shape in current model is torch.Size([512, 512]).
        size mismatch for convs.8.conv.modulation.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([512]).
        size mismatch for convs.8.activate.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([256]).
        size mismatch for convs.9.conv.weight: copying a param with shape torch.Size([1, 128, 128, 3, 3]) from checkpoint, the shape in current model is torch.Size([1, 256, 256, 3, 3]).
        size mismatch for convs.9.conv.modulation.weight: copying a param with shape torch.Size([128, 512]) from checkpoint, the shape in current model is torch.Size([256, 512]).
        size mismatch for convs.9.conv.modulation.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([256]).
        size mismatch for convs.9.activate.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([256]).
        size mismatch for convs.10.conv.weight: copying a param with shape torch.Size([1, 64, 128, 3, 3]) from checkpoint, the shape in current model is torch.Size([1, 128, 256, 3, 3]).
        size mismatch for convs.10.conv.modulation.weight: copying a param with shape torch.Size([128, 512]) from checkpoint, the shape in current model is torch.Size([256, 512]).
        size mismatch for convs.10.conv.modulation.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([256]).
        size mismatch for convs.10.activate.bias: copying a param with shape torch.Size([64]) from checkpoint, the shape in current model is torch.Size([128]).
        size mismatch for convs.11.conv.weight: copying a param with shape torch.Size([1, 64, 64, 3, 3]) from checkpoint, the shape in current model is torch.Size([1, 128, 128, 3, 3]).
        size mismatch for convs.11.conv.modulation.weight: copying a param with shape torch.Size([64, 512]) from checkpoint, the shape in current model is torch.Size([128, 512]).
        size mismatch for convs.11.conv.modulation.bias: copying a param with shape torch.Size([64]) from checkpoint, the shape in current model is torch.Size([128]).
        size mismatch for convs.11.activate.bias: copying a param with shape torch.Size([64]) from checkpoint, the shape in current model is torch.Size([128]).
        size mismatch for to_rgbs.3.conv.weight: copying a param with shape torch.Size([1, 3, 256, 1, 1]) from checkpoint, the shape in current model is torch.Size([1, 3, 512, 1, 1]).
        size mismatch for to_rgbs.3.conv.modulation.weight: copying a param with shape torch.Size([256, 512]) from checkpoint, the shape in current model is torch.Size([512, 512]).
        size mismatch for to_rgbs.3.conv.modulation.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([512]).
        size mismatch for to_rgbs.4.conv.weight: copying a param with shape torch.Size([1, 3, 128, 1, 1]) from checkpoint, the shape in current model is torch.Size([1, 3, 256, 1, 1]).
        size mismatch for to_rgbs.4.conv.modulation.weight: copying a param with shape torch.Size([128, 512]) from checkpoint, the shape in current model is torch.Size([256, 512]).
        size mismatch for to_rgbs.4.conv.modulation.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([256]).
        size mismatch for to_rgbs.5.conv.weight: copying a param with shape torch.Size([1, 3, 64, 1, 1]) from checkpoint, the shape in current model is torch.Size([1, 3, 128, 1, 1]).
        size mismatch for to_rgbs.5.conv.modulation.weight: copying a param with shape torch.Size([64, 512]) from checkpoint, the shape in current model is torch.Size([128, 512]).
        size mismatch for to_rgbs.5.conv.modulation.bias: copying a param with shape torch.Size([64]) from checkpoint, the shape in current model is torch.Size([128]).

@snow1929 are you sure you're running with the right --size argument? What is your base model and what is your full train command?

@rinongal
Here is my full command for training.
 You need to either modify things here or at the point mentioned in this prior reply

I try modifying line 22 of ZSSGAN/model/ZSSGAN.py
def __init__(self, checkpoint_path, latent_size=512, map_layers=2, img_size=256, channel_multiplier=2, device='cuda:0'):

But the error still happened.

Sorry, looks like the line moved in one of the commits since that reply. It's this line: https://github.com/rinongal/StyleGAN-nada/blob/dc8406ae2173ad186f8f03f3cadf65e613ac9364/ZSSGAN/model/ZSSGAN.py#L126
Basically if you want to change the map layers or the channel multiplier, you need to change the number there, or modify the training script to actually pass and use these arguments.

@rinongal
Thanks for heling me a lot.
I fix the error by this method.

@snow1929 Looks like your model has 2 mapping network layers and you're trying to load into a model that expects 8. Try to adjust it here: https://github.com/rinongal/StyleGAN-nada/blob/dc8406ae2173ad186f8f03f3cadf65e613ac9364/ZSSGAN/generate_videos.py#L236 or promote it to a full arg and pass that when running the script.

@rinongal
what kind of model you used to translate the image to laten space ?ReStyle or StyleFlow?
would you mind to share the detail of cmd code ?
I face trouble on this part.
please.  😞 😞 😞


VMR codespace fails to build aspnetcore because of missing node

    /workspaces/dotnet/src/aspnetcore/src/Components/WebView/WebView/src/Microsoft.AspNetCore.Components.WebView.csproj(68,5): warning : '../../../Web.JS/dist/Release/blazor.webview.js' does not exist. Falling back to checked-in copy.
    /workspaces/dotnet/src/aspnetcore/src/Assets/Microsoft.AspNetCore.App.Internal.Assets.csproj(46,5): error : '/workspaces/dotnet/src/aspnetcore/src/Components/Web.JS/dist/Release/blazor.web.js' does not exist. Run 'npm run build' in the repo root to generate the file.
    CSC : error CS1566: Error reading resource '_framework/blazor.webview.js' -- 'Could not find a part of the path '/workspaces/dotnet/src/aspnetcore/src/Components/Web.JS/dist/Release/blazor.webview.js'.' [/workspaces/dotnet/src/aspnetcore/src/Components/WebView/WebView/src/Microsoft.AspNetCore.Components.WebView.csproj]
        1 Warning(s)
        2 Error(s)

cc @wtgodbe
What's the expectation here? Will we always need to have node installed when building aspnetcore inside the VMR on Linux and Windows or will this get fixed with https://github.com/dotnet/aspnetcore/issues/59114? Note that the VMR aspnetcore build must be a 100% complete build as eventually the VMR becomes the shipping vehicle.

This should be fixed now with https://github.com/dotnet/aspnetcore/pull/59118


Problem when calling memset on raspi4

Hi,
I am currently trying to call memset to initialize bss section on my raspi4, but found out the board halted as a result.
After digging into the implementation of memset, I realized that the alignment fault occurred when calling this function.
According to Armv8-A architecture profile, when stage 1 address translation is disabled, the memory will be defined as device type (in D5.2.9 section).
However, any unaligned access to device type memory will cause an alignment fault and halt the board (in B2.5.2 section).
In current stage, AMT does not enable MMU. Any unaligned access to the memory may cause this fault and freeze the kernel.
Is there any idea to overcome this problem? Or we should turn on the setting for MMU?

I haven't dug into the implementation so I assume that aligning .bss in the linker script isn't enough. Since memset itself will do something unaligned during the course of the copy.
So initialising the MMU would be the way to go. Ideally just to a flat layout that allows unaligned accesses.

Now that Raspi4 has landed I think we can call this fixed.


Add instructions suitable for FreeBSD and similar.

Not clear what other systems have people maintaining GHC & Cabal packages, but that's the place to start for most other people.

@mwm Cabal installing alex & happy into your user package-db will only work if ~/.cabal/bin is in your $PATH.

@bitemyapp good catch. Fixed. Also added instructions to update cabal-install to save the readers a step.

@mwm why are there add'l spaces in lines 209-214?

Thank you @mwm !
:smile_cat:


Watchdog support for Spark-3.2

Why are the changes needed?

move spark-3.1 ForcedMaxOutputRowsRule to spark-common and rename to ForcedMaxOutputRowsBase
move spark-3.1 MarkAggregateOrderRule to spark-common and rename to MarkAggregateOrderBase
handle WithCTE logical plan in spark-3.2
move spark-3.1 MaxPartitionStrategy to spark-common
add netsted cte unit test for ForcedMaxOutputRowsRule

How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[x] Run test locally before make a pull request

Codecov Report

Merging #1591 (364fc26) into master (df1d9f3) will decrease coverage by 1.05%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##             master    #1591      +/-   ##
============================================
- Coverage     59.02%   57.97%   -1.06%
+ Complexity      196      140      -56
============================================
  Files           256      256
  Lines         12708    12708
  Branches       1601     1600       -1
============================================
- Hits           7501     7367     -134
- Misses         4570     4719     +149
+ Partials        637      622      -15

Impacted Files
Coverage Δ

.../kyuubi/sql/watchdog/ForcedMaxOutputRowsBase.scala
0.00% <0.00%> (ø)

.../kyuubi/sql/watchdog/KyuubiWatchDogException.scala
0.00% <ø> (ø)

...che/kyuubi/sql/watchdog/MaxPartitionStrategy.scala
0.00% <ø> (ø)

...che/spark/sql/PruneFileSourcePartitionHelper.scala
0.00% <ø> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update df1d9f3...364fc26. Read the comment docs.

cc @ulysses-you and watchdog original author @zhouyifan279 @i7xh

thanks, merging to master


Update getting-started.md for type errors regarding routing functionality

Before the server can be started via npm run dev, the code for routeTree.gen.ts file needs to be created. This is done by, first, running npm run build.

npm run dev should also create the file, does this not work for you?


GNU Screen Permission Error

Heyyo, So after i did a yearly os reset by installing a brand new centos 7 minimal on the machine i decided to give this panel another try. I installed GNU Screen, Apache, PHP 5.4.16 & Java 8. I doubt it would work this time, But it didnt. It threw same error at me as before:
Running as user: apache
Cannot make directory '/var/run/screen/S-apache': Permission denied
I tried everything i could. I'm starting to wounder if its the apache name thats incorrect for me and the permissions. The webserver files are owned by "www-data" as you might have expected, But if i do the exec whoami in php it tells me apache. And same does your script when saying: "Running as user: apache". Shall i set the webserver files to apache too? Or what permission changes are required for this to work.
It would be awesome if you could help me make this work, And i really hate opening a new issue for this.
Thanks! ~Julle

chmod lol?

To what? It dosent accept anything.

You seems not to have correct permissions setup for the /var/run/screen folder for the webserver user.
A ls -al should show the set permission in the directory.


Remove babelrc config loading on runtime

Hello,
I clean all the noise of loading the babel config stuff and run all the JS files with babel-node, then should be easy to move into Babel 6.

Run all the npm scripts with babel-node (that automatically get the .babelrc)
Refactor and Move the bin/ into api/api.js and server.js`` and remove server.babel.js`
Remove babel/polyfill in all the files
Refactor the imports in webpack configs to ES6

PS. I can't try the stuff on Windows, can any Microsoft user try it before merge it?
Cheers

I actually had this same thought, but see https://github.com/erikras/react-redux-universal-hot-example/pull/414 -- babel-node is not for production use.

Ops, sorry I didn't know it.


Compile Errors in UIBinderGenerator and UIBinderParser

Hi
I'm using some of your libraries (ahome-core, ahome-touch, ahome-touch-ux and ahome-sencha-shared). After I updated these libraries and fixed some errors in my code, there remain some last compile errors in the ahome-core library.
The errors occur in the UIBinderGenerator and UIBinderParser classes.
Do I have to include an another library?
These are the errors:

The constructor UiBinderWriter(JClassType, String, String, TypeOracle, MortalLogger, FieldManager, MessagesWriter, DesignTimeUtils, UiBinderContext, Boolean, boolean, String, ResourceOracle) is undefined

UiBinderGenerator.java
/lib_touch4j/srcAhomeCore/com/google/gwt/uibinder/rebind
line 167
Java Problem

The method getResource(String) is undefined for the type ResourceOracle

UiBinderGenerator.java
/lib_touch4j/srcAhomeCore/com/google/gwt/uibinder/rebind
line 217
Java Problem

StringCase cannot be resolved

UiBinderParser.java
/lib_touch4j/srcAhomeCore/com/google/gwt/uibinder/rebind
line 515
Java Problem

The import com.google.gwt.core.shared.impl.StringCase cannot be resolved

UiBinderParser.java
/lib_touch4j/srcAhomeCore/com/google/gwt/uibinder/rebind
line 26
Java Problem

The method createCssResource(String, String[], JClassType, String, LinkedHashSet) in the type ImplicitClientBundle is not applicable for the arguments (String, String[], JClassType, String, LinkedHashSet, Boolean, ResourceOracle)

UiBinderParser.java
/lib_touch4j/srcAhomeCore/com/google/gwt/uibinder/rebind
line 456
Java Problem

The method createDataResource(String, String) in the type ImplicitClientBundle is not applicable for the arguments (String, String, String, Boolean)

UiBinderParser.java
/lib_touch4j/srcAhomeCore/com/google/gwt/uibinder/rebind
line 191
Java Problem

What version of GWT are you using ?
Do you use maven or are you using the Touch4j.jar file ?

Hi
Thank you for your reply.
I tried GWT versions 2.5, 2.6 and 2.6.1.
I downloaded the zip files here on github and included the sources into my project.

Hi,
currently the lib supports only GWT 2.7 and above. Also make sure that the lib has a higher priority  als the GWT SDK in the classpath.

Great, thank you, now it works. I didn't knew there's already a final build of 2.7, because I always found the RC1.


"Reserved" words not working in Model

After the recent release, it seems the Reserved words are not working. Here is my config:
    public function sluggable()
    {
        return [
            'slug' => [
                'source' => 'title',
                'maxLength' => 50,
                'reserved' => array('a','about','above','after','again','all','am','an','and','any','are','as','at','be','because','been','before','being','below','both','but','by','could','did','do','does','doing','down','during','each','few','for','from','further','had','has','have','having','he','he\'d','he\'ll','he\'s','her','here','here\'s','hers','herself','him','himself','his','how','how\'s','I','I\'d','I\'ll','I\'m','I\'ve','if','in','into','is','it','it\'s','its','itself','let\'s','me','more','most','my','myself','nor','of','on','once','only','or','other','ought','our','ours','ourselves','out','over','own','same','she','she\'d','she\'ll','she\'s','should','so','some','such','than','that','that\'s','the','their','theirs','them','themselves','then','there','there\'s','these','they','they\'d','they\'ll','they\'re','they\'ve','this','those','through','to','too','until','up','very','was','we','we\'d','we\'ll','we\'re','we\'ve','were','what','what\'s','when','when\'s','where','where\'s','which','while','who','who\'s','whom','why','why\'s','with','would','you','you\'d','you\'ll','you\'re','you\'ve','your','yours','yourself','yourselves'),
            ]
        ];
    }

I'm still seeing a slug containing "with" such as this:
toshiba-pa3917u-1aca-pa-1650-81-compatible-with

Ah, I see what the issue is.  reservedWordsis a list that is checked against the final slug, not individual words within the slug.
The logic here is that you might not want a string of "admin tools are awesome" to slug down to "admin", and then result in a URL of http://example.com/admin ... which is a special route on your site.
In your case, you want to clean up the source string of specific words before generating the slug.  So, remove the reservedWords list from your configuration.  Then set your source to cleanedTitle.  Finally, create a new accessor method on your model, something like this (untested):
/**
 * Get a cleaned version of the title, with bad words removed.
 *
 * @param  string $value
 * @return string
 */
public function getCleanedTitle($value) {
	$badWords = ['a', 'about', 'above', 'after', ...];

	// convert each word into a regex
	array_walk($badWords, function(&$word) {
		$word = '/\b' . preg_quote($word, '/') . '\b/';
	});

	// strip the bad words from the title
	$value = preg_replace($badWords, '', $value);

	// return the cleaned title
	return $value;
}

That seemed to do the trick!
+1 - Thanks @cviebrock
For others, I used this as my sluggable function.
    public function sluggable()
    {
        return [
            'slug' => [
                'source' => $this->getCleanedTitle('title'),
                'maxLength' => 70
            ]
        ];
    }

You don't need to set the cleanTitle on your model.  Setting the title is good enough. My code adds an accessor so that $product->cleanedTitle returns the correct string.
So you can remove this line from the code above:
$product->cleanTitle = $product->getCleanedTitle($this->title);

Also it looks like the source in your sluggable config has "cleanTitle" instead of "cleanedTitle".  Fixing that should help. ;)

@cviebrock thanks for the help.
I did that, getting this error
In Connection.php line 664:

  SQLSTATE[23000]: Integrity constraint violation: 1048 Column 'slug' cannot be null (SQL: insert into `products` (`sku`, `title`, `list_price`, `node_id`, `slug`, `brand_id`, `updated_at
  `, `created_at`) values (B075V234VL, Samsung Qi Certified Fast Charge Wireless Charger (Universally compatible with all Qi enabled phones) - Special Edition  – Black/Blue, 49.99, 125576
  39011, , 1, 2017-12-12 15:11:54, 2017-12-12 15:11:54))

In Connection.php line 458:

  SQLSTATE[23000]: Integrity constraint violation: 1048 Column 'slug' cannot be null

In the docs, it mentions getVarAttribute, so should it be getCleanedTitleAttribute() instead? I'm still new to Laravel and learning, I greatly appreciate your help and I understand this is probably beyond the the scope of a closed ticket. Hoping someone else can benefit from the testing.
In my model
    public function sluggable()
    {
        return [
            'slug' => [
                'source' => 'cleanedTitle',
                'maxLength' => 70
            ]
        ];
    }

@cviebrock a little help from #laravel and it's solved. Thanks again for your great package and your hard work supporting it.
    public function sluggable()
    {
        return [
            'slug' => [
                'source' => 'clean_title',
                'maxLength' => 70
            ]
        ];
    }
...
    public function getCleanTitleAttribute($value) {
        $badWords = ['w/','a','about','above','after','again','all','am','an','and','any','are','as','at','be','because','been','before','being','below','both','but','by','could','did','do','does','doing','down','during','each','few','for','from','further','had','has','have','having','he','he\'d','he\'ll','he\'s','her','here','here\'s','hers','herself','him','himself','his','how','how\'s','I','I\'d','I\'ll','I\'m','I\'ve','if','in','into','is','it','it\'s','its','itself','let\'s','me','more','most','my','myself','nor','of','on','once','only','or','other','ought','our','ours','ourselves','out','over','own','same','she','she\'d','she\'ll','she\'s','should','so','some','such','than','that','that\'s','the','their','theirs','them','themselves','then','there','there\'s','these','they','they\'d','they\'ll','they\'re','they\'ve','this','those','through','to','too','until','up','very','was','we','we\'d','we\'ll','we\'re','we\'ve','were','what','what\'s','when','when\'s','where','where\'s','which','while','who','who\'s','whom','why','why\'s','with','would','you','you\'d','you\'ll','you\'re','you\'ve','your','yours','yourself','yourselves'];

        // convert each word into a regex
        array_walk($badWords, function(&$word) {
            $word = '/\b' . preg_quote($word, '/') . '\b/';
        });

        // strip the bad words from the title
        $value = preg_replace($badWords, '', $this->attributes['title']);

        // return the cleaned title
        return $value;
    }

Ah yeah ... oops.  I got the name of my accessor method wrong. :)


[SMALLFIX]javadoc for TestBlockOutStream#TestBlockOutStream

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/14436/
Test PASSed.

Merged build finished. Test PASSed.


seq and negative input

In ~/.loda-rust/analytics/programs_invalid_verbose.csv there are the programs that is causing trouble and what seems to be the cause. It's been 3 months since I last looked at it and I don't recall seeing this problem. Now there are 90 invalid programs with EvalSequenceWithNegativeParameter. Probably recently introduced.
Problem
The webeditor can't execute the A004001 program, since the input is negative in first iteration.
lpb $0
  sub $0,1
  mov $2,$0
  seq $2,93879 ; First differences of A004001.
  add $1,$2
lpe
add $1,1
mov $0,$1

git history for A004001.asm: The file was added to git 5 days ago.
Solution A
LODA-CPP rejecting negative input, before accepting a new program / update to existing program.
Solution B
There are 90 programs that uses negative inputs when computing the 1st term. These will have to be deleted manually (a long list).
A004001
A006454
A006998 calls A205591 which uses negative input
A007000
A007417
A024450
A024924
A025475
A048148
A049628
A051293
A066077
A069891
A072004
A072114
A075994
A076793
A080358
A080677
A080704
A084505
A086052
A086525
A092401
A097332
A099958
A103116
A108906
A112073
A112556
A113510
A116466
A117276
A127719
A131118
A131804
A135339
A141253
A157100
A159793
A159795
A160118
A160119
A160428
A160744
A161426
A161832
A164158
A167549
A168472
A171645
A172243
A180016
A182716
A182843
A188557
A193215
A205591
A211634
A211637
A211638
A211646
A217069
A224914
A227844
A229506
A245380
A249435
A252749
A266532
A268395
A268672
A268708
A268905
A270593
A273324
A273780
A273781
A275112
A278491
A278603
A279557
A279927
A282282
A285985
A289898
A326118
A336415
A356005

This is a bug in loda-cpp caused by enabling IE in checks. Needs to be fixed there

Already fixed


function that does not exist: initLedger

Under ⇧⌘P I could not find IBM Blockchain Platform: Package a Smart Contract Project, so I used  IBM Blockchain Platform: Export Package.
Instead of the<EMAIL_ADDRESS>I saw<EMAIL_ADDRESS>which I used.
Later, when I instantiated the smart contract I got the error message:
Error instantiating smart contract: transaction returned with failure: Error: You've asked to invoke a function that does not exist: initLedger

remark: I did the local fabcar demo

I deleted the<EMAIL_ADDRESS>restarted the Macbook, opened the contract folder (not sure if I did this before). Then it was working.


[Relay] CompilerAttrs hasn't registered with TVM_REGISTER_NODE_TYPE

if expr has CompilerAttrs , and use astext() to show it, compiler will report error: relay.attrs.CompilerAttrs is not registered via TVM_REGISTER_NODE_TYPE.  test_pass_partition_graph.py doesn't report the error because it doesn't contain any log.

cc @comaniac


I can´t login

sorry for my ignorance , how to login? That error appear in tab credentials too, i need configure something in site/model.xml?

The pseudo and password fields on the "Login" tab should not contain your credentials but the name of the fields that should be posted on the form on the URL (which you didn't fill).
For example on Gelbooru, the login page is here:
http://www.gelbooru.com/index.php?page=account&s=login&code=00
By looking at the source code, we can see:

the name of pseudo field is "user" (<input type="text" name="user" value="" />)
the password field is "pass" (<input type="password" name="pass" value="" />)
the URL is "http://www.gelbooru.com/index.php?page=account&s=login&code=00"
the method is "POST" (<form method="post" action="">)

The cookie is harder to get for non-technical people but it's not really important, it's only used for the testing feature (you didn't set it, that's why you get the "unable to test" message). For Gelbooru it's "user_id", but it's very often "PHPSESSID".
Your actual username and password should be set in the "Credentials" tab. If you can't get it to work, I'll do my best to help if you tell me which source you're trying to add.

sorry, i forget post the source D:

Sankaku is already configured by default. You have to re-check "through url" and put your credentials in the right tab. It can't be tested by design, though (you'll get a "unable to test" message if you try). But since you're logging in I guess it's to bypass some limitation for guest accounts, so you can just check if they are properly removed after putting your credentials (page limit maybe?).

i trying download this tag and can see 11877 but just download 1000 images or below
I need to do something with this ?

Sorry for my bad english

i think this now fixed

reached 101 pages

So you got it to work eventually? :smile:
From this screenshot:

It looked like you didn't hash your password at that time, but seems like it's good now so I'm closing this issue.

I have this same problem trying to log in for Sankaku with "Unable to test" but I don't have a "Hash a password" button. So I don't see any way to resolve it.


Can you provide and fix the Amd instinct mi250x driver.

There is a AMD instinct mi250x which is 102-D65201. Which cannot recognize by amdgpu and rocm. It's a mi250x from hpe cray ex235a. Thank you.

Is that AMD instinct mi250x in HPE Cray ex235a Only supported in hpe Cray OS? And cannot used in Suse Linux or ubuntu? It Seems that ROCm doesn't support this Amd mi250x model type 102-D65201.

MI250 is supported in ROCm across the list of supported OS+kernel configurations at https://rocm.docs.amd.com/projects/install-on-linux/en/latest/reference/system-requirements.html. It's supported in SLE15 SP3 and SP4 (both of which I've used MI250X in) and Ubuntu 20.04 and 22.04 (both of which I've used MI250X in). You'll need to give more info beyond "not recognized".
What did you do to install it? Which ROCm version are you using? Which OS+kernel are you using? Does dmesg say anything? Does lspci see the card?
If you look at the support list, all GPUs in that list are tested and working in all OS+kernel combinations unless otherwise stated. That's how we compile those lists of supported configurations. MI250X works in all of the supported distros, and is tested by QA. And I've personally used it in those listed OSes, so we'll need more info to help out here.

This is exactly my card and problem.
https://github.com/amd/amd-lab-notes/discussions/10
So, is that this AMD instinct mi250x which is 113-D65201 need a specialized and modified AMDGPU version?

So I am thinking that the VBIOS on there is the issue. I don't see any internal notes for version X06 on that GPU, which leads me to believe that it's a test VBIOS or some intermediary VBIOS on there. Now normally for MI products when you're buying from AMD, you get in touch with a dedicated rep who can provide the necessary resources like VBIOSes and such. That's why we don't have them searchable on our website. Due to this, we don't have a VBIOS that we can provide at this time.
However if you were to use your favourite search engine, you might find something like https://support.hpe.com/connect/s/softwaredetails?language=en_US&softwareId=MTX_3d12b8be3c1844998007a58aa1&tab=releaseNotes , which would appear to have a download option for a software package that also includes VBIOS 113-D65201-046-609321 , which would be compatible with your card. We can't officially recommend this approach from an AMD support perspective, however. This is because there's no official support for 2nd-hand-purchases of MI products. But If you intended to try to do things yourself, you could save the existing X06 VBIOS via amdvbflash, and try to flash this other non-AMD-distributed VBIOS to see if that resolves the issue. If it doesn't work, you can always re-flash the X06 VBIOS back on there. Again, not an official AMD recommendation, but something you might have found by yourself that has potential to resolve the issue (since 2nd-hand purchase means no options for RMA if the card is faulty)
But considering that I don't see X06 anywhere in any officially-supported internal documentation, it would lead me to believe that you've got an unsupported VBIOS on there (either a test one, one that has known issues and was subsequently patched, or one that got corrupted), which is why the system is hanging on reading the VBIOS out in the kernel driver. amdvbflash will just dump contents, that's fine. The kernel driver processes the VBIOS and such differently, which would explain the different behaviour.
I can't recommend anything officially, nor do we have an official solution here. But if you were to try to find a VBIOS specific for that card, like in the link above, and try to flash that, you might have luck from an independent perspective. MI250X should just work with any supported ROCm build without any special change for any special versions of the SKU. In this case, it looks like something is stuck in the VBIOS parsing in the kernel, and the first solution in that case is usually "reflash or update the VBIOS".

Thank you. So good to hear that rocm support all SKU.
amdvbflash doesn't flash its vbios successfully. HPE use another way to flash the BIOS, probably a tool called FAS. I am trying to fix it using HPE way.
Any other flashing ways besides Amdvbflash?
Thank you so much!

If the system is up, you can use debugfs to dump VBIOSes (amdgpu_vbios is the filename), but in the case where you can't get GPUs to init, you're stuck with the amdvbflash tool, unfortunately. HPE may have their own flashing tool but the only software AMD tool is amdvbflash. They may be able to offer some support in that regard. Fingers crossed!

so if the GPU doesn't support IFWI update, there is nothing that AMDVBFLASH can do?

HI, I'm also interested in this GPU. I've found something on HPE software repo. There is a firmware for MI200 series in https://downloads.hpe.com/pub/softlib2/software1/pubsw-linux/p1601384579/v243081/HFP-firmware-23.12.0.tar.gz, maybe try flashing it on can solve the problem of ROCm drivers.
By the way, I read a paper about Cray EX235a. The MI250X they use is 113-D65201. 102-D65201 seems to be an engeneering sample. Flashing the BIOS may solve the issue.

HI, I'm also interested in this GPU. I've found something on HPE software repo. There is a firmware for MI200 series in https://downloads.hpe.com/pub/softlib2/software1/pubsw-linux/p1601384579/v243081/HFP-firmware-23.12.0.tar.gz, maybe try flashing it on can solve the problem of ROCm drivers. By the way, I read a paper about Cray EX235a. The MI250X they use is 113-D65201. 102-D65201 seems to be an engeneering sample. Flashing the BIOS may solve the issue.

 So basically there is no solution yet. I knew the firmware you mentioned.

Maybe you can try to look for a friend who has an HPE server and flash it using fwupdate？HPE's firmware flashing utility should be compatiable for all their BIOSes

I got a MI250X of the same model, 102-D65201.
I haven't tested it because I lacked parts like PSU and wires，but I am also curious about the driver issue.
I found that many people try to use OAM to PCIe adapter cards provided by NVIDIA (which was originally designed to test A100 OAMs) to activate MI250X.
However, it is said that the adapter has only one available OAM connector available，with another one non-functional. That means only one GPU is activated. Since the 2 GPUs are packaged together，I guess that might lead to the problem.
I bought a SuperMicro 4-Module OAM board，and I am trying to connect it to my computer via 2 PCIE x16 slots.

I got a MI250X of the same model, 102-D65201.
I haven't tested it because I lacked parts like PSU and wires，but I am also curious about the driver issue.
I found that many people try to use OAM to PCIe adapter cards provided by NVIDIA (which was originally designed to test A100 OAMs) to activate MI250X.
However, it is said that the adapter has only one OAM connector available，with another one non-functional. That means only one GPU is activated. Since the 2 GPUs are packaged together，I guess that might lead to the problem.
I bought a SuperMicro 4-Module OAM board，and I am trying to connect it to my computer via 2 PCIE x16 slots.

I got a MI250X of the same model, 102-D65201.
I haven't tested it because I lacked parts like PSU and wires，but I am also curious about the driver issue.
I found that many people try to use OAM to PCIe adapter cards provided by NVIDIA (which was originally designed to test A100 OAMs) to activate MI250X.
However, it is said that the adapter has only one OAM connector available，with another one non-functional. That means only one GPU is activated. Since the 2 GPUs are packaged together，I guess that might lead to the problem.
I bought a SuperMicro 4-Module OAM board，and I am trying to connect it to my computer via 2 PCIE x16 slots.

Well,  I deleted and registered the account. Hope to figure out how to use this card together :)

nice move

I believe flashing the BIOS won't solve the issue since there is no difference between the HPE BIOS and official one. Changing the carrier board may solve it, but I don't think the Nvidia carrier has an nonfunctional OAM connector, maybe AMD/Nvidia is using a costomized pinout？That's possible when they want their product only availiable for their partners(?

I believe flashing the BIOS won't solve the issue since there is no difference between the HPE BIOS and official one. Changing the carrier board may solve it, but I don't think the Nvidia carrier has an nonfunctional OAM connector, maybe AMD/Nvidia is using a costomized pinout？That's possible when they want their product only availiable for their partners(?

I believe that MI250X should appear as two descrete GPUs in the OS.
Though many related pictures of MI250Xs have been shown in the Idlefish market in China, I am conviced that the owners of the module all used NV carriers since no custom OAM adapters designed by single individual have been proved effective.
(btw, one of my friends is working on OAM-PCIe cards currently, but he is waiting for MI300X and won't start pre-production in the coming 8 months.)

I believe flashing the BIOS won't solve the issue since there is no difference between the HPE BIOS and official one. Changing the carrier board may solve it, but I don't think the Nvidia carrier has an nonfunctional OAM connector, maybe AMD/Nvidia is using a costomized pinout？That's possible when they want their product only availiable for their partners(?

Actually, those who used NV boards just activated GCD0 of MI250X,  but the card requires two connectors' clock signal. I am working on the Supermicro board. Lack of money : (

I saw someone using MI250x on Windows with Radeon Software(Pro?), it's a single GPU(
By the way, 原来你也……
获取Outlook for Androidhttps://aka.ms/AAb9ysg

From: Lunae Radiance @.>
Sent: Tuesday, March 19, 2024 7:58:47 PM
To: ROCm/ROCm @.>
Cc: FeatherXavier @.>; Comment @.>
Subject: Re: [ROCm/ROCm] Can you provide and fix the Amd instinct mi250x driver. (Issue #2852)
I believe flashing the BIOS won't solve the issue since there is no difference between the HPE BIOS and official one. Changing the carrier board may solve it, but I don't think the Nvidia carrier has an nonfunctional OAM connector, maybe AMD/Nvidia is using a costomized pinout？That's possible when they want their product only availiable for their partners(?
Actually, those who used NV boards just activated GCD0 of MI250X, but the card requires two connectors' clock signal. I am working on the Supermicro board. Lack of money : (
―
Reply to this email directly, view it on GitHubhttps://github.com/ROCm/ROCm/issues/2852#issuecomment-2006993179, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ASSDCVCIZKC2CIBZRBZTCPLYZAR7PAVCNFSM6AAAAABCQZRPVCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBWHE4TGMJXHE.
You are receiving this because you commented.Message ID: @.***>

I saw someone using MI250x on Windows with Radeon Software(Pro?), it's a single GPU(
By the way, 原来你也……
获取Outlook for Androidhttps://aka.ms/AAb9ysg

From: Lunae Radiance @.>
Sent: Tuesday, March 19, 2024 7:58:47 PM
To: ROCm/ROCm @.>
Cc: FeatherXavier @.>; Comment @.>
Subject: Re: [ROCm/ROCm] Can you provide and fix the Amd instinct mi250x driver. (Issue #2852)
I believe flashing the BIOS won't solve the issue since there is no difference between the HPE BIOS and official one. Changing the carrier board may solve it, but I don't think the Nvidia carrier has an nonfunctional OAM connector, maybe AMD/Nvidia is using a costomized pinout？That's possible when they want their product only availiable for their partners(?
Actually, those who used NV boards just activated GCD0 of MI250X, but the card requires two connectors' clock signal. I am working on the Supermicro board. Lack of money : (
―
Reply to this email directly, view it on GitHubhttps://github.com/ROCm/ROCm/issues/2852#issuecomment-2006993179, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ASSDCVCIZKC2CIBZRBZTCPLYZAR7PAVCNFSM6AAAAABCQZRPVCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBWHE4TGMJXHE.
You are receiving this because you commented.Message ID: @.***>

折腾这卡的全是国人  一共七张的样子

I saw someone using MI250x on Windows with Radeon Software(Pro?), it's a single GPU(
By the way, 原来你也……

经典国人外网英文聊天
实际上折腾这个卡的都是国人，一共好像7张卡来着

I saw someone using MI250x on Windows with Radeon Software(Pro?), it's a single GPU(
By the way, 原来你也……

Hey guy, stop posting our intelligence at Idlefish okay? Please delete your comments about NV boards under the MI250X page at Idlefish.
Let's keep it here on github. It's not always a good choice to share intelligence of a rare object at a marketplace full of... people who know little but own a crazy amount of money.

It seemed that Supermicro board requires PEX PCIe switches. The original chip is Broadcom PEX88096. I have no idea whether it has to be Gen4 switch chip, but according to the previous owner of the MI250X, he failed to boot the card by linking the board directly to PCIe slots.

It seemed that Supermicro board requires PEX PCIe switches. The original chip is Broadcom PEX88096. I have no idea whether it has to be Gen4 switch chip. Hope PEX8747 will do the work.
According to the previous owner of the MI250X, he failed to boot the card by linking the board directly to PCIe slots.
If I have to get a Gen4 switch, that would cost too much. I'll consider sell the GPUs.

Not really. PEX chips are just for retiming and expanding PCIe slots. You can make a converter board with 1 PCIe x16 and two MCIO since there's indeed two reference clock. There's an even easier way: just use PCIe bifurcate cards. No documents show that you need a fully connected PCIe.
获取Outlook for Androidhttps://aka.ms/AAb9ysg

From: Lunae Radiance @.>
Sent: Monday, April 1, 2024 10:57:06 PM
To: ROCm/ROCm @.>
Cc: FeatherXavier @.>; Comment @.>
Subject: Re: [ROCm/ROCm] Can you provide and fix the Amd instinct mi250x driver. (Issue #2852)
It seemed that Supermicro board requires PEX PCIe switches. The original chip is Broadcom PEX88096. I have no idea whether it has to be Gen4 switch chip. Hope PEX8747 will do the work.
According to the previous owner of the MI250X, he failed to boot the card by linking the board directly to PCIe slots.
If I have to get a Gen4 switch, that would cost too much. I'll consider sell the GPUs.
―
Reply to this email directly, view it on GitHubhttps://github.com/ROCm/ROCm/issues/2852#issuecomment-2029891868, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ASSDCVAT6FAXFJSMXB2WS7LY3FYUFAVCNFSM6AAAAABCQZRPVCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRZHA4TCOBWHA.
You are receiving this because you commented.Message ID: @.***>

Not really. PEX chips are just for retiming and expanding PCIe slots. You can make a converter board with 1 PCIe x16 and two MCIO since there's indeed two reference clock. There's an even easier way: just use PCIe bifurcate cards. No documents show that you need a fully connected PCIe.

What you said is actually what have been done before. But nothing happened.
This might be a cable issue, but the original solution is that each MI250X is connected to a discrete PEX88096.
Don't you think there are more switches than actual needs?
My friend also holds the belief that it is the board that requires PEX.

I also need to note that even with no PEXs, the PCIe lanes are enough for 4 MI250Xs on the MB H12DGQ with 2 EPYCs.

Something in PCIe 4.0+ specification shows you need a better pcb material to make sure signals are not interfered, or a retimer to use on FR-4 boards. I believe that's the purpose of those switches. There are some cheaper retimer chips made in China, and they can also solve the issue of 100MHz clock.
By the way, that 156MHz clock is not used by MI250x. MI300 uses this signal.
获取Outlook for Androidhttps://aka.ms/AAb9ysg

From: Lunae Radiance @.>
Sent: Tuesday, April 2, 2024 11:10:22 PM
To: ROCm/ROCm @.>
Cc: FeatherXavier @.>; Comment @.>
Subject: Re: [ROCm/ROCm] Can you provide and fix the Amd instinct mi250x driver. (Issue #2852)
I also need to note that even with no PEXs, the PCIe lanes are enough for 4 MI250Xs on the MB H12DGQ with 2 EPYCs.
So what's the meaning of the extra 4 switches?
―
Reply to this email directly, view it on GitHubhttps://github.com/ROCm/ROCm/issues/2852#issuecomment-2032316875, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ASSDCVH4S4OHWAOSKDJUBN3Y3LC55AVCNFSM6AAAAABCQZRPVCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZSGMYTMOBXGU.
You are receiving this because you commented.Message ID: @.***>

Sad news from AMD: all MI250X you see (with ES on their cores) are not able to use because there are hardware defects in their cores. What is interesting is that Supermicro TNMI systems are using a different sku than Frontier systems. I think we can end this topic now, and wait for MI300X(
Anyway there's one last try, flash a MI250/210 bios and try to pass through bugs. If this doesn't work, you can go to bed and sleep.
获取Outlook for Androidhttps://aka.ms/AAb9ysg

From: Lunae Radiance @.>
Sent: Tuesday, April 2, 2024 11:10:22 PM
To: ROCm/ROCm @.>
Cc: FeatherXavier @.>; Comment @.>
Subject: Re: [ROCm/ROCm] Can you provide and fix the Amd instinct mi250x driver. (Issue #2852)
I also need to note that even with no PEXs, the PCIe lanes are enough for 4 MI250Xs on the MB H12DGQ with 2 EPYCs.
So what's the meaning of the extra 4 switches?
―
Reply to this email directly, view it on GitHubhttps://github.com/ROCm/ROCm/issues/2852#issuecomment-2032316875, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ASSDCVH4S4OHWAOSKDJUBN3Y3LC55AVCNFSM6AAAAABCQZRPVCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZSGMYTMOBXGU.
You are receiving this because you commented.Message ID: @.***>

However it is true that the MI250X comes from Frontier. Someone's card detected the whole memory size of the Frontier system (though that's not a good sign)
If all MI250Xs cannot be used，then what is running on Frontier?

I mean MI250X on Idlefish. Why would they replace GPUs that are working well?
And since Frontier is using a different sku than TNMI, I don't really think your gpu would work on a supermicro ubb.

From: Lunae Radiance @.>
Sent: Thursday, April 4, 2024 6:29:39 PM
To: ROCm/ROCm @.>
Cc: FeatherXavier @.>; Comment @.>
Subject: Re: [ROCm/ROCm] Can you provide and fix the Amd instinct mi250x driver. (Issue #2852)
However it is true that the MI250X comes from Frontier. Someone's card detected the whole memory size of the Frontier system (though that's not a good sign)
If all MI250Xs cannot be used，then what is running on Frontier?
―
Reply to this email directly, view it on GitHubhttps://github.com/ROCm/ROCm/issues/2852#issuecomment-2036795196, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ASSDCVFCSAC5F42YO4X76ILY3UTRHAVCNFSM6AAAAABCQZRPVCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZWG44TKMJZGY.
You are receiving this because you commented.Message ID: @.***>

What AMD also told us is that the released version of MI250X also comes with ES signs on the GPU.

In that case I might keep it as a decoration. Thanks anyway.

Idk, but I do think those MI250X are not something easy enough to use. And maybe for some reasons, AMD is trying to prevent us from lighting it up.
I got some information for PEX chips on Supermicro boards: they are definitely not necessary, just for retiming. PCIe retimer chips with bifurcation is the cheapest solution I can try, but I still haven't got my GPU. The price is getting high rapidly(

From: Lunae Radiance @.>
Sent: Thursday, April 4, 2024 6:41:03 PM
To: ROCm/ROCm @.>
Cc: FeatherXavier @.>; Comment @.>
Subject: Re: [ROCm/ROCm] Can you provide and fix the Amd instinct mi250x driver. (Issue #2852)
What AMD also told us is that the released version of MI250X also comes with ES signs on the GPU.
―
Reply to this email directly, view it on GitHubhttps://github.com/ROCm/ROCm/issues/2852#issuecomment-2036816601, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ASSDCVG6O3RBDS3G7LQ24PDY3UU37AVCNFSM6AAAAABCQZRPVCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZWHAYTMNRQGE.
You are receiving this because you commented.Message ID: @.***>

Hey, look what have I found...

From: Lunae Radiance @.>
Sent: Thursday, April 4, 2024 6:50:32 PM
To: ROCm/ROCm @.>
Cc: FeatherXavier @.>; Comment @.>
Subject: Re: [ROCm/ROCm] Can you provide and fix the Amd instinct mi250x driver. (Issue #2852)
In that case I might keep it as a decoration. Thanks anyway.
―
Reply to this email directly, view it on GitHubhttps://github.com/ROCm/ROCm/issues/2852#issuecomment-2036833164, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ASSDCVA4Z7JO7366HSOB5GLY3UV7RAVCNFSM6AAAAABCQZRPVCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZWHAZTGMJWGQ.
You are receiving this because you commented.Message ID: @.***>

Hey, look what I have found...

Not really. I've got the firmware, MI250X BIOS and Cary os for Frontier EX235A, there's nothing suspicious except for vram size in vbios. You can change it in RBE and flash it on (via programmer).
Unfortunately I can't do this cuz I still have no GPU, just can't afford it(
But I believe when MI300Xs come to town, there'll be the same issue.

From: yanbosmu @.>
Sent: Friday, April 5, 2024 11:19:03 PM
To: ROCm/ROCm @.>
Cc: FeatherXavier @.>; Comment @.>
Subject: Re: [ROCm/ROCm] Can you provide and fix the Amd instinct mi250x driver. (Issue #2852)
I don't think this motherboard will make this GPU function well.

―
Reply to this email directly, view it on GitHubhttps://github.com/ROCm/ROCm/issues/2852#issuecomment-2040070195, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ASSDCVGQU6VDWOYR22G4YNTY326GPAVCNFSM6AAAAABCQZRPVCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBQGA3TAMJZGU.
You are receiving this because you commented.Message ID: @.***>

Not really. I've got the firmware, MI250X BIOS and Cary os for Frontier EX235A, there's nothing suspicious except for vram size in vbios. You can change it in RBE and flash it on (via programmer).
Unfortunately I can't do this cuz I still have no GPU, just can't afford it(
But I believe when MI300Xs come to town, there'll be the same issue.

From: yanbosmu @.***>
Sent: Friday, April 5, 2024 11:19:03 PM
To: ROCm/ROCm @.***>
Cc: FeatherXavier @.>; Comment @.>
Subject: Re: [ROCm/ROCm] Can you provide and fix the Amd instinct mi250x driver. (Issue #2852)
I don't think this motherboard will make this GPU function well.

―
Reply to this email directly, view it on GitHubhttps://github.com/ROCm/ROCm/issues/2852#issuecomment-2040070195, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ASSDCVGQU6VDWOYR22G4YNTY326GPAVCNFSM6AAAAABCQZRPVCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBQGA3TAMJZGU.
You are receiving this because you commented.Message ID: @.***>

Bios Is on their website.  And I bought a hardware for flashing but I still haven't got any other BIOS files except this d65201.

Other BIOSes are on AMD's software repo.
https://repo.radeon.com/fwupdater/amdfwflash/latest/deb/pool/main/a/amdfwflash/amdfwflash_<IP_ADDRESS>_amd64.deb
They are located in a subfolder, including BIOSes for MI210 and MI250.

发件人: yanbosmu @.>
发送时间: 2024年4月6日 下午 12:32
收件人: ROCm/ROCm @.>
抄送: FeatherXavier @.>; Comment @.>
主题: Re: [ROCm/ROCm] Can you provide and fix the Amd instinct mi250x driver. (Issue #2852)
Not really. I've got the firmware, MI250X BIOS and Cary os for Frontier EX235A, there's nothing suspicious except for vram size in vbios. You can change it in RBE and flash it on (via programmer).
Unfortunately I can't do this cuz I still have no GPU, just can't afford it(
But I believe when MI300Xs come to town, there'll be the same issue.

From: yanbosmu @.***>
Sent: Friday, April 5, 2024 11:19:03 PM
To: ROCm/ROCm @.***>
Cc: FeatherXavier @.>; Comment @.>
Subject: Re: [ROCm/ROCm] Can you provide and fix the Amd instinct mi250x driver. (Issue #2852https://github.com/ROCm/ROCm/issues/2852)
I don't think this motherboard will make this GPU function well.

D
Reply to this email directly, view it on GitHub#2852 (comment)https://github.com/ROCm/ROCm/issues/2852#issuecomment-2040070195, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ASSDCVGQU6VDWOYR22G4YNTY326GPAVCNFSM6AAAAABCQZRPVCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBQGA3TAMJZGU.
You are receiving this because you commented.Message ID: @.***>
Bios Is on their website.  And I bought a hardware for flashing but I still haven't got any other BIOS files except this d65201.
―
Reply to this email directly, view it on GitHubhttps://github.com/ROCm/ROCm/issues/2852#issuecomment-2040962128, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ASSDCVGGY5DMXOVSISIPF6LY353GBAVCNFSM6AAAAABCQZRPVCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBQHE3DEMJSHA.
You are receiving this because you commented.Message ID: @.***>

https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=dp00004178en_us
Have you guys checked the HPE support document for EX systems?
I think the vbios and driver should be available in COS and HPC firmware packs.

Haven't tried vbios for VMware os. I installed VMware Os already. But I don't know how to use this vbios. And if it still needs the identical HPE Cray os, There Is now way we can make it

umm cray os and hfp are open for downloading on their support site. But idk if we're able to install cos on non-hpe setups.

umm cray os and hfp are open for downloading on their support site. But idk if we're able to install cos on non-hpe setups.

Lol.  You need contract to download.

Here comes the BIOS for D65201. It seems that it can be flashed through amdvbflash. But I believe you need to modify it.
For reference: https://www.edvisonblog.cn/2023/06/19/vbios%e5%8e%9f%e7%90%86%e5%8f%8a%e4%bf%ae%e6%94%b9%e6%96%b9%e5%bc%8f/
image.zip

I happened to have found the firmware pack too. Is this bin file extracted from the rpm package inside HFP?
Also there is another image.bin inside the MI200RM package and I wonder what it is.

Here comes the BIOS for D65201. It seems that it can be flashed through amdvbflash. But I believe you need to modify it.
For reference: https://www.edvisonblog.cn/2023/06/19/vbios%e5%8e%9f%e7%90%86%e5%8f%8a%e4%bf%ae%e6%94%b9%e6%96%b9%e5%bc%8f/
image.zip

Flashing it directly through official amdvbflash failed. Maybe this rom need modified. But That Is beyond My capability now

米青采彡

米青采彡
意料之中 都是国人
干脆开Q群讨论算了()

点击链接加入群聊【ROCm/issues/2852】：https://qm.qq.com/q/DMnWBpCqFG

@yanbosmu Since this appears to be a board issue with the vbios, can you please close this ticket? If needed, please open another ticket under discussion. Thanks!


Add checksum validation for SCP file transfer on Windows Testbed

Fixes #5653
This enhancement improves the reliability of the file transfer process, reducing the risk of importing corrupted or incomplete images.

/test-windows-containerd-e2e

/test-rancher-all

We will observe whether the same issue persists after PR #6325 is merged when using the Linux testbed to build Windows images. If there are no issues, we will close this PR.

Close this PR as there have been no issues after the testbed migration.


SCD30 forced calibration within MyAmbiance?

I really like the ability to do calibration of the SCD40/41 within the MyAmbiance app.  Of course I want to be able to do it with the SCD30 too!  Is that possible?  If not, do you know of an Arduino sketch that does forced calibration?  I've been having trouble using what limited stuff is out there. I'm not clever enough to do it from the data sheet. Thank you.

Hi,
our Arduino driver implements the force calibration for SCD30 (see: https://github.com/Sensirion/arduino-i2c-scd30/blob/master/src/SensirionI2cScd30.h#L217)
However we don't have a ready to use example for it yet. To do a force re-calibration you have to run the SCD30 for at lease 2 minutes in a stable environment, before setting the reference value of this environment.
In MyAmbience we currently have no plan to  to support force re-calibration of SCD30 sensors.
Let me know if you need further support.