Suggestion: should non-null assert propagate?

I was wondering whether ! should be taken into account by the dataflow analysis, and I don't see why it shouldn't.
It is like a cast that says "this value is not null", so from this point I guess the compiler could remove undefined/null from future uses.
For example, I use ! to work around #9631, so I have code that looks like (simplified):
protected dataArrayChanged(changes: ChangeRecord[]) {
  // #9631: TS incorrectly infers `change: ChangeRecord | undefined`
  for (const change of changes) {
    for (let i = 0; i < change!.removed.length; i++)
      this.dt.row(i).remove();
    if (change!.addedCount > 0)
      this.dt.rows.add(this.data.slice(change!.index, change!.index + change!.addedCount));
  }
  this.dt.draw();
}

Observe how I added 5 ! to make change not null.
The first one could have been enough. After all, once I say "change is not undefined", there is no reason to assume it could be until I modify the variable again.
// Let's say I know x is not undefined
const x: number | undefined;
// Here x: number | undefined, so x! is required
x!.toString();
// Here x: number because of x! above
x.toString(); // ok

I'm not in favor of this because of the legibility of the assertion. Unlike the leading ! in a if (!change) the ! in for (let i = 0; i < change!.removed.length; i++) is almost entirely invisible, which would make discovering such mutations to the type much harder for human beings.
Why not simply add a check for undefined to strip it out instead?
for (const change of changes) {
  // TODO: Remove when typescript#9631 is resolved
  if (change === undefined) continue;
  // change is now *just* ChangeRecord unless I'm missing something
}

@svieira Sure, it would work. But it kind of defeats the purpose of !, doesn't it?
Also, ! is a cast that generates no code. Your code has no effect in practice but will be emitted to JS.
BTW what is the scenario you have in mind here? I don't think adding a ! will ever create errors downstream, as the compiler doesn't warn if you do things like if (a === null) when a: number.
So I can't imagine a case where you would be hunting that extra ! that creates errors in your code.
If your issue is rather that you find ! unreadable in itself, then that's a different problem (probably #9637) -- and you don't have to use it. I believe that for those of us who use ! occasionnally, narrowing the variable type is a logical thing to do.

Where I don't follow you is here:
const anythingElse = value!.some.property.access;
// value is now stripped of undefined even if that wasn't the intent

If there should have been a check and in the future value suddently becomes unexpectedly null then your code crash right at value!.some. At this point I'm not sure if it matters what happens with the code after it. Buggy code is buggy.

I agree the example is not compelling as written. But consider something very similar:
// in external-lib.d.ts
 // File written pre-strictNullCheck; in reality accepts string | undefined
declare function doSomething(x: string): void;

// in my-file.ts
function foo(x: string | undefined): string {
  // Use ! to silence error about function call that I know is actually valid
  doSomething(x!);
  return x;
}

let y = foo(undefined); // oops, y: string but has value 'undefined'

@RyanCavanaugh I see. This is an usage of  ! that I never considered. You are not marking x as not null, you are hacking around an incorrect definition.
When I have an incomplete .d.ts  (which of course happens frequently at this point) I fix it. I have a feeling that starting to hack calls with ! is a very slippery slope and I won't be happy with a code base like this one year from now. Is this a scenario that you really want to support? In this case my suggestion is not playing nice, indeed.
Nitpicker's corner: your example is broken because you explicitly say foo(): string, which would be an error from the coder rather than the compiler. I guess you meant to have an implicitly typed return value.

Re: Nitpick - correct, I'm saying that under the proposed behavior, there wouldn't be an error, which is bad.
I'm not sure what you're getting at with the first 2 paragraphs there. Is there some way to meaningfully distinguish the 'bad' cases from the 'good' cases?

@RyanCavanaugh What I meant is this:
I think of ! as a special cast that removes the undefined/null types.
In other words when I see this code: doSomething(x!) I think: x is nullable but it is guaranteed to be not null here for some reason.
When in fact your usage is different. x could very well be null here, but you want to satisfy the typechecker that only let you pass a non-null value to the function, because of an incorrect definition.
This is really the same as this situation:
// bad .d.ts, maybe because it's for an old version of a library that was not updated yet
// Actually, frob now also accepts an array: `x: number | number[]`
function frob(x: number): void;

// So in your code you do:
let some = [1, 2, 3];
frob(<number>some);

I would not consider this good style, and to me your ! example is the same thing. You lie about the real type of x to pass a bad definition.

Is there some way to meaningfully distinguish the 'bad' cases from the 'good' cases?

Well yes: if at runtime x! is guaranteed to be non null that looks like a good case. If it could actually be null I think you're doing something fishy.

@jods4
Precisely because ! is a cast that removed the undefined/null types, it shouldn't behave differently than any other cast.
What you're saying is not necessarily 'I know it is not undefined' but more 'I want to treat it as not undefined in this expression'.
In my opinion, this suggestion would be the same as accepting something like this
function foo(): number | number[] { return 3; }
function frob(x: number): void {}

let some = foo(); //some is of type 'number | number[]'
frob(some as number); //we might know it's a number, so we do this. Okay.
frob(some); //If casts "teach about" types, this would not be an error. Currently, it is.

Another problem would be that, when closures are involved, a variable being 'not undefined' in one line doesn't necessarily mean that it will still be 'not undefined' in the next line.
However, this is a problem of type assertions in general. For example, the following legal code throws because typescript can't detect that a is undefined, and it can't really be expected to.
(function (){
    var a: number | undefined = 3;
    var b = () => { a = undefined; };

    if(a) {
        b();
        console.log(a.toFixed());
    }
})();

@buola this actually crossed my mind when I wrote this suggestion. I did not want to push it too far but I think that it would be perfectly logical that a cast narrows the actual type of a union when it's used, yes.
I would certainly accept your example. Think about it: either the call frob(<number>some) is a coding error and can fail at runtime, or there is really no reason why the call on the next line frob(some) wouldn't work. So I say it would make perfect sense for the compiler to allow it.
Casting means "hey compiler I know better than you what type this variable really is in this location". Then why shouldn't the compiler learn from our hints?
As you noted, your second problem is unrelated to this suggestion. It is a limitation of the type analysis as it exists today.

Related #10421
It's not at all clear what the "scope of effect" of the assertion should be. For example let x = y ? a!.foo : b; -- is a still non-null on the next line? Maybe y was a discriminator for a's nullness, maybe it wasn't. Unless we had a very clear understanding of what made an assertion appear in the first place, it's not going to be good for us to guess what the scope of the assertion should be. And if we did understand, arguably we should make the compiler understand what introduced the assertion in the first place.
It's not clear there's a good answer here. Sort of unrelatedly, the OP was caused by a compiler bug (for / in should not introduce undefined into the iterator type), so that much is at least solved already. And there's always the const xx = x!; workaround.

ok maybe I didn't think this as thoroughly as you guys, but couldn't it use the same rules as the data flow analysis that's already in place?
In your example a!.foo is in a branch and there is no way to extrapolate that it is also valid on the main code path. As you said, y could very well have been key to the non-null aspect.
This is pretty much the same thing as an if: type guards inside the "then" block do not propagate outside of it.
BTW although the idea still makes sense to me, I have been using null-strict code for a while and lack of this hasn't really been an issue. Mostly because 99% if the time my code is correctly typed. So maybe it's not worth pursuing if you feel the benefits are not worth the effort.

I think the confusing part is that you might write code like this
function fn(x; string | undefined) {
  /* do something here that makes x necessarily not-null that the compiler can't detect */

  // lots more lines
  console.log(x!);
  // lots more lines
  return x.substr(3); // OK
}

Then you refactor a bit
function fn(x; string | undefined) {
  /* do something here that makes x necessarily not-null that the compiler can't detect */

  if (enableLogging) {
    // lots more lines
    console.log(x!);
    // lots more lines
  }
  return x.substr(3); // Error!
}

and it's really not clear why you're getting an error on the last line now, especially if the console.log(x!) had been some more complex expression where it was easy to miss that it was x! instead of x

I agree all the inferred stuff can sometimes be a little obscure, this is already the case today. Especially when you encounter shortcomings of the typing engine.
In your example, yes there is a new error. There is a good reason for that, even though I agree it can be a little obscure. Reviewing the changes (hopefully people use version control), I think it is reasonably clear that moving a type assertion (the !) into a branch might cause that. And the fix is trivial: if it's still true that x is not null, add a !.
If there are so many lines that you can't see what you're doing, it's a different problem. Overly long functions are a code smell.
But again I'm not particularly concerned by this anymore. If you do something in this area, I hope it's something more general, like narrowing types in a block for casts as well, etc.

Just adding a use case. This would be nice for writing real assertions:
class Foo {
   prop: string | null;

   propLength() {
       assert(this.prop!);
       return this.prop.length;
  }
}

function assert(value: any) {
    if (!value) {
        throw new AssertError("Null or undefined");
    }
}

This way, if I use ! only inside asserts calls I make sure that it has been properly asserted.

@pmoleri TypeScript 3.7 introduced assertion functions exactly for this purpose, and they have clearer syntax and no need for redundant !, while your code suffers from huge in-readability because you use the non-null assertion where the operand can be null, and even expected to.


x/image: riff: missing RIFF chunk header

What version of Go are you using (go version)?
1.11
Does this issue reproduce with the latest release?
Yes.
What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/acim/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/acim/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/acim/Projects/shit/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build158015811=/tmp/go-build -gno-record-gcc-switches"

What did you do?
	file, err := os.Open("1.webp")
	if err != nil {
		panic(err)
	}
	defer file.Close()

	image, err := webp.Decode(file)
	if err != nil {
		panic(err)
	}
       ...

What did you expect to see?
Image gets decoded.
What did you see instead?
Panic from  webp.Decod:
riff: missing RIFF chunk header
I tried several webp images at this happens with all of them. However, webp.DecodeConfig works well with the same files.

CC @nigeltao

Can you attach (or give a link to) an image file that triggers the issue?

Here is one.

https://files.fm/u/9gqueavn

My bad, had to open file again because io.Reader has been emptied by DecodeConfig. Thank you, you can close this.


Maven central?

I noticed http://mvnrepository.com/artifact/sh.tak.appbundler/appbundle-maven-plugin does not have the latest version of the plugin?

Oh. It's my mistake. Thanks for reporting.
I'll fix soon.

Deployed. https://oss.sonatype.org/content/repositories/releases/sh/tak/appbundler/appbundle-maven-plugin/1.0.2/
It will be indexed into maven search soon.


Implement delete verb

Before I can implement automatic removal of old backups on a destination, when that destination is running low on available space, I need to implement the ability to delete specific prior backups. macOS's tmutil supports this with the delete verb, which is described in tmutil(8) thusly:
delete [-d backup_mount_point -t timestamp] [-p path]
             Deletes the backups with the specified timestamp from the backup volume mounted at the
             specified mountpoint. The -t option followed by a timestamp can be used multiple times to
             specify multiple backups to delete. For HFS backup disks, a specific path to delete can
             also be specified using the -p option. This verb can delete items from backups that were
             not made by, or are not claimed by, the current machine. Requires root and Full Disk
             Access privileges.

For the tempus-machina implementation, I'm pretty much mimicking the HFS implementation using rsync, so tmutil delete -p <path> would be the primary way I'd use this. That said, tmutil delete -d <destination_mount> -t <timestamp> use should be able to just assemble the backup path and internally handle it as if it were given tmutil delete -p <path>.

I've already started some of the backup deletion implementation as part of issue #2 (deleteinprogress verb).

I've now implemented the delete verb using the existing delete_backup() implementation (see issue #2, deleteinprogress verb) and updated the README.

Now merged into main.


Show triggered alert name in notification text

Hi!
we have several alert rules that trigger telegram notifications.
Is it possible to somehow inject triggered rule name in notification text?
Otherwise it's not clear is it first-time error or threshold error.
Thanks!

I don't think it's possible in current version.
Take a look at the source - https://github.com/butorov/sentry-telegram/blob/master/sentry_telegram/plugin.py#L102
You could probably add this ability easily, althrough it seems the plugin has been abandoned for quite some time now.


Replace libjepg with libjpeg-turbo submodule

https://github.com/libjpeg-turbo/libjpeg-turbo

Note that libjpeg-turbo does not not support sub-project builds, which may be a dealbreaker


file reloading FF #jscad-web-v1

Fix file reloading on firefox.
Using lastModified is not really a valid assumption per the spec. It seems to work on chrome, but I would really like to make that reliable.
It probably means we need to re-read all the files at each check, and compare with the contents in our virtual fs. Not ideal, but it's how the existing jscad works, and seems fine even if its inefficient for large files.

Possibly we do not ned to read file contents but just requires the File again and then read lastModified.

https://stackoverflow.com/questions/54627538/automatically-upload-local-file-even-if-its-changed-in-firefox
looks like a dead end for now


feat: lite version of transaction information

Description
PRD: https://vs0cjf.axshare.com/?id=3giphq&p=_189_how_to_support_customers_and_developers_opera&g=1
Desktop design: https://www.figma.com/file/6XNoimRDbFTTNm016rbIdU/Magickbase?node-id=13042%3A37425&t=lriivPlPTOL3A8WN-1
Mobile design: https://www.figma.com/file/6XNoimRDbFTTNm016rbIdU/Magickbase?mode=dev
Resolve: https://github.com/Magickbase/ckb-explorer-public-issues/issues/189
Backend PR: https://github.com/nervosnetwork/ckb-explorer/pull/1384
Preview example: https://ckb-explorer-git-transaction-lite-magickbase.vercel.app/transaction/0x21b9d887b8a68bf56f1ae6bc7cd28d79395efd2241de3de57e6fdfd36714b2ee?layout=lite
Notes:

the original PR is #5, but closed for work handover
cell tags are not implemented yet (eg: "mint", "nft transfer" tags)

Please have a review @Keith-CY , The comments here are mostly on legacy codes, some of them are not very maintainable, I just refactored some for this PR. What if we remove styled and BEMs? just by using SCSS could simplify the style sheets, I can gradually do the refactor in future development

Please have a review @Keith-CY , The comments here are mostly on legacy codes, some of them are not very maintainable, I just refactored some for this PR. What if we remove styled and BEMs? just by using SCSS could simplify the style sheets, I can gradually do the refactor in future development

Sure, removing styled-component is one of our goals, so do BEM

Please have a review @FrederLu @silySuper
And please resolve the conflicts @zhangyouxin

1、Information such as CKB is missing under the address on the page.
URL  https://ckb-explorer-4rek3x5hw-magickbase.vercel.app/transaction/0xa60274bbf76d6f9d5c507335a09ba7a5044bef9cea170578d64f6a03ae3227f3
UI   https://www.figma.com/file/6XNoimRDbFTTNm016rbIdU/Magickbase?type=design&node-id=13058-39866&mode=design&t=3yAFERidb61M1zrX-0

2、Nervos DAO action tags are missing in Lite. At the same time, there must be a prompt message when the mouse hovers.
URL https://ckb-explorer-4rek3x5hw-magickbase.vercel.app/transaction/0x46e8c6fd3e8326b25d507b146cb715ecbf63c5933d575c1ed8e701a29a3ff4ba

3、When Nervos DAO Deposit/Nervos DAO Withdraw appears in Professional for the same transaction and the same address, you can click the icon on the list page to view the display data to determine whether Deposit/Withdraw is displayed.
URL https://ckb-explorer-4rek3x5hw-magickbase.vercel.app/transaction/0x63a9653f593f243dfddee8075877d1f6e074946a39d9a720b2425fd05290d783
https://github.com/Magickbase/ckb-explorer-frontend/assets/25676576/920a528d-2ce1-4314-a407-c48814da9499

Any update for the feedback at https://github.com/Magickbase/ckb-explorer-frontend/pull/25#issuecomment-1707940472

How is it going

Some of the issues have been addressed, but the whole thing is still being worked on.
@zhangyouxin You can synchronize some information.

For the comments:
1,3,4. the design is being adjusted @Sven-TBD
2. added
5. the api/v2/das_accounts api for address ckt1qqexmutxu0c2jq9q4msy8cc6fh4q7q02xvr7dc347zw3ks3qka0m6qgres90ptu3rh2qs5acer07ay9n97x3ajkkq0xq4u90jywagzznhrydlm5skvhc68k26cucdqcn on staging env didn’t bind an alias @rabbitz
6. fixed
7. fixed

For the comments:
1,3,4. the design is being adjusted @Sven-TBD 2. added 5. the api/v2/das_accounts api for address ckt1qqexmutxu0c2jq9q4msy8cc6fh4q7q02xvr7dc347zw3ks3qka0m6qgres90ptu3rh2qs5acer07ay9n97x3ajkkq0xq4u90jywagzznhrydlm5skvhc68k26cucdqcn on staging env didn’t bind an alias @rabbitz 6. fixed 7. fixed

6, and 7 have been verified, others need to be adjusted.
The label at point 2 has been added, but the size is too large, affecting the font size later.

The UI link has been updated, still a few points should be aware of:

For each different Asset type, there will be a line.
Only for DAO-related transaction, a tag prompt is required

For the comments:
1,3,4. the design is being adjusted @Sven-TBD 2. added 5. the api/v2/das_accounts api for address ckt1qqexmutxu0c2jq9q4msy8cc6fh4q7q02xvr7dc347zw3ks3qka0m6qgres90ptu3rh2qs5acer07ay9n97x3ajkkq0xq4u90jywagzznhrydlm5skvhc68k26cucdqcn on staging env didn’t bind an alias @rabbitz 6. fixed 7. fixed

root@ckb-explorer-api-6ccbcb8d9d-zgx76:/usr/src# curl -X POST http://das-account-indexer:8122/v1/reverse/record -d'{"type": "blockchain","key_info":{"coin_type": "", "chain_id": "1", "key": "0xcc0af0af911dd40853b8c8dfee90b32f8d1ecad6"}}'
{"err_no":0,"err_msg":"","data":{"account":"","account_alias":""}}

There are no available domains for this address (testnet).

The UI link has been updated, still a few points should be aware of:

For each different Asset type, there will be a line.
Only for DAO-related transaction, a tag prompt is required

@rabbitz please check this out, there should be some changes for the api

The UI link has been updated, still a few points should be aware of:

For each different Asset type, there will be a line.
Only for DAO-related transaction, a tag prompt is required

@rabbitz please check this out, there should be some changes for the api

@rabbitz  According to the new design, the sUDT and NFT transfers would contain capacity ocupation amount, NFT transfers would show nft id:

@rabbitz According to the new design, the sUDT and NFT transfers would contain capacity ocupation amount, NFT transfers would show nft id:

@Sven-TBD
I saw () in the number part in the screenshot. Where did the value 61CKB come from?

related pr: https://github.com/nervosnetwork/ckb-explorer/pull/1463

1、In UDT transactions, +/— matches and displays abnormally, and the capacity value displays abnormally.
https://ckb-explorer-frontend-in-magickbase-repo-jm1nw532y-magickbase.vercel.app/transaction/0xd8668c851f09559868d4b91340670c4ee9a6dcf50184019cf42dcd8fa39a745b?layout=lite

2、The layout on the mobile terminal needs to be adjusted with reference to the UI.

The page needs to be displayed in real time based on the api data, and the - symbol is currently missing.
https://ckb-explorer-frontend-in-magickbase-repo-12npn9j73-magickbase.vercel.app/transaction/0x6a98ccfa27fe375febd3648ac4bab9984da64de8c36f8268454c5cb6b79f6969?layout=lite

@zhangyouxin
Referring to the interface data, if cell_type: "m_nft_class", CKB needs to be displayed as m-NFT-class on the page.
https://ckb-explorer-frontend-in-magickbase-repo-5q0j3pbmh-magickbase.vercel.app/transaction/0x3752b2a5ecb7a05e1fbb6ed89f2d5df4db991c957975703883daf4739a78d918?layout=lite


Add htsPreCompile get tokenInfo acceptance tests

Description:
Add tests for:

getFungibleTokenInfo
getTokenInfo
getNonFungibleTokenInfo - depends on mintToken
mintToken

Related issue(s):
Fixes #400
Notes for reviewer:
The branch is created from 399-add-htsPrecompile-allowance-approval-tests, some of the new logic was already implemented there.
Checklist

[ ] Documented (Code comments, README, etc.)
[ ] Tested (unit, integration, etc.)

Codecov Report

Merging #450 (0ccd42b) into main (81a0f11) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #450   +/-   ##
=======================================
  Coverage   76.46%   76.46%
=======================================
  Files          12       12
  Lines         922      922
  Branches      143      143
=======================================
  Hits          705      705
  Misses        165      165
  Partials       52       52

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


TypeCastExpression could not be resolved

Hi,
I was just upgrading eslint and all its plugins and now getting this error when trying to lint with the following setup:

"eslint": "4.16.0",
"eslint-config-airbnb": "16.1.0",
"eslint-plugin-flowtype": "2.41.0",
"eslint-plugin-import": "2.8.0",
"eslint-plugin-jsx-a11y": "6.0.3",
"eslint-plugin-mocha": "4.11.0",
"eslint-plugin-react": "7.5.1",
"babel-eslint": "8.2.1",

Full Error Message:
Error: The prop value with an expression type of TypeCastExpression could not be resolved.
  Please file issue to get this fixed immediately.
    at Object.extract [as JSXExpressionContainer] (/Users/mil/Development/wizard/node_modules/jsx-ast-utils/lib/values/expressions/index.js:179:11)
    at getValue (/Users/mil/Development/wizard/node_modules/jsx-ast-utils/lib/values/index.js:48:27)
    at extractValue (/Users/mil/Development/wizard/node_modules/jsx-ast-utils/lib/getPropValue.js:24:12)
    at getPropValue (/Users/mil/Development/wizard/node_modules/jsx-ast-utils/lib/getPropValue.js:41:10)
    at /Users/mil/Development/wizard/node_modules/eslint-plugin-jsx-a11y/lib/rules/no-static-element-interactions.js:78:95
    at Array.some (<anonymous>)
    at JSXOpeningElement (/Users/mil/Development/wizard/node_modules/eslint-plugin-jsx-a11y/lib/rules/no-static-element-interactions.js:77:52)
    at listeners.(anonymous function).forEach.listener (/Users/mil/Development/wizard/node_modules/eslint/lib/util/safe-emitter.js:47:58)
    at Array.forEach (<anonymous>)
    at Object.emit (/Users/mil/Development/wizard/node_modules/eslint/lib/util/safe-emitter.js:47:38)

Any idea?

Found it. I'm adding a test case and getting this fixed.

@7yp0 , Fixed in version v2.2.1. This will log an error in the console, but it will no longer throw.


Gravit ~ https://gravit.io/

Gravit - The versatile Design Tool for Windows, Mac, Chrome OS and the Browser

Hmm, so this is no longer open source but still free, and the site seems to have moved to designer.io

Looks neat but as you mentioned, is no longer OSS.


[Fix] Compile error related to Item.id

Description
ting: See Swift 3906 - it involves a bug in Item.ID's optionality, and the final solution is hard to explain in a few words.

Should be ready to merge. Will create the dummy app after this.


:robot: Arm: Do not compress nor upload artifacts on PR

What this PR does / why we need it:
Makes no sense to bindly compress and upload the arm artifacts on each PR and each PR commit as its a very costly step.
This PR disables compression and artifact upload on the arm pipelines when running on PRs.
Master merges keep the same behaviour as before
This should speed up our PR pipelines.
Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

I guess we can do the same on provider-kairos?

images still too big, will rework and resend :/


docs: update set price function comments

Type of change

Documentation

Changes
The following changes have been made:

Updated set_price documentation comments

Notes
Related Issues

Closes #228

I believe https://github.com/FuelLabs/sway-applications/pull/256 needs to be merged in for CI to pass

Good ol' formatter.
When checks pass DM me to take another look. Note that I've marked that as a GFI label so that someone external can have a step that they can contribute through.


Github Organization

Hi.
I've been exploring the programming language you've created, and I'm really impressed! I have a suggestion.
Instead of managing Gliimly under a personal account, I recommend creating a GitHub organization. An organization offers several advantages over a personal account, such as:
Adding new members: Collaborate with others more easily.
Using Kanban boards: Streamline project management.
Centralizing related repositories: Keep all your project resources in one place.
I'm eager to contribute to this language and would appreciate it if you could migrate the repository to a GitHub organization.

Hi, please see CONTRIBUTING.md for more details on this.


Theme is Broken

When loading the site with this theme, the styling is missing and found in the console that these files are missing from the theme directory.
assets/css/app.css
assets/js/manifest.js
assets/js/vendor.js
assets/js/app.js
assets/js/home.js
assets/js/manifest.js
assets/js/manifest.js
You can see the same in the screenshot below.

Do not clone from github. Just download from this releases url

Please take a look to https://github.com/eddiesigner/liebling#download


Recognize GoogleOther

I consider GoogleOther a bot, maybe you do as well :-)
https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers
Example agent string
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.5993.117 Mobile Safari/537.36 (compatible; GoogleOther)

Sure, but haven't seen that one in the wild :)
I will add support for it in the next iteration.


Video APIs

Provides initial support for the Google Video Intelligence API.

GoogleVideoIntelligenceAPIExtractor is a general extractor that allows users to specify which features(s) they want to extract, contains main logic for extraction and parsing
GoogleVideoAPILabelDetectionExtractor extracts labels present throughout an entire video, user specified segments, or key frames. This extraction sometimes takes a long time to run, especially if combined with another feature via the parent GoogleVideoIntelligenceAPIExtractor class. Use with caution
GoogleVideoAPIShotDetectionExtractor extracts a binary "shot" feature with onsets and durations corresponding to the onsets of shot change detection
GoogleVideoAPIExplicitDetectionExtractor extracts explicit annotations for key frames (ranging from VERY_UNLIKELY to VERY_LIKELY).
Each transformer is accompanied by timeout and request_rate parameters that dictate the speed and duration of polling the Google API service for completion

Also expands our support of the Clarifai API by including the ability to extract on videos. This required making the ClarifaiAPIExtractor class abstract and refactoring the existing class into a ClarifaiAPIImageExtractor and implementing a new ClarifaiAPIVideoExtractor.
Resolves #280
Resolves #277
TODO: look for required updates in docs

Coverage increased (+0.2%) to 94.149% when pulling 3ddf2a06b5a194b943b788aabec13be04ef0db67 on video-apis into 2374bbf9aa702c04f668194ea5635286e095010a on master.

@tyarkoni are you ok with this structure of the tests: run the video extraction with a default timeout of 90 seconds
-If it failed to finish, pass test
-Otherwise, go through asserts
It will be harder to know when the API has changed and/or is failing, but I can't find a great way to make tests that we know will finish quickly.

@tyarkoni ready for final review. I commented on the major issues/things in the PR since it's pretty big


DatabaseBranchPassword

Add resource for DatabaseBranchPassword please. I may take care myself too.

@s1ntaxe770r
s1ntaxe770 would you please merge and cut a release? We are blocked now with this release.

Oof, I'm so sorry hold on.


Bug in CreateTextItem() method

I'm opening this issue for tracking purposes.
The CreateTextItem() method in SqlCrud does not set the Id of the inserted item properly, so the SortPosition is not assigned.
I have found a possible fix, but it appears to require changing a significant portion of my code.

This has been resolved by pull request #6.


[SPARK-18890][CORE](try 2) Move task serialization from the TaskSetManager to the CoarseGrainedSchedulerBackend

What changes were proposed in this pull request?
See https://issues.apache.org/jira/browse/SPARK-18890
In the case of stage has a lot of tasks, this PR can improve the scheduling performance of 15%
The test code:

val rdd = sc.parallelize(0 until 100).repartition(100000)
rdd.localCheckpoint().count()
rdd.sum()
(1 to 10).foreach{ i=>
  val serializeStart = System.currentTimeMillis()
  rdd.sum()
  val serializeFinish = System.currentTimeMillis()
  println(f"Test $i: ${(serializeFinish - serializeStart) / 1000D}%1.2f")
}

and spark-defaults.conf file:
spark.master                                      yarn-client
spark.executor.instances                          20
spark.driver.memory                               64g
spark.executor.memory                             30g
spark.executor.cores                              5
spark.default.parallelism                         100
spark.sql.shuffle.partitions                      100
spark.serializer                                  org.apache.spark.serializer.KryoSerializer
spark.driver.maxResultSize                        0
spark.ui.enabled                                  false
spark.driver.extraJavaOptions                     -XX:+UseG1GC -XX:+UseStringDeduplication -XX:G1HeapRegionSize=16M -XX:MetaspaceSize=512M
spark.executor.extraJavaOptions                   -XX:+UseG1GC -XX:+UseStringDeduplication -XX:G1HeapRegionSize=16M -XX:MetaspaceSize=256M
spark.cleaner.referenceTracking.blocking          true
spark.cleaner.referenceTracking.blocking.shuffle  true

The test results are as follows
The table is out of date, to be updated

SPARK-17931
941b3f9

17.116 s
21.764 s

How was this patch tested?
Existing tests.

Test build #73681 has started for PR 17116 at commit b67bdaf.

Test build #73684 has started for PR 17116 at commit f33b077.

Test build #73681 has finished for PR 17116 at commit b67bdaf.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73681/
Test FAILed.

Test build #73684 has finished for PR 17116 at commit f33b077.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73684/
Test FAILed.

Test build #73692 has started for PR 17116 at commit 1c26e8c.

Test build #73692 has finished for PR 17116 at commit 1c26e8c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73692/
Test FAILed.

Test build #73728 has started for PR 17116 at commit 752eeb7.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73728/
Test FAILed.


Export not found: mono_aot_get_method

Hello,
I can't get the modify_class_function_argument.js script to work.
I'm working with this app: https://github.com/xamarin/xamarin-forms-samples/tree/master/WebServices/TodoREST
I'm trying to intercerpt the GetTasksAsync function of the TodoItemManager:
using System;
using System.Collections.Generic;
using System.Threading.Tasks;

namespace TodoREST
{
	public class TodoItemManager
	{
		IRestService restService;

		public TodoItemManager (IRestService service)
		{
			restService = service;
		}

		public Task<List<TodoItem>> GetTasksAsync ()
		{
			return restService.RefreshDataAsync ();
		}

		public Task SaveTaskAsync (TodoItem item, bool isNewItem = false)
		{
			return restService.SaveTodoItemAsync (item, isNewItem);
		}

		public Task DeleteTaskAsync (TodoItem item)
		{
			return restService.DeleteTodoItemAsync (item.ID);
		}
	}
}

Source for the class: https://github.com/xamarin/xamarin-forms-samples/blob/master/WebServices/TodoREST/TodoREST/Data/TodoItemManager.cs
I'm using this script:
import { MonoApiHelper, MonoApi } from '../vendors/frida-mono-api'
import ClassHelper from '../libraries/class_helper'

// Intercept settings
var settingClassName = "TodoREST.TodoItemManager";
var settingMethodName = "GetTasksAsync";
var settingMethodArgCount = 0;

// The root AppDomain is the initial domain created by the runtime when it is initialized. Programs execute on this AppDomain.
const domain = MonoApi.mono_get_root_domain()

console.log('domain: ' + classInformation);

// Get a reference to a certain class within the Xamarin application.
var classInformation = ClassHelper.getClassByName(settingClassName);

console.log('classInformation: ' + classInformation);

// Get the pointer to the ahead-of-time (AOT) compiled method
let methodInformation = MonoApiHelper.ClassGetMethodFromName(classInformation, settingMethodName, settingMethodArgCount)

console.log('methodInformation: ' + methodInformation);

// Allocate enough memory for MonoError initialization
let monoErrorMemory = Memory.alloc(32)

// Get the pointer to the method
let nativeMethodPointer = MonoApi.mono_aot_get_method(domain, methodInformation, monoErrorMemory)

// Attach interceptor and fish out the first method argument
Interceptor.attach(nativeMethodPointer, {
    onEnter: function(args) {
        console.log("Entered " + settingMethodName + " with " + settingMethodArgCount + " argument(s).");
        console.log("Value of `string id`: " + MonoApiHelper.StringToUtf8(args[3]));

        args[3] = MonoApiHelper.StringNew('This is the replaced value of `string c`.', domain);
    },
    onLeave: function onLeave(log, retval, state) {
        console.log("Left " + settingMethodName + ".");
    }
})

console.log(`'modify_function_argument.js' attached and ready.`)

The stacktrace I'm getting:
node fridax.js inject --scripts scripts/todo-modify.js
[*] Awaiting storage initialization.
[*] Awaiting USB device.
[*] Up and running on Android Emulator 5554.
? Which application do you want to inject? TodoREST
[*] Happy hacking.
[*] Attached to application (session: 3803).
[*] Injected a test script (this runs from within the injected application)!
domain: undefined
classInformation: 0x98f63330
methodInformation: 0x84b95058
Error: Export not found: mono_aot_get_method
    at vendors/frida-mono-api/mono-api.js:799
    at scripts/todo-modify.js:45
    at o (node_modules/browser-pack/_prelude.js:1)
    at r (node_modules/browser-pack/_prelude.js:1)
    at /script2.js:1217
^C[*] Script unloaded.
[*] Script unloaded.```

If you need more info let me know.

I just fixed this in the latest release. The mono_aot_get_method function was one of the signatures that I removed yesterday. I added it again and used some code from https://github.com/freehuntx/frida-mono-api/pull/6 to prevent Fridax from crashing if some of the defined signatures do not exist in the Mono runtime.

By the way, please be aware of the fact that GetTasksAsync returns a Task instead of a raw value. You will need some extra code to be able to read the results of that task. I think a colleague of mine has some code snippets of this in his notes, feel free to ping me for those notes.

@tijme Bad news but I don't think that this issue is resolved, you actually never removed the mono_aot_get_method function from the signatures. It is now twice in the list and I'm still getting the same error.

I will debug this when I have the time. I think it could be due to the APK not being compiled AOT, but I'm not sure. It looks like somehow the mono_aot_get_method method is not exported in the Mono runtime.

Would you mind sharing the APK that you compiled?

It could very well be that your APK is compiled with an older version of Xamarin/Mono. AOT compilation for Android was introduced in version 5.1 (source, search for AotAssemblies).
Additionally, AOT compilation is not the default on Android.

Hum then that must be the issue:

AOT Compilation
The AOT Compilation option (on the Packaging Properties page) enables Ahead-of-Time (AOT) compilation of assemblies. When this option is enabled, Just In Time (JIT) startup overhead is minimized by precompiling assemblies before runtime. The resulting native code is included in the APK along with the uncompiled assemblies. This results in shorter application startup time, but at the expense of slightly larger APK sizes.
The AOT Compilation option requires an Enterprise license or higher. AOT compilation is available only when the project is configured for Release mode, and it is disabled by default. For more information about AOT Compilation, see AOT.

Source: https://docs.microsoft.com/en-us/xamarin/android/deploy-test/release-prep/?tabs=macos#aot-compilation
Is there another way to hook into functions if AOT Compilation is disabled ?

This is the last APK I compiled (just rename it to .apk):
com.companyname.todorest-Signed.zip

Is there another way to hook into functions if AOT Compilation is disabled ?

As I stated, I think you need to use mono_compile_method (source) to JIT-compile the method. This will return a pointer to the native code produced, which you can use to intercept. You can use this script as a reference, since it uses mono_compile_method and I think it does exactly what you want.
When I have the time I will take a look and see if I can make an example script for this. Or just a method which allows you to intercept both JIT and AOT compiled methods. 👍

Thanks I'll look into that !

As I stated, I think you need to use mono_compile_method

That's correct.
Just as a heads up, in the code you merged from my PR, I have included a helper method to hook JITTed/unJITTed (non-AOT) methods. Like @Techbrunch  said, AOT must be hooked differently.
I'm not sure if you fully merged it or only took the bits that deal with mono runtime image name, but the PR adds MonoApiHelper.Intercept which behaves like Interceptor but takes a mono class definition, a method name and hooks. There's no documentation as of right now (we have an upcoming blog on hooking managed methods in Mono, but it's been backlogged since February. Once it's done I'll link it to you if interested.)
The method is at https://github.com/freehuntx/frida-mono-api/pull/6/files#diff-1620eb23fbd98d7f220961acb8cf39b3R98
Hope this can help a bit!

@tijme @alxbl It works like a charm thanks !
My script:
import { MonoApiHelper, MonoApi } from '../vendors/frida-mono-api'
import ClassHelper from '../libraries/class_helper'

// Intercept settings
var className = "CompanyName.ProjectName.Views.Web.Html.HtmlWebView";

// The root AppDomain is the initial domain created by the runtime when it is initialized. Programs execute on this AppDomain.
const domain = MonoApi.mono_get_root_domain()

// Get a reference to a certain class within the Xamarin application.
var classInformation = ClassHelper.getClassByName(className);

console.log('classInformation: ' + classInformation);

MonoApiHelper.Intercept(classInformation, 'GetElement', {
    onEnter: function(args) {
        console.log("Entered GetElement");

        var arg0 = MonoApiHelper.StringToUtf8(args[0]);
        console.log('Old arg0: ' + arg0)
        var newArg0 = MonoApiHelper.StringNew(url.replace('old', 'new'), domain);
        console.log('New arg0: ' + MonoApiHelper.StringToUtf8(newArg0))
        args[0] = newArg0;
    },
    onLeave: function onLeave(log, retval, state) {
        console.log("Left GetElement.");
    }
})

console.log(`'modify_function_argument.js' attached and ready.`)

FYI: I was also able to patch the APK permanantly using Objection (https://github.com/sensepost/objection/pull/329#issuecomment-609438408)
@alxbl Looking forward to read the blog post, I might write one in the meantime.

I'm glad that it's working for you @Techbrunch. Thanks to @alxbl for writing this method.

@Techbrunch The blog is live now if you're still interested in reading it. It showcases how to bypass certificate pinning in Xamarin using Frida.
The work was done before Fridax released, but I added a reference to fridax at the end since I feel like this project's goal is a superset of what we were trying to achieve.
Cheers!

@alxbl Thanks, for the link. I was wondering how did you go about intercepting the traffic in the first place ? In my case I had to redirect inbound requests by modifying the host file on the phone as suggested by Burp documentation (https://portswigger.net/support/using-burp-suites-invisible-proxy-settings-to-test-a-non-proxy-aware-thick-client-application). I was wondering if you had a better solution ?

So, I replied in PM, but I'll leave the relevant part in this issue in case it can help others.
I setup MITMProxy in transparent mode, and then used USB reverse tunneling (adb reverse) and iptables to force HTTP/S traffic through the reverse tunnel and through MITMProxy.
The commands:
# On the MITM with rooted Android phone plugged through USB
adb reverse tcp:8080 tcp:8080

# On the Phone (adb shell, su)
iptables -t nat -A OUTPUT -p tcp --dport 80 -j DNAT --to-destination <IP_ADDRESS>:8080
iptables -t nat -A OUTPUT -p tcp --dport 443 -j DNAT --to-destination <IP_ADDRESS>:8080


Tips for getting this running

Do you have any tips for getting this running?  I've got a very large and well developed corpus of wgsl shader files from a previous fuzzing run which I would like to try with this fuzzer. I am attempting to get this running on Ubuntu 24.04.1.  Rust nightly toolchain but could use any version of linux which will run on bare metal on HP Proliant servers.
Rough steps completed so far:
Grabbed the code.
Grabbed afl++4.10c, applied the patch, make dist.  Stopped short of installing it because I dont think its needed?
Followed harnesses/dawn/README to succesfully build tint_afl_all_fuzzer
Followed harnesses/dxcompiler/README.md to succesfully create the libdxcompiler.so
LD_LIBRARY_PATH=/home/alan/darthshader/harnesses/dxcompiler/DirectXShaderCompiler/out/build/lib /home/alan/darthshader/target/release/darthshader --debug-child --input corpus --output output /home/alan/darthshader/harnesses/dawn/dawn/out/build/tint_afl_all_fuzzer
$ LD_LIBRARY_PATH=/home/alan/ /home/alan/darthshader/target/release/darthshader --debug-child --input corpus --output output /home/alan/darthshader/harnesses/dawn/dawn/out/build/tint_afl_all_fuzzer
Workdir: "/home/alan/darthshader_run"
Out dir at "output" already exists.
Let's fuzz :)
dxcomplib: (nil)
Coverage map size: Some(2621440)
"corpus//*"
from file "corpus/05a79f06cf3f67f726dae68d18a2290f6c9a50c9.wgsl"
Attempting to parse as AST: "corpus/05a79f06cf3f67f726dae68d18a2290f6c9a50c9.wgsl"
File instrumentation/afl-compiler-rt.o.c, line 897: Error(wrong forkserver message from AFL++ tool): Success
Failed to load initial corpus at "corpus//*" Unknown("Unable to request new process from fork server (OOM?)", ErrorBacktrace)
I've also tried with a corpus with a single file 1.wgsl with empty content or simply '1' in there.  Lastly I tried with a simple file from googles tint_wgsl_fuzzer_seed_corpus.zip.  Always getting the same 'error' message.  Box is an HP DL360p Gen10 with 256GB ram.  No signs of running out of memory.

Above is with deliberately truncated LD path - trying to see if running without libdxcompiler.so would work.
LD_LIBRARY_PATH=/home/alan/darthshader/harnesses/dxcompiler/DirectXShaderCompiler/out/build/lib /home/alan/darthshader/target/release/darthshader --debug-child --input corpus --output output -t 12000 /home/alan/darthshader/harnesses/dawn/dawn/out/build/tint_afl_all_fuzzer
Workdir: "/home/alan/darthshader_run"
Out dir at "output" already exists.
Let's fuzz :)
dxcomplib: 0x51a000000080
dxcCreateInstance: 0x7b370b463700
Coverage map size: Some(2621440)
"corpus//*"
from file "corpus/05a79f06cf3f67f726dae68d18a2290f6c9a50c9.wgsl"
Attempting to parse as AST: "corpus/05a79f06cf3f67f726dae68d18a2290f6c9a50c9.wgsl"
File instrumentation/afl-compiler-rt.o.c, line 897: Error(wrong forkserver message from AFL++ tool): Success
Failed to load initial corpus at "corpus//*" Unknown("Unable to request new process from fork server (OOM?)", ErrorBacktrace)
Is with correct path.
I've also tried to debug this by running the fuzz harness directly:
LD_LIBRARY_PATH=/home/alan/darthshader/harnesses/dxcompiler/DirectXShaderCompiler/out/build/lib /home/alan/darthshader/harnesses/dawn/dawn/out/build/tint_afl_all_fuzzer
[-] FATAL: forkserver is already up, but an instrumented dlopen() library loaded afterwards. You must AFL_PRELOAD such libraries to be able to fuzz them or LD_PRELOAD to run outside of afl-fuzz.
To ignore this set AFL_IGNORE_PROBLEMS=1 but this will lead to ambiguous coverage data.
In addition, you can set AFL_IGNORE_PROBLEMS_COVERAGE=1 to ignore the additional coverage instead (use with caution!).
Aborted
Which doesnt seem to work; however looking in the code it seems you're not meant to run it like this anyway as the rust sets an ENV variable...

  I checked out the wrong afl++ version.  Serves me right for trying to get it going at 3am I guess.

The issue is probably caused by the changes in the forkserver interface of recent versions of AFL++. The libAFL version this fuzzer is built upon doesn't support the new interface. Going forward, it'd be best to upgrade libAFL (which is not a walk in the park because so many things break).

Thanks for coming back to me.  Currently trying to get the older version of AFL to build on 24.04.1 but not having a lot of luck.
It would probably be easiest to reinstall the box with whatever version of linux you used?

Sure thing. I'm using the same Ubuntu version, i.e. 24.04. There is also this archive of Darthshader https://zenodo.org/records/13302737 with a dockerfile for building. It builds plenty of other things (e.g., wgslsmith) which you might not need/want, however it contains the exact build steps I've been taking. If you figure out the issue please let me know so the description can be updated.

Do you have the steps you took to compile afl++ documented?  Its not building afl-clang-fast/++ for me which I recall from over a year ago (last time I built this) is a build dependency issue.  I've added llvm-dev which gets it a lot further.  If it doesnt manage to compile afl-clang-fast it symlinks it to afl-cc which creates obvious errors when trying to build the harnesses.

Looks like you dont actually need the afl-clang-fast binary per say.  But you do need it to build with llvm.  So basically I followed the instructions in docs/install and just added apt install llvm-dev and did make all instead of make distrib.

alan@dl360p10fuzz:~/darthshader_run$ cat corpus_empty/1.wgsl
@vertex
fn main(
@builtin(vertex_index) VertexIndex : u32
) -> @builtin(position) vec4f {
var pos = array<vec2f, 3>(
vec2(0.0, 0.5),
vec2(-0.5, -0.5),
vec2(0.5, -0.5)
);
return vec4f(pos[VertexIndex], 0.0, 1.0);
}
alan@dl360p10fuzz:~/darthshader_run$ LD_LIBRARY_PATH=/home/alan/darthshader/harnesses/dxcompiler/DirectXShaderCompiler/out/build/lib /home/alan/darthshader/target/release/darthshader --debug-child --input corpus_empty --output output -t 12000 /home/alan/darthshader/harnesses/dawn/dawn/out/build/tint_afl_all_fuzzer
Workdir: "/home/alan/darthshader_run"
Out dir at "output" already exists.
Let's fuzz :)
Coverage map size: Some(2621440)
"corpus_empty/**/*"
from file "corpus_empty/1.wgsl"
dxcomplib: 0x51a000000080
dxcCreateInstance: 0x700030663760
result 0
[Stats #0] run time: 0h-0m-0s, clients: 1, corpus: 0, objectives: 0, executions: 0, exec/sec: 0.000, shared_mem: 67039/2621440 (2%)
Client 000:
NaN: Scheduler
NaN: Manager
Feedbacks:
NaN: Not Measured
from file "output/queue/1727614531_000000.ron"
Offending file: "output/queue/1727614531_000000.ron"
thread 'main' panicked at src/layeredinput.rs:301:17:
file import error
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
alan@dl360p10fuzz:~/darthshader_run$ cat output/queue/1727614
1727614471_000000.ron<PHONE_NUMBER>_000000.ron
alan@dl360p10fuzz:~/darthshader_run$ cat output/queue/1727614531_000000.ron
IR((module:(types:[(name:None,inner:Scalar(kind:Uint,width:4)),(name:None,inner:Vector(size:Quad,kind:Float,width:4)),(name:None,inner:Vector(size:Bi,kind:Float,width:4)),(name:None,inner:Array(base:3,size:Constant(3),stride:8))],special_types:(ray_desc:None,ray_intersection:None,predeclared_types:{}),constants:[],global_variables:[],const_expressions:[Literal(I32(3))],functions:[],entry_points:[(name:"main",stage:Vertex,early_depth_test:None,workgroup_size:(0,0,0),function:(name:Some("main"),arguments:[(name:Some("l"),ty:1,binding:Some(BuiltIn(VertexIndex)))],result:Some((ty:2,binding:Some(BuiltIn(Position(invariant:false))))),local_variables:[(name:Some("l"),ty:4,init:Some(11))],expressions:[FunctionArgument(0),Literal(F32(0.0)),Literal(F32(0.5)),Compose(ty:3,components:[2,3]),Literal(F32(-0.5)),Literal(F32(-0.5)),Compose(ty:3,components:[5,6]),Literal(F32(0.5)),Literal(F32(-0.5)),Compose(ty:3,components:[8,9]),Compose(ty:4,components:[4,7,10]),LocalVariable(1),Access(base:12,index:1),Load(pointer:13),Literal(F32(0.0)),Literal(F32(1.0)),Compose(ty:2,components:[14,15,16])],named_expressions:{1:"VertexIndex"},body:[Emit((start:3,end:4)),Emit((start:6,end:7)),Emit((start:9,end:10)),Emit((start:10,end:11)),Emit((start:12,end:13)),Emit((start:13,end:14)),Emit((start:16,end:17)),Return(value:Some(17))]))])))

Some progress.  Its not clear to me if it 'likes' malformed wgsl files, which my huge corpus will be full of.  So I am trying with a corpus of a simple wgsl file:
$ cat corpus_empty/1.wgsl
@vertex
fn main(
@builtin(vertex_index) VertexIndex : u32
) -> @builtin(position) vec4f {
var pos = array<vec2f, 3>(
vec2(0.0, 0.5),
vec2(-0.5, -0.5),
vec2(0.5, -0.5)
);
return vec4f(pos[VertexIndex], 0.0, 1.0);
}
$ LD_LIBRARY_PATH=/home/alan/darthshader/harnesses/dxcompiler/DirectXShaderCompiler/out/build/lib /home/alan/darthshader/target/release/darthshader --debug-child --input corpus_empty --output output -t 12000 /home/alan/darthshader/harnesses/dawn/dawn/out/build/tint_afl_all_fuzzer
Workdir: "/home/alan/darthshader_run"
Out dir at "output" already exists.
Let's fuzz :)
Coverage map size: Some(2621440)
"corpus_empty/**/*"
from file "corpus_empty/1.wgsl"
dxcomplib: 0x51a000000080
dxcCreateInstance: 0x700030663760
result 0
[Stats #0] run time: 0h-0m-0s, clients: 1, corpus: 0, objectives: 0, executions: 0, exec/sec: 0.000, shared_mem: 67039/2621440 (2%)
Client 000:
NaN: Scheduler
NaN: Manager
Feedbacks:
NaN: Not Measured
from file "output/queue/1727614531_000000.ron"
Offending file: "output/queue/1727614531_000000.ron"
thread 'main' panicked at src/layeredinput.rs:301:17:
file import error
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
$ cat output/queue/1727614531_000000.ron
IR((module:(types:[(name:None,inner:Scalar(kind:Uint,width:4)),(name:None,inner:Vector(size:Quad,kind:Float,width:4)),(name:None,inner:Vector(size:Bi,kind:Float,width:4)),(name:None,inner:Array(base:3,size:Constant(3),stride:8))],special_types:(ray_desc:None,ray_intersection:None,predeclared_types:{}),constants:[],global_variables:[],const_expressions:[Literal(I32(3))],functions:[],entry_points:[(name:"main",stage:Vertex,early_depth_test:None,workgroup_size:(0,0,0),function:(name:Some("main"),arguments:[(name:Some("l"),ty:1,binding:Some(BuiltIn(VertexIndex)))],result:Some((ty:2,binding:Some(BuiltIn(Position(invariant:false))))),local_variables:[(name:Some("l"),ty:4,init:Some(11))],expressions:[FunctionArgument(0),Literal(F32(0.0)),Literal(F32(0.5)),Compose(ty:3,components:[2,3]),Literal(F32(-0.5)),Literal(F32(-0.5)),Compose(ty:3,components:[5,6]),Literal(F32(0.5)),Literal(F32(-0.5)),Compose(ty:3,components:[8,9]),Compose(ty:4,components:[4,7,10]),LocalVariable(1),Access(base:12,index:1),Load(pointer:13),Literal(F32(0.0)),Literal(F32(1.0)),Compose(ty:2,components:[14,15,16])],named_expressions:{1:"VertexIndex"},body:[Emit((start:3,end:4)),Emit((start:6,end:7)),Emit((start:9,end:10)),Emit((start:10,end:11)),Emit((start:12,end:13)),Emit((start:13,end:14)),Emit((start:16,end:17)),Return(value:Some(17))]))])))

Full backtrace:
stack backtrace:
0:     0x59db002fe309 - std::backtrace_rs::backtrace::libunwind::trace::h9294c60f1c519a6b
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/std/src/../../backtrace/src/backtrace/libunwind.rs:116:5
1:     0x59db002fe309 - std::backtrace_rs::backtrace::trace_unsynchronized::hb186c7e77bffad8a
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
2:     0x59db002fe309 - std::sys::backtrace::_print_fmt::hcb03e5d9b54439b1
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/std/src/sys/backtrace.rs:66:9
3:     0x59db002fe309 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h9f6912c11cb864e2
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/std/src/sys/backtrace.rs:39:26
4:     0x59db000d4223 - core::fmt::rt::Argument::fmt::h3d03db0338420f78
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/core/src/fmt/rt.rs:177:76
5:     0x59db000d4223 - core::fmt::write::h8debadf270d89c55
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/core/src/fmt/mod.rs:1186:21
6:     0x59db002d10e2 - std::io::Write::write_fmt::h2027e8c4646c73e6
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/std/src/io/mod.rs:1833:15
7:     0x59db00304583 - std::sys::backtrace::BacktraceLock::print::h414cd546f6b36885
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/std/src/sys/backtrace.rs:42:9
8:     0x59db00304abc - std::panicking::default_hook::{{closure}}::h40ea18cfe0adea53
9:     0x59db00304abc - std::panicking::default_hook::h8d94df497eda232c
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/std/src/panicking.rs:295:9
10:     0x59db00304abc - std::panicking::rust_panic_with_hook::h9a5479236c4bea0b
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/std/src/panicking.rs:801:13
11:     0x59db00304625 - std::panicking::begin_panic_handler::{{closure}}::hf9517d9cf0232ee4
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/std/src/panicking.rs:667:13
12:     0x59db003045b9 - std::sys::backtrace::__rust_end_short_backtrace::h623c1073442a1748
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/std/src/sys/backtrace.rs:170:18
13:     0x59db003045ac - rust_begin_unwind
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/std/src/panicking.rs:665:5
14:     0x59db000d27cf - core::panicking::panic_fmt::hc1611ab468cf2998
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/core/src/panicking.rs:74:14
15:     0x59dafff9025a - <darthshader::layeredinput::LayeredInput as libafl::inputs::Input>::from_file::h7aac364fe088f659
at /home/alan/darthshader/src/layeredinput.rs:301:17
16:     0x59dafff9025a - <libafl::corpus::inmemory_ondisk::InMemoryOnDiskCorpus as libafl::corpus::Corpus>::load_input_into::haad394b0697bb104
at /home/alan/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libafl-0.11.2/src/corpus/inmemory_ondisk.rs:153:25
17:     0x59dafffb131a - libafl::corpus::testcase::Testcase::load_len::h08a0f70f7f64a616
at /home/alan/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libafl-0.11.2/src/corpus/testcase.rs:346:21
18:     0x59dafffb131a - <libafl::schedulers::testcase_score::LenTimeMulTestcaseScore as libafl::schedulers::testcase_score::TestcaseScore>::compute::hc6140e8be637741a
at /home/alan/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libafl-0.11.2/src/schedulers/testcase_score.rs:43:15
19:     0x59dafffa523e - libafl::schedulers::minimizer::MinimizerScheduler<CS,F,M>::update_score::h52a54ba0cca9cb8f
at /home/alan/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libafl-0.11.2/src/schedulers/minimizer.rs:268:26
20:     0x59dafffa523e - <libafl::schedulers::minimizer::MinimizerScheduler<CS,F,M> as libafl::schedulers::Scheduler>::on_add::hc2bdcbe1d68e303a
at /home/alan/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libafl-0.11.2/src/schedulers/minimizer.rs:204:9
21:     0x59daffff775a - <libafl::fuzzer::StdFuzzer<CS,F,OF,OT> as libafl::fuzzer::ExecutionProcessor>::process_execution::hac8bda4042be0823
at /home/alan/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libafl-0.11.2/src/fuzzer/mod.rs:390:17
22:     0x59daffff775a - <libafl::fuzzer::StdFuzzer<CS,F,OF,OT> as libafl::fuzzer::EvaluatorObservers>::evaluate_input_with_observers::hd1a65812049d490b
at /home/alan/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libafl-0.11.2/src/fuzzer/mod.rs:471:9
23:     0x59daffff775a - <libafl::fuzzer::StdFuzzer<CS,F,OF,OT> as libafl::fuzzer::Evaluator<E,EM>>::evaluate_input_events::h62735126c63aaf8b
at /home/alan/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libafl-0.11.2/src/fuzzer/mod.rs:495:9
24:     0x59daffff775a - libafl::fuzzer::Evaluator::evaluate_input::h0a42049ea75bdc4e
at /home/alan/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libafl-0.11.2/src/fuzzer/mod.rs:124:9
25:     0x59db0008036a - libafl::state::StdState<I,C,R,SC>::continue_loading_initial_inputs_custom::h38a08f7cf40ba174
at /home/alan/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libafl-0.11.2/src/state/mod.rs:665:40
26:     0x59db0008036a - libafl::state::StdState<I,C,R,SC>::load_initial_inputs_custom_by_filenames::h85910f404f1eb5aa
at /home/alan/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libafl-0.11.2/src/state/mod.rs:638:9
27:     0x59db0008036a - libafl::state::StdState<I,C,R,SC>::load_initial_inputs_by_filenames::h14363f970c175769
at /home/alan/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libafl-0.11.2/src/state/mod.rs:703:9
28:     0x59db0008036a - darthshader::fuzz::h6777bbb425b9a098
at /home/alan/darthshader/src/main.rs:406:14
29:     0x59db00076246 - darthshader::main::hcb1c4bdbdc4a0397
at /home/alan/darthshader/src/main.rs:221:5
30:     0x59dafffb2716 - core::ops::function::FnOnce::call_once::h7c402fe0424445f2
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/core/src/ops/function.rs:250:5
31:     0x59dafffb2716 - std::sys::backtrace::__rust_begin_short_backtrace::hb45b1ed673021213
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/std/src/sys/backtrace.rs:154:18
32:     0x59db0008b32d - main
33:     0x7ca30ce2a1ca - __libc_start_call_main
at ./csu/../sysdeps/nptl/libc_start_call_main.h:58:16
34:     0x7ca30ce2a28b - __libc_start_main_impl
at ./csu/../csu/libc-start.c:360:3
35:     0x59dafff8db95 - _start
36:                0x0 -
Unfortunately day job is calling, so I will have to come back to this later on.  Any thoughts would be valued.

I could probably add some code that simple skips samples that cannot be imported. Would that help?

Yeah; that or I will figure out a way to pre process the files.  It seems to just be ones with UTF8 chars in them that are tripping it up so far.
I got it working - I think, but at least so far I can only get the version from your docker file to work.  The version from github just refuses with errors above.
Output looks like this: [Testcase #0] run time: 0h-1m-10s, clients: 1, corpus: 218, objectives: 2, executions: 302, exec/sec: 4.273, shared_mem: 56391/2621440 (2%), execs_err: 176, execs_suc: 75
repeating with incrementing numbers.  Looks single threaded, so presumably I need to run multiple instances to use my large boxes.

A quick and dirty python script to split files to ascii only and non ascii did help but it still eventually chokes on a file.  I think your suggestion about skipping over unimportable files makes the most sense.
These files are whats left after I modifed one of Googles fuzzers in the summer to generate wgsl files by brute force, ignorance and a lot of compute power.  This is the minimized corpus that was left after I got bored.  I found an almost untriagable amount of crashes with quite a few genuine bugs but sadly nothing that crossed the security line.

I had a good go at sanitizing my corpus this evening.  I think the fuzzer 'likes' input files which 1) dont output errors when fed into some kind of tint operation and 2) dont contain any non ascii characters.  Sadly the tint operation(s) the fuzzer wants to pass are evading me and my rust skills are almost non existent so I've not managed to work it out by reading the code yet.  (I thought maybe it was looking for no errors in wgsl->ast conversion, however after trying various ways it keeps finding new errors)

First couple that came out:
fn f(){let
l=modf(0);}
fn
b(){let
t=modf(dpdx(5));}
I think the fuzzer loads shortest files first, there are many more complex files than this along with enormous amounts of files probably containing pretty much every kind of syntax error imaginable.
I left the fuzzer running overnight on the box and was greeted this morning with an enormous amount of crashers.  It seems that triaging these is not entirely straight forward because the fuzzer saves no relevant output to help me deduce why a file ended up in there.  So far I've figured out that the 'lifter' executable turns the ron files into wgsl.  It then seems sensible to transform those into hlsl (tint filename.wgsl -o filename.hlsl) and at this point I can feed them into dxc standalone: /home/alan/darthshader/harnesses/dxcompiler/DirectXShaderCompiler/out/build/bin/dxc-3.7 filename -T something -HV year.  Not knowing which variants of -T are interesting I slapped together a triage script that brute force tests every possible variant in parallel.  I guess this works but might be wasteful (its been running for hours now) and perhaps there are other CLI based triage processes that might highlight the exciting crashers?

It seems the documentation lacks a couple of important steps. I'll update that, here is some info already:
Regarding the -T parameter: you'll need to configure the type of shader you want to compile: vertex (vs_), fragment (ps_), or compute (cs_). The type of the shader you need to set depends on the entry point present in the shader.
Furthermore, you need to specify the shader version. The version should be between 6_2 and 6_6. So a compute shader in version 6.3 becomes -T cs_6_3. There are more recent shader versions, e.g. 6.7. AFAIK those are not supported by dawn/tint, those reports will hence (likely) excluded from the VRP.
I recommend using the same flags for dxc as specified in the harness, see https://github.com/wgslfuzz/darthshader/blob/6ca6258bbd91fbc3111ce9b4b8a58384afaeee23/harnesses/dawn/patches/dawn_3de0f00.diff#L569-L581
On older versions of dxc (in particular before March 2024) you should find plenty of crashes rather quickly, considering you mentioned having access to a couple of cores. Many issues have been fixed since then, so I'd expect bugs to be rather sparse.

Thanks.  This is really helpful.
Based upon this I think the dxc command I should triage with is /home/alan/darthshader/harnesses/dxcompiler/DirectXShaderCompiler/out/build/bin/dxc-3.7 -E unused_entry_point -Gis -Zpr -enable-16bit-types -HV 2018 -T cs_6_6 file.hlsl; since I dont know for any given test case what kind of shader we are dealing with I figure just try all three relevant ones.
I imported a few seeds from my corpus (basically until it blew up the import process, then restarted it with no input selected) which resulted in thousands of 'crashes' over night.  Following the triage process manually I get output as below on final step; which doesnt look like a crash to my eyes leading me to wonder why this was in the crashes directory?
dxc-3.7 -E unused_entry_point -Gis -Zpr -enable-16bit-types -HV 2018 -T cs_6_6<PHONE_NUMBER>_000067.wgsl.hlsl
;
; Input signature:
;
; Name                 Index   Mask Register SysValue  Format   Used
; -------------------- ----- ------ -------- -------- ------- ------
; no parameters
;
; Output signature:
;
; Name                 Index   Mask Register SysValue  Format   Used
; -------------------- ----- ------ -------- -------- ------- ------
; no parameters
; shader hash: 37007ca9dcc6243ff5cf1bc084abd224
;
; Pipeline Runtime Information:
;
;PSVRuntimeInfo:
; Compute Shader
; NumThreads=(1,1,1)
; MinimumExpectedWaveLaneCount: 0
; MaximumExpectedWaveLaneCount:<PHONE_NUMBER>
; UsesViewID: false
; SigInputElements: 0
; SigOutputElements: 0
; SigPatchConstOrPrimElements: 0
; SigInputVectors: 0
; SigOutputVectors[0]: 0
; SigOutputVectors[1]: 0
; SigOutputVectors[2]: 0
; SigOutputVectors[3]: 0
; EntryFunctionName: unused_entry_point
;
;
; Buffer Definitions:
;
;
; Resource Bindings:
;
; Name                                 Type  Format         Dim      ID      HLSL Bind  Count
; ------------------------------ ---------- ------- ----------- ------- -------------- ------
;
target datalayout = "e-m:e-p:32:32-i1:32-i8:8-i16:16-i32:32-i64:64-f16:16-f32:32-f64:64-n8:16:32:64"
target triple = "dxil-ms-dx"

define void @unused_entry_point() {
  ret void
}

!llvm.ident = !{!0}
!dx.version = !{!1}
!dx.valver = !{!2}
!dx.shaderModel = !{!3}
!dx.entryPoints = !{!4}

!0 = !{!"dxc(private) <IP_ADDRESS>46 (main, 75ff50caa)"}
!1 = !{i32 1, i32 6}
!2 = !{i32 1, i32 8}
!3 = !{!"cs", i32 6, i32 6}
!4 = !{void ()* @unused_entry_point, !"unused_entry_point", null, null, !5}
!5 = !{i32 0, i64 8388608, i32 4, !6}
!6 = !{i32 1, i32 1, i32 1}

The output indicates that for some reason the standalone build of dxc didn't crash. This can have multiple root causes:

you compiled the wrong entry point (ruled out by your attempt to compile each of them)
a mismatch in build flags between the dxc build used for fuzzing vs. the one you used to verify the crash. E.g., the verifier build is without ASAN whereas the fuzzing build is

Assuming there is no mismatch in build flags I recommend the following: build the tint harness + dxc without AFL but just use a regular clang (but do use the same ASAN/UBSAN/... sanitizer flags). This will allow you to pass the wgsl file to dxc exactly as during the fuzzing progress. This should trigger a crash, either in tint or in dxc.

Another option for not observing the crash in dxc could be your input crashing in tint during fuzzing. Make sure to use the same build flags (modulo instrumentation) for converting from wgsl to hlsl

Thanks, how do you define 'crashed'?  I've been watching the triage process and it seems tint outputs some errors and warning which appear to be related to malformed code - eg warning unreachable, one or two other error messages.  This might not be unexpected as my input files may not have been valid wgsl in the first place (in fact most likely were not - they came from a lot of random monkeys smashing a wgsl code keyboard).  One avenue where I may be missing stuff here is sometimes tint generating invalid code in a conversion is considered a security issue in itself?  Maybe thats why you save these?
Beyond that; I tried to edit the rust to skip over malformed files as at the minute its only importing a few hundred files out of my ~100k samples.  But unfortunately my rust skills may as well not exist and all I did was make it blow up in literally hundreds of errors.  Any pointers here would be very much appreciated + others I am certain would find the ability to skip over unacceptable inputs useful.

I just pushed a version that should skip all seeds that cannot be imported. Its EOB here but I'll come back regarding the crashes tomorrow.

We have some progress I think as I can now build the version direct from git, whereas previously I had to build the version from the docker image.
This morning before going to my first normal job of the day I tried throwing my complete "monkey" corpus at it.  I also tried the complete corpus with only printable files.  Sadly its blowing up quite early with attached error log/backtrace.  I think its choking on attached file.
badfile.zip
crash.txt

I also tried with an empty corpus and a corpus which includes just one valid file, which is also blowing up.  I'm not sure if this issue might be the same one that stopped me running the original copy I grabbed from git?  This particular crash went away when I just built the version from your docker file.  Attached crash trace.
crash.txt

I believe we're facing 2 different issues: the former one (the one from badfile.zip) contains a shader that triggers a panic in naga. The following happens during import:

the shader is imported by the fuzzer
the shader is passed to naga in order to lift it to IR
naga crashes/panics during parsing, which in turn terminates the entire fuzzer

This is a bit of an annoying situation; I'll tomorrow add some code to catch these errors.
Regarding the latter crash: I can't quite make sense of this one, the file was successfully lifted (to<PHONE_NUMBER>_000000.ron), but when attempting to load the .ron file the fuzzer crashes.

Thanks for your efforts so far :-)
What I dont understand why does the copy I extracted from your docker file build and run with an empty corpus, when the version from git never has?  Something feels off there...
Maybe I do something wrong in building the git version.  Here is my complete process; excluding installing build prereqs etc:
git clone https://github.com/wgslfuzz/darthshader.git
cd darthshader
cargo build
wget https://github.com/AFLplusplus/AFLplusplus/archive/refs/tags/v4.10c.zip
cd AFLplusplus-4.10c
make -j112 source-only
cd dawn
git clone https://dawn.googlesource.com/dawn
cd dawn
git checkout 3de0f00
cp scripts/standalone.gclient .gclient
gclient sync # this might trigger an error but should be fine anyways
cp ../patches/dawn_3de0f00.diff .
git apply dawn_3de0f00.diff
mkdir -p out/build
cd out/build
CC=/home/alan/darthshader/AFLplusplus-4.10c/afl-clang-fast CXX=/home/alan/darthshader/AFLplusplus-4.10c/afl-clang-fast++ cmake -GNinja -DTINT_BUILD_AFL_FUZZER=ON -DDAWN_ENABLE_ASAN=ON -DTINT_BUILD_MSL_WRITER=ON -DTINT_BUILD_SPV_WRITER=ON -DTINT_BUILD_HLSL_WRITER=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_COMPILER_AR=/usr/bin/llvm-ar-18 -DCMAKE_CXX_COMPILER_RANLIB=/usr/bin/llvm-ranlib-18 -DCMAKE_C_COMPILER_AR=/usr/bin/llvm-ar-18 -DCMAKE_C_COMPILER_RANLIB=/usr/bin/llvm-ranlib-18 -DCMAKE_CXX_FLAGS="-fuse-ld=lld" -DCMAKE_C_FLAGS="-fuse-ld=lld" ../..
ninja tint_afl_all_fuzzer
cd ~/darthshader/harnesses/dxcompiler/
git clone https://github.com/microsoft/DirectXShaderCompiler.git
cd DirectXShaderCompiler
git submodule init
git submodule update
cd DirectXShaderCompiler
mkdir -p out/build
cd out/build
CC=/home/alan/darthshader/AFLplusplus-4.10c/afl-clang-fast CXX=/home/alan/darthshader/AFLplusplus-4.10c/afl-clang-fast++ cmake ../../ -C ../../cmake/caches/PredefinedParams.cmake -DCMAKE_BUILD_TYPE=Release -DDXC_DISABLE_ALLOCATOR_OVERRIDES=ON -DENABLE_SPIRV_CODEGEN=OFF -DSPIRV_BUILD_TESTS=OFF -DLLVM_USE_SANITIZER=Address -DLLVM_ENABLE_LTO=Off -G Ninja
ninja
cd
mkdir darthshader_run2
RUST_BACKTRACE=full LD_LIBRARY_PATH=/home/alan/darthshader/harnesses/dxcompiler/DirectXShaderCompiler/out/build /home/alan/darthshader/target/debug/darthshader --output output -t 12000 /home/alan/darthshader/harnesses/dawn/dawn/out/build/tint_afl_all_fuzzer 2>&1 | tee crash
cat crash

$ cat crash
Workdir: "/home/alan/darthshader_run2"
Let's fuzz :)
Coverage map size: Some(212800)
[Stats #0] run time: 0h-0m-0s, clients: 1, corpus: 0, objectives: 0, executions: 0, exec/sec: 0.000, shared_mem: 5227/212800 (2%)
Client 000:
NaN: Scheduler
NaN: Manager
Feedbacks:
NaN: Not Measured
from file "output/queue/1727906272_000000.ron"
Offending file: "output/queue/1727906272_000000.ron"
thread 'main' panicked at src/layeredinput.rs:301:17:

Yet in the same folder the version I untarred from your large docker tarball just runs and all I did was untar it and run docker run in the relevant folder:
LD_LIBRARY_PATH=/home/alan/darthshader/harnesses/dxcompiler/DirectXShaderCompiler/out/build /home/alan/scratch/aedarthshader/buildfiles/darthshader/target/debug/darthshader --output output -t 12000 /home/alan/darthshader/harnesses/dawn/dawn/out/build/tint_afl_all_fuzzer

I added a panic handler that should allow you to better deal with files posing issues during import.

Sadly its now blowing up in main.  Here is a sample of files from my corpus that blow it up and a crash trace.
Client 000:
NaN: Scheduler
NaN: Manager
Feedbacks:
NaN: Not Measured
from file "output/queue/1728167311_000000.ron"
Offending file: "output/queue/1728167311_000000.ron"
thread 'main' panicked at src/layeredinput.rs:304:17:
file import error
stack backtrace:
0:     0x5de96f395e9a - std::backtrace_rs::backtrace::libunwind::trace::h9294c60f1c519a6b
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/std/src/../../backtrace/src/backtrace/libunwind.rs:116:5
1:     0x5de96f395e9a - std::backtrace_rs::backtrace::trace_unsynchronized::hb186c7e77bffad8a
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
2:     0x5de96f395e9a - std::sys::backtrace::_print_fmt::hcb03e5d9b54439b1
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/std/src/sys/backtrace.rs:66:9
3:     0x5de96f395e9a - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h9f6912c11cb864e2
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/std/src/sys/backtrace.rs:39:26
4:     0x5de96f3bf063 - core::fmt::rt::Argument::fmt::h3d03db0338420f78
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/core/src/fmt/rt.rs:177:76
5:     0x5de96f3bf063 - core::fmt::write::h8debadf270d89c55
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/core/src/fmt/mod.rs:1186:21
6:     0x5de96f392583 - std::io::Write::write_fmt::h2027e8c4646c73e6
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/std/src/io/mod.rs:1833:15
7:     0x5de96f395ce2 - std::sys::backtrace::BacktraceLock::print::h414cd546f6b36885
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/std/src/sys/backtrace.rs:42:9
8:     0x5de96f396cc7 - std::panicking::default_hook::{{closure}}::h40ea18cfe0adea53
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/std/src/panicking.rs:268:22
9:     0x5de96f396af6 - std::panicking::default_hook::h8d94df497eda232c
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/std/src/panicking.rs:295:9
10:     0x5de96f3972f7 - std::panicking::rust_panic_with_hook::h9a5479236c4bea0b
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/std/src/panicking.rs:801:13
11:     0x5de96f397156 - std::panicking::begin_panic_handler::{{closure}}::hf9517d9cf0232ee4
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/std/src/panicking.rs:667:13
12:     0x5de96f396379 - std::sys::backtrace::__rust_end_short_backtrace::h623c1073442a1748
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/std/src/sys/backtrace.rs:170:18
13:     0x5de96f396e1c - rust_begin_unwind
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/std/src/panicking.rs:665:5
14:     0x5de96f3bc470 - core::panicking::panic_fmt::hc1611ab468cf2998
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/core/src/panicking.rs:74:14
15:     0x5de96e9a26a4 - <darthshader::layeredinput::LayeredInput as libafl::inputs::Input>::from_file::hbc4f999f44a994b2
at /home/alan/darthshader/src/layeredinput.rs:304:17
16:     0x5de96ea6a134 - <libafl::corpus::inmemory_ondisk::InMemoryOnDiskCorpus as libafl::corpus::Corpus>::load_input_into::h20cf7da0932ee256
at /home/alan/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libafl-0.11.2/src/corpus/inmemory_ondisk.rs:153:25
17:     0x5de96ea05e7b - libafl::corpus::testcase::Testcase::load_len::h87a22b2c4ce223d7
at /home/alan/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libafl-0.11.2/src/corpus/testcase.rs:346:21
18:     0x5de96e87568d - <libafl::schedulers::testcase_score::LenTimeMulTestcaseScore as libafl::schedulers::testcase_score::TestcaseScore>::compute::hb88046b5ceb81a8f
at /home/alan/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libafl-0.11.2/src/schedulers/testcase_score.rs:43:15
19:     0x5de96e9baab5 - libafl::schedulers::minimizer::MinimizerScheduler<CS,F,M>::update_score::hd3c164667d9429a4
at /home/alan/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libafl-0.11.2/src/schedulers/minimizer.rs:268:26
20:     0x5de96e9ba820 - <libafl::schedulers::minimizer::MinimizerScheduler<CS,F,M> as libafl::schedulers::Scheduler>::on_add::h2ff1bedc6a51127c
at /home/alan/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libafl-0.11.2/src/schedulers/minimizer.rs:204:9
21:     0x5de96ea1b1c2 - <libafl::fuzzer::StdFuzzer<CS,F,OF,OT> as libafl::fuzzer::ExecutionProcessor>::process_execution::h691113821321cad0
at /home/alan/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libafl-0.11.2/src/fuzzer/mod.rs:390:17
22:     0x5de96ea1a89e - <libafl::fuzzer::StdFuzzer<CS,F,OF,OT> as libafl::fuzzer::EvaluatorObservers>::evaluate_input_with_observers::hc20781ca70eded51
at /home/alan/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libafl-0.11.2/src/fuzzer/mod.rs:471:9
23:     0x5de96ea19333 - <libafl::fuzzer::StdFuzzer<CS,F,OF,OT> as libafl::fuzzer::Evaluator<E,EM>>::evaluate_input_events::h980ab0fc0b27bab8
at /home/alan/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libafl-0.11.2/src/fuzzer/mod.rs:495:9
24:     0x5de96ea1c70c - libafl::fuzzer::Evaluator::evaluate_input::h4503e43f443b1cb1
at /home/alan/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libafl-0.11.2/src/fuzzer/mod.rs:124:9
25:     0x5de96e930ad1 - libafl::state::StdState<I,C,R,SC>::continue_loading_initial_inputs_custom::hcd27d4cfa4d95ce4
at /home/alan/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libafl-0.11.2/src/state/mod.rs:665:40
26:     0x5de96e931343 - libafl::state::StdState<I,C,R,SC>::load_initial_inputs_custom_by_filenames::h07cee6fed918b687
at /home/alan/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libafl-0.11.2/src/state/mod.rs:638:9
27:     0x5de96e9305b8 - libafl::state::StdState<I,C,R,SC>::load_initial_inputs_by_filenames::h03e6ff95b8b8ad6e
at /home/alan/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libafl-0.11.2/src/state/mod.rs:703:9
28:     0x5de96e9cf7a8 - darthshader::fuzz::h196a74a56f319e51
at /home/alan/darthshader/src/main.rs:410:9
29:     0x5de96e9cca85 - darthshader::main::h7233ff72aaa519ea
at /home/alan/darthshader/src/main.rs:221:5
30:     0x5de96eab641b - core::ops::function::FnOnce::call_once::h0f4f2b07cc69ec71
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/core/src/ops/function.rs:250:5
31:     0x5de96e883e1e - std::sys::backtrace::__rust_begin_short_backtrace::hf4e7800c3143f518
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/std/src/sys/backtrace.rs:154:18
32:     0x5de96e9a5fe1 - std::rt::lang_start::{{closure}}::h6bf73343aceb52ee
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/std/src/rt.rs:164:18
33:     0x5de96f38d320 - core::ops::function::impls::<impl core::ops::function::FnOnce for &F>::call_once::h05c6bccc07d3f6f8
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/core/src/ops/function.rs:284:13
34:     0x5de96f38d320 - std::panicking::try::do_call::ha7775c5d151b7535
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/std/src/panicking.rs:557:40
35:     0x5de96f38d320 - std::panicking::try::hc3e9847a907a671c
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/std/src/panicking.rs:520:19
36:     0x5de96f38d320 - std::panic::catch_unwind::hfd59bc98b3ebc4b9
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/std/src/panic.rs:348:14
37:     0x5de96f38d320 - std::rt::lang_start_internal::{{closure}}::hc28d404ae901ba2f
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/std/src/rt.rs:143:48
38:     0x5de96f38d320 - std::panicking::try::do_call::h3bf19b3691b9641b
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/std/src/panicking.rs:557:40
39:     0x5de96f38d320 - std::panicking::try::h89d7b1c2144c0423
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/std/src/panicking.rs:520:19
40:     0x5de96f38d320 - std::panic::catch_unwind::hf2b61873f94e33f2
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/std/src/panic.rs:348:14
41:     0x5de96f38d320 - std::rt::lang_start_internal::hc6683a15201d56bc
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/std/src/rt.rs:143:20
42:     0x5de96e9a5fba - std::rt::lang_start::hff7787f09a5cbb23
at /rustc/fa724e5d8cbbdfbd1e53c4c656121af01b694406/library/std/src/rt.rs:163:17
43:     0x5de96e9d078e - main
44:     0x7ab97ea2a1ca - __libc_start_call_main
at ./csu/../sysdeps/nptl/libc_start_call_main.h:58:16
45:     0x7ab97ea2a28b - __libc_start_main_impl
at ./csu/../csu/libc-start.c:360:3
46:     0x5de96e86c645 - _start
47:                0x0 -
sample.tar.gz

The version with the docker file and the initial git commit should be basically identical, except for some parts in the README and the license.
I'm PTO this week and don't have access to my machine but I will investigate next week. Could you also provide the<PHONE_NUMBER>_000000 file?
Sorry for all the trouble and thanks for providing feedback. The fuzzer is supposed to be easy to setup and run, but alas, it isn't (yet).

RON file attached.
I am on holiday at the minute as well, though when you're self employed you never truly leave work behind so I am sitting on the beach at the minute writing this.    But ultimately I think there is mileage in your fuzzer mutating these files (some of which are actual WGSL code) to try and find interesting and new crashes.  I've alreay broken DXC a few times as it is.
1728167311_000000.zip

Well, it's a fuzzers job to deal with malformed inputs so expecting it to handle malformed files is a reasonable assumption.
The fuzzer attempts to generate wgsl files from scratch (until the corpus fails to grow for 100 generated samples). Hence an empty seed corpus should not be a problem unless there is an issue with the instrumentation).

hi, I found the code
"wgsl" => {
                let res = std::panic::catch_unwind(|| {
                    naga::front::wgsl::parse_str(&input)
                        .map_err(|e| Error::illegal_argument(e.to_string()))
                });

                let Ok(Ok(module)) = res else {
                    println!("Attempting to parse as AST: {:?}", path.as_ref());
                    let ast = match Ast::try_from_wgsl(input.as_bytes()) {
                        Ok(ast) => ast,
                        Err(_) => Ast::try_from_wgsl("".as_bytes()).unwrap(),
                    };
                    return Ok(LayeredInput::Ast(ast));
                };
                module
            }
            _ => {
                println!("Offending file: {:?}", path.as_ref());
                panic!("file import error");
            }

the fucntion Ast::try_from_wgsl   will return error, so it could cause above panic.
I will continue to research

So there was a small discrepancy between the github version and the artifact eval version after all. How the regression got introduced is beyond me; it should be fixed with commit cdc28b5.  The fuzzer now imports the files from sample.tar.gz without triggering a panic.

Thanks for persevering with this!  I combined the original monkey corpus with the results of leaving the fuzzers hammering away for another week and fed that to your fuzzer.  Looks like its going to work now.
Last few questions I think...  I've got a large amount of compute power to throw at this; several hundred cores.  I presume that scaling this out is as simple as starting multiple instances of the fuzzer on each box, all pointed at the same output directory?  Presumably there is none of the AFL master/slave style requirement?  I assume that there is some random element to the input selection, such that they hammer out different results?  It doesnt look like there is any way to 'resume' a fuzzing session?  So I should start with an empty output directory for each 'run'?  Usually I would periodically minimize/merge my corpus across the nodes to create a new 'starting point' corpus.  Is this a valuable strategy with your fuzzer?  If so I didnt spot any method of minimizing/merging the corpus?

I've been running the fuzzer with the monkey corpus today and it seems to be fuzzing well now.  I'm running $numberofcores copies of the fuzzer all with the same input and output directory set which appears to work.  I note that the fuzzer outputs huge number of duplicated crash files.  Hard to know if this is normal for any fuzzer as most name the crashers as the hash of the contents, naturally deduplicating them.
I have lots of output like this from the running fuzzer which seems good?:
[Testcase #0] run time: 6h-53m-21s, clients: 1, corpus: 14003, objectives: 30, executions: 903251, exec/sec: 36.47, execs_suc: 217448, shared_mem: 49329/212800 (23%), stability: 211766/212800 (99%), execs_err: 661580
Client 000:
0.0445: Scheduler
0.0000: Manager
Stage 0:
0.0000: GetInputFromCorpus
0.3555: Mutate
0.0001: MutatePostExec
0.1046: TargetExecution
0.0758: PreExecObservers
0.1788: PostExecObservers
Feedbacks:
0.0000: ConstFeedback
0.0000: time
0.0000: CrashFeedback
0.0000: exit
0.1147: mapfeedback_metadata_shared_mem
0.1260: Not Measured
[Objective #0] run time: 6h-56m-48s, clients: 1, corpus: 13978, objectives: 65, executions: 731899, exec/sec: 29.30, shared_mem: 49435/212800 (23%), execs_suc: 155620, stability: 211851/212800 (99%), execs_err: 555251
Client 000:
0.0298: Scheduler
0.0000: Manager
Stage 0:
0.0000: GetInputFromCorpus
0.4856: Mutate
0.0001: MutatePostExec
0.0819: TargetExecution
0.0606: PreExecObservers
0.1423: PostExecObservers
Feedbacks:
0.0000: exit
0.0921: mapfeedback_metadata_shared_mem
0.0000: CrashFeedback
0.0000: time
0.0000: ConstFeedback
0.1076: Not Measured
I hacked together a script to process the crashes into a digestible bunch of log files.  I will share it here because maybe someone else will find it useful.  I am not a developer, so coding crimes have probably comitted! Any thoughts re the process I am following to triage this would be useful.  I do have some sanitizer hits when I process the files using the script but nothing exciting so far.
#!/bin/bash
fdupes -N -d . #the fuzzer outputs huge amounts of duplicated crashes so we dedupe here
ls -v | cat -n | while read n f; do mv -n "$f" "$n.ron"; done #rename the files to number.ron because lifter doesnt like the file names the fuzzer outputs
for i in ls | grep ron; do /home/alan/scratch/aedarthshader/buildfiles/darthshader/target/debug/lifter --file $i 2>&1 | tee lift$i.log; done #lift the ron file to wgsl
for i in ls | grep wgsl; do /home/alan/dawn/out/libfuzzerasan/tint $i -o $i.hlsl 2>&1 | tee tint$i.log; done #convert the wgsl file to hlsl
export DXC_PATH="/home/alan/darthshader/harnesses/dxcompiler/DirectXShaderCompiler/out/build/bin/dxc-3.7"
find *hlsl -type f | while read -r f; do
cat ../types | while read -r t; do #target shader types
cat ../years | while read -r h; do #target hardware years
grep () $f | grep void | cut -d " " -f 2 | cut -d ( -f 1 | sort -u | while read -r e; do #entry point seems to be defined as void something() this programmatically finds everything that looks like an entry point
echo "$DXC_PATH -E $e -Gis -Zpr -enable-16bit-types -HV $h -T $t $f 2>&1 | tee $e$f$t$h.log" #run dxc with the various options im interested in
done
done
done
done | xargs -P 56 -I {} bash -c "{}" #hacky way to parallelise the dxc runs
fdupes -N -d . #lots of duplicated log files so I dedupe here
To run a triage I run the following:
mkdir expo
cp output/crashes/* expo
cd expo
cp ../process_results.sh .
./process_results.sh
I'm targeting types: vs_6_6 ps_6_6 cs_6_6 and years 2018 / 2021.  Does this seem sensible?  If other combos are interesting I guess I need to run without 16bit types?
Once completed I run cat *log | sort -u and manually look at the output.  Often grep SUMM *log is probably all we're interested in.  Most of what I see are 'caught errors' and warnings which I dont think are interesting from a security standpoint?

The above errors are due to building afl without llvm-dev libclang-dev clang installed.  After rebuilding with those dependencies I was left with the missing } in dawn/src/tint/fuzzers/tint_wgsl_reader_all_writer_fuzzer.cc.  I dont get how this is missing as the } is in the patch.  Adding it manually allowed the harness to build.

I presume that scaling this out is as simple as starting multiple instances of the fuzzer on each box, all pointed at the same output directory?

The same output directory might work, I generally have a separate output directory per fuzzer instance

Presumably there is none of the AFL master/slave style requirement?

Correct. Note that different fuzzer instance won't import the respective findings of other instances.

I assume that there is some random element to the input selection, such that they hammer out different results?

The is no deterministic mode or anything like that, so different instances of the fuzzer indeed generate different inputs

It doesnt look like there is any way to 'resume' a fuzzing session? So I should start with an empty output directory for each 'run'?

That is correct, resuming a campaign is not supported so far.

Usually I would periodically minimize/merge my corpus across the nodes to create a new 'starting point' corpus. Is this a valuable strategy with your fuzzer? If so I didnt spot any method of minimizing/merging the corpus?

The fuzzer can import .ron files.  If you merge the .ron files from multiple campaigns into a single directory you could start a new campaign with this directory as seed. The initial corpus import should keep only those files that increase coverage.

I have lots of output like this from the running fuzzer which seems good?:

The fuzzer spends quite some time with mutations and in PostExecObservers. Is this a release build of the fuzzer? Other than that, the output looks okay.
The fuzzer indeed finds plenty of crashes that turn out to be duplicates. I generally ran all findings through a separate build of dxc and grouped by PC of the detected ASAN violation. The "script/coding" looks though. What I can recommend as well is compiling dxc with asserts and grouping crashes by the respective assertion violation. Both attempts are just a rough approximation though; neither approach will correctly group by root cause. So what I did in the past was: report a bug, wait for it to be fixed, replay all previous findings and see which inputs continue to reproduce. Repeat until no more crashes remaining. One word of caution though: chrome started to ship a custom version of dxc with asserts enabled, see PR #4 . This implies that the Chrome VRP will no longer consider memory safety issues in dxc unless they also reproduce in a dxc build with asserts enabled.

I'm targeting types: vs_6_6 ps_6_6 cs_6_6 and years 2018 / 2021. Does this seem sensible? If other combos are interesting I guess I need to run without 16bit types?

You can also try other versions of the shader version, anything between 6.2 and 6.6 is a valid target. When in comes to years: at least for chromium/edge only 2018 seems relevant ATM. If you're also interested in bugs that are unrelated to chromium/webgpu other combinations are equally valid.

Most of what I see are 'caught errors' and warnings which I dont think are interesting from a security standpoint?

That is correct, when reprocessing the findings I just look for ASAN crashes and discard the rest.

After rebuilding with those dependencies I was left with the missing } in dawn/src/tint/fuzzers/tint_wgsl_reader_all_writer_fuzzer.cc. I dont get how this is missing as the } is in the patch.

You're right, there seems to be an issue with the patch file. I'll investigate.

BTW if I overlooked any other question in this ticket let me know

You've gone above and beyond my expectations.  Fantastic really :-)  Are you able to share any suggestions for minimizing crashers?  I dont think anybody will entertain my crashes so far as itll be like why yes here is a stack trace for an invalid write/buffer overrun and a 2GB test case.  Even I am lost as to what to do with those crazy things!

Well, I was going to write about testcase minimization in issue #6 but here we go: I can recommend the creduce tool. It has a --not-c mode that works quite well for wgsl files. I found its interface to be a bit strange, but once you get it to work its awesome.

The harness patch file should work again, thanks for pointing this one out.

Just updated the README to contain some more info on crash reduction

How many files to you have in the queue? Furthermore, could you please provide some stats on the on-disk size of the .ron files + the on-disk size of the .metadata files?

I concatenated all of the queue directories into one folder (about 2.2 million files), deduplicated the files (leaving about 250k files), then took the remaining files and started a single thread while watching carefully. Once it said "imported x files from disk" I stopped it and moved the queue directory to be my input and started the process again. The entire procedure took about 10 hours to complete! Therefore I don't have any useful stats at the minute, but I will take a look tomorrow morning.

The number of files is a bit surprising. Did you compile with vanilla AFL++ 4.10c or did you also apply the patch file (for AFL++)?

Good spot!  I actually forgot to apply the patch in my 'setup notes' above as well.  Patch now applied, harnesses rebuilt and run restarted.  Crossing fingers :-)

Thanks.  Stupid question time, but I dont see it mentioned anywhere...  On the first node I installed my afl build for lto mode is failing because I have a conflicting dependency installed.  I understand this well and decided to make sure I avoided the issue on the other nodes.  When running the other nodes, they produce significantly less crash files than the first node.  Looking at the output; the first node; built without lto mode in afl has execs_suc: 1788976 / execs_err: 14858016 in its output.  Whereas the new nodes always report 0 for this metric.  Additionally the first node has many more 'objectives' (whatever that is?) than the newer nodes.
I cant decide whether the enormous amount of crashes along with execs_succ/err output is 'correct' or if the quieter output of the newer nodes with execs_succ/err = 0 is correct?

libAFL implements bug oracles other than crashes (such as detecting differential executions). Hence it calls fuzzer findings "objectives". In the case of this fuzzers, objective and crash can be used interchangeably.
The execs_suc: 1788976 / execs_err: 14858016 looks much better. What we're measuring here is the number of shaders that could be compiled successfully (1788976) and the number of shaders which failed to compile (14858016). This metric is not supposed to be 0; I'm not sure how this is happening.
When re-processing crashes I tend to set ASAN flags such that leaks are ignored.

however the first line of the test case should make the test case invalid for type vs

Do you mean that the validation errors should prevent the crash?

Nah, I tried to parralelise the dxc processing because I've got many cores available.  But it turns out a bunch of the test cases effectively want to allocate infinite quanities of memory and parallelising those causes the box to OOM before any of the safeguards can act.  This probably causes stuff to fail in indeterminate ways.
I havnt got my head around why the 'new' nodes are reporting succ/err to be zero.  The only difference between the afl compilations is LTO mode being disabled on the older box.  But then I spotted you explicitly turn that mode off in the harness compilation.  So I am stumped at the minute.

Turning off LTO wasn't a thought-out decision; the root cause should be addresses. Essentially, just compiled dxc (and tint?) with LTO enabled?