TypeError: Object of type bytes is not JSON serializable

Problem Description
When loading a Flow / Dump from past sniffs it gives this error
Steps to reproduce the behavior:

Open mitm ui
Load a flow / dump of sniff
Error occurs in console

System Information
Addon error: Traceback (most recent call last):
File "mitmproxy\addons\view.py", line 612, in update
File "blinker\base.py", line 266, in send
File "blinker\base.py", line 266, in
File "mitmproxy\tools\web\master.py", line 57, in sig_view_update
File "mitmproxy\tools\web\app.py", line 214, in broadcast
File "json_init.py", line 234, in dumps
File "json\encoder.py", line 199, in encode
File "json\encoder.py", line 257, in iterencode
File "json\encoder.py", line 179, in default
TypeError: Object of type bytes is not JSON serializable
Addon error: Traceback (most recent call last):
File "mitmproxy\addons\view.py", line 571, in response
File "mitmproxy\addons\view.py", line 612, in update
File "blinker\base.py", line 266, in send
File "blinker\base.py", line 266, in
File "mitmproxy\tools\web\master.py", line 57, in sig_view_update
File "mitmproxy\tools\web\app.py", line 214, in broadcast
File "json_init.py", line 234, in dumps
File "json\encoder.py", line 199, in encode
File "json\encoder.py", line 257, in iterencode
File "json\encoder.py", line 179, in default
TypeError: Object of type bytes is not JSON serializable
Addon error: Traceback (most recent call last):
File "mitmproxy\addons\view.py", line 612, in update
File "blinker\base.py", line 266, in send
File "blinker\base.py", line 266, in
File "mitmproxy\tools\web\master.py", line 57, in sig_view_update
File "mitmproxy\tools\web\app.py", line 214, in broadcast
File "json_init.py", line 234, in dumps
File "json\encoder.py", line 199, in encode
File "json\encoder.py", line 257, in iterencode
File "json\encoder.py", line 179, in default
TypeError: Object of type bytes is not JSON serializable
Addon error: Traceback (most recent call last):
File "mitmproxy\addons\view.py", line 565, in request
File "mitmproxy\addons\view.py", line 503, in add
File "blinker\base.py", line 266, in send
File "blinker\base.py", line 266, in
File "mitmproxy\tools\web\master.py", line 50, in sig_view_add
File "mitmproxy\tools\web\app.py", line 214, in broadcast
File "json_init.py", line 234, in dumps
File "json\encoder.py", line 199, in encode
File "json\encoder.py", line 257, in iterencode
File "json\encoder.py", line 179, in default
TypeError: Object of type bytes is not JSON serializable
Addon error: Traceback (most recent call last):
File "mitmproxy\addons\view.py", line 612, in update
File "blinker\base.py", line 266, in send
File "blinker\base.py", line 266, in
File "mitmproxy\tools\web\master.py", line 57, in sig_view_update
File "mitmproxy\tools\web\app.py", line 214, in broadcast
File "json_init.py", line 234, in dumps
File "json\encoder.py", line 199, in encode
File "json\encoder.py", line 257, in iterencode
File "json\encoder.py", line 179, in default
TypeError: Object of type bytes is not JSON serializable
Addon error: Traceback (most recent call last):
File "mitmproxy\addons\view.py", line 612, in update
File "blinker\base.py", line 266, in send
File "blinker\base.py", line 266, in
File "mitmproxy\tools\web\master.py", line 57, in sig_view_update
File "mitmproxy\tools\web\app.py", line 214, in broadcast
File "json_init.py", line 234, in dumps
File "json\encoder.py", line 199, in encode
File "json\encoder.py", line 257, in iterencode
File "json\encoder.py", line 179, in default
TypeError: Object of type bytes is not JSON serializable
Addon error: Traceback (most recent call last):
File "mitmproxy\addons\view.py", line 571, in response
File "mitmproxy\addons\view.py", line 612, in update
File "blinker\base.py", line 266, in send
File "blinker\base.py", line 266, in
File "mitmproxy\tools\web\master.py", line 57, in sig_view_update
File "mitmproxy\tools\web\app.py", line 214, in broadcast
File "json_init.py", line 234, in dumps
File "json\encoder.py", line 199, in encode
File "json\encoder.py", line 257, in iterencode
File "json\encoder.py", line 179, in default
TypeError: Object of type bytes is not JSON serializable
Addon error: Traceback (most recent call last):
File "mitmproxy\addons\view.py", line 612, in update
File "blinker\base.py", line 266, in send
File "blinker\base.py", line 266, in
File "mitmproxy\tools\web\master.py", line 57, in sig_view_update
File "mitmproxy\tools\web\app.py", line 214, in broadcast
File "json_init.py", line 234, in dumps
File "json\encoder.py", line 199, in encode
File "json\encoder.py", line 257, in iterencode
File "json\encoder.py", line 179, in default
TypeError: Object of type bytes is not JSON serializable
Addon error: Traceback (most recent call last):
File "mitmproxy\addons\view.py", line 565, in request
File "mitmproxy\addons\view.py", line 503, in add
File "blinker\base.py", line 266, in send
File "blinker\base.py", line 266, in
File "mitmproxy\tools\web\master.py", line 50, in sig_view_add
File "mitmproxy\tools\web\app.py", line 214, in broadcast
File "json_init.py", line 234, in dumps
File "json\encoder.py", line 199, in encode
File "json\encoder.py", line 257, in iterencode
File "json\encoder.py", line 179, in default
TypeError: Object of type bytes is not JSON serializable
Addon error: Traceback (most recent call last):
File "mitmproxy\addons\view.py", line 612, in update
File "blinker\base.py", line 266, in send
File "blinker\base.py", line 266, in
File "mitmproxy\tools\web\master.py", line 57, in sig_view_update
File "mitmproxy\tools\web\app.py", line 214, in broadcast
File "json_init.py", line 234, in dumps
File "json\encoder.py", line 199, in encode
File "json\encoder.py", line 257, in iterencode
File "json\encoder.py", line 179, in default
TypeError: Object of type bytes is not JSON serializable
Addon error: Traceback (most recent call last):
File "mitmproxy\addons\view.py", line 612, in update
File "blinker\base.py", line 266, in send
File "blinker\base.py", line 266, in
File "mitmproxy\tools\web\master.py", line 57, in sig_view_update
File "mitmproxy\tools\web\app.py", line 214, in broadcast
File "json_init.py", line 234, in dumps
File "json\encoder.py", line 199, in encode
File "json\encoder.py", line 257, in iterencode
File "json\encoder.py", line 179, in default
TypeError: Object of type bytes is not JSON serializable
Addon error: Traceback (most recent call last):
File "mitmproxy\addons\view.py", line 571, in response
File "mitmproxy\addons\view.py", line 612, in update
File "blinker\base.py", line 266, in send
File "blinker\base.py", line 266, in
File "mitmproxy\tools\web\master.py", line 57, in sig_view_update
File "mitmproxy\tools\web\app.py", line 214, in broadcast
File "json_init.py", line 234, in dumps
File "json\encoder.py", line 199, in encode
File "json\encoder.py", line 257, in iterencode
File "json\encoder.py", line 179, in default
TypeError: Object of type bytes is not JSON serializable
Addon error: Traceback (most recent call last):
File "mitmproxy\addons\view.py", line 612, in update
File "blinker\base.py", line 266, in send
File "blinker\base.py", line 266, in
File "mitmproxy\tools\web\master.py", line 57, in sig_view_update
File "mitmproxy\tools\web\app.py", line 214, in broadcast
File "json_init.py", line 234, in dumps
File "json\encoder.py", line 199, in encode
File "json\encoder.py", line 257, in iterencode
File "json\encoder.py", line 179, in default
TypeError: Object of type bytes is not JSON serializable
Addon error: Traceback (most recent call last):
File "mitmproxy\addons\view.py", line 565, in request
File "mitmproxy\addons\view.py", line 503, in add
File "blinker\base.py", line 266, in send
File "blinker\base.py", line 266, in
File "mitmproxy\tools\web\master.py", line 50, in sig_view_add
File "mitmproxy\tools\web\app.py", line 214, in broadcast
File "json_init.py", line 234, in dumps
File "json\encoder.py", line 199, in encode
File "json\encoder.py", line 257, in iterencode
File "json\encoder.py", line 179, in default
TypeError: Object of type bytes is not JSON serializable
Addon error: Traceback (most recent call last):
File "mitmproxy\addons\view.py", line 612, in update
File "blinker\base.py", line 266, in send
File "blinker\base.py", line 266, in
File "mitmproxy\tools\web\master.py", line 57, in sig_view_update
File "mitmproxy\tools\web\app.py", line 214, in broadcast
File "json_init.py", line 234, in dumps
File "json\encoder.py", line 199, in encode
File "json\encoder.py", line 257, in iterencode
File "json\encoder.py", line 179, in default
TypeError: Object of type bytes is not JSON serializable
Addon error: Traceback (most recent call last):
File "mitmproxy\addons\view.py", line 612, in update
File "blinker\base.py", line 266, in send
File "blinker\base.py", line 266, in
File "mitmproxy\tools\web\master.py", line 57, in sig_view_update
File "mitmproxy\tools\web\app.py", line 214, in broadcast
File "json_init.py", line 234, in dumps
File "json\encoder.py", line 199, in encode
File "json\encoder.py", line 257, in iterencode
File "json\encoder.py", line 179, in default
TypeError: Object of type bytes is not JSON serializable
Addon error: Traceback (most recent call last):
File "mitmproxy\addons\view.py", line 571, in response
File "mitmproxy\addons\view.py", line 612, in update
File "blinker\base.py", line 266, in send
File "blinker\base.py", line 266, in
File "mitmproxy\tools\web\master.py", line 57, in sig_view_update
File "mitmproxy\tools\web\app.py", line 214, in broadcast
File "json_init.py", line 234, in dumps
File "json\encoder.py", line 199, in encode
File "json\encoder.py", line 257, in iterencode
File "json\encoder.py", line 179, in default
TypeError: Object of type bytes is not JSON serializable
Addon error: Traceback (most recent call last):
File "mitmproxy\addons\view.py", line 612, in update
File "blinker\base.py", line 266, in send
File "blinker\base.py", line 266, in
File "mitmproxy\tools\web\master.py", line 57, in sig_view_update
File "mitmproxy\tools\web\app.py", line 214, in broadcast
File "json_init.py", line 234, in dumps
File "json\encoder.py", line 199, in encode
File "json\encoder.py", line 257, in iterencode
File "json\encoder.py", line 179, in default
TypeError: Object of type bytes is not JSON serializable
Addon error: Traceback (most recent call last):
File "mitmproxy\addons\view.py", line 565, in request
File "mitmproxy\addons\view.py", line 503, in add
File "blinker\base.py", line 266, in send
File "blinker\base.py", line 266, in
File "mitmproxy\tools\web\master.py", line 50, in sig_view_add
File "mitmproxy\tools\web\app.py", line 214, in broadcast
File "json_init.py", line 234, in dumps
File "json\encoder.py", line 199, in encode
File "json\encoder.py", line 257, in iterencode
File "json\encoder.py", line 179, in default
TypeError: Object of type bytes is not JSON serializable
Addon error: Traceback (most recent call last):
File "mitmproxy\addons\view.py", line 612, in update
File "blinker\base.py", line 266, in send
File "blinker\base.py", line 266, in
File "mitmproxy\tools\web\master.py", line 57, in sig_view_update
File "mitmproxy\tools\web\app.py", line 214, in broadcast
File "json_init.py", line 234, in dumps
File "json\encoder.py", line 199, in encode
File "json\encoder.py", line 257, in iterencode
File "json\encoder.py", line 179, in default
TypeError: Object of type bytes is not JSON serializable
Addon error: Traceback (most recent call last):
File "mitmproxy\addons\view.py", line 612, in update
File "blinker\base.py", line 266, in send
File "blinker\base.py", line 266, in
File "mitmproxy\tools\web\master.py", line 57, in sig_view_update
File "mitmproxy\tools\web\app.py", line 214, in broadcast
File "json_init.py", line 234, in dumps
File "json\encoder.py", line 199, in encode
File "json\encoder.py", line 257, in iterencode
File "json\encoder.py", line 179, in default
TypeError: Object of type bytes is not JSON serializable
Addon error: Traceback (most recent call last):
File "mitmproxy\addons\view.py", line 571, in response
File "mitmproxy\addons\view.py", line 612, in update
File "blinker\base.py", line 266, in send
File "blinker\base.py", line 266, in
File "mitmproxy\tools\web\master.py", line 57, in sig_view_update
File "mitmproxy\tools\web\app.py", line 214, in broadcast
File "json_init.py", line 234, in dumps
File "json\encoder.py", line 199, in encode
File "json\encoder.py", line 257, in iterencode
File "json\encoder.py", line 179, in default
TypeError: Object of type bytes is not JSON serializable
Addon error: Traceback (most recent call last):
File "mitmproxy\addons\view.py", line 612, in update
File "blinker\base.py", line 266, in send
File "blinker\base.py", line 266, in
File "mitmproxy\tools\web\master.py", line 57, in sig_view_update
File "mitmproxy\tools\web\app.py", line 214, in broadcast
File "json_init.py", line 234, in dumps
File "json\encoder.py", line 199, in encode
File "json\encoder.py", line 257, in iterencode
File "json\encoder.py", line 179, in default
TypeError: Object of type bytes is not JSON serializable
ERROR:tornado.application:Uncaught exception GET /flows.json (<IP_ADDRESS>)
HTTPServerRequest(protocol='http', host='<IP_ADDRESS>:8081', method='GET', uri='/flows.json', version='HTTP/1.1', remote_ip='<IP_ADDRESS>')
Traceback (most recent call last):
File "tornado\web.py", line 1702, in execute
File "mitmproxy\tools\web\app.py", line 229, in get
File "mitmproxy\tools\web\app.py", line 126, in write
File "tornado\escape.py", line 75, in json_encode
File "json_init.py", line 231, in dumps
File "json\encoder.py", line 199, in encode
File "json\encoder.py", line 257, in iterencode
File "json\encoder.py", line 179, in default
TypeError: Object of type bytes is not JSON serializable
ERROR:tornado.access:500 GET /flows.json (<IP_ADDRESS>) 93.02ms

Which mitmproxy version are you on?

Which mitmproxy version are you on?

@mhils 6.0.1 the latest downloaded from github

Could you please paste the output of mitmproxy --version?

Could you please paste the output of mitmdump --version?

Mitmproxy: 6.0.1 binary
Python:    3.9.0
OpenSSL:   OpenSSL 1.1.1i  8 Dec 2020
Platform:  Windows-10-10.0.19041-SP0
@mhils

Thanks! This is fixed by 0407ce3. 6.0.2 is just going through our CI pipeline, once https://github.com/mitmproxy/mitmproxy/actions/runs/423683015 is done in about 15 minutes you should be able to upgrade! 😃


Implement go to function definition for Jupyter Notebooks

With CTRL+CLICK

Is there any update on this?

Dup of https://github.com/microsoft/vscode-python/issues/8323


Load local .GLB file on react-native

Hello everyone!
I'm getting a problem when i'm trying to load a locally GLB file on react-native...
Somebody can help-me about it?
My code:

My files tree:

The error:

I suggest. it worked for me.
rename test2.gltf to test2.json
import gltf from "../../../test2"
...
...
const [gltfBlob] = useState(Buffer.from(JSON.stringify(gltf)).toString("base64"))
const [urlBase, setUrlBase] = useState(false)
useMemo(() => void setUrlBase("data:application/octet-stream;base64," + gltfBlob), [gltfBlob])
useMemo(() => urlBase && new GLTFLoader().load(urlBase, (gltf) => setObj(gltf.scene.children[0])), [urlBase])

you cant, threejs loaders are based on browser tech. you need to look into how expo does it, they have forked these loaders.

@theinvensi
Import those:
import usePromise from "react-promise-suspense";
import { Asset } from "expo-asset";

You will need to know uri first:
const getUrl = async () => {
  const asset = Asset.fromModule(require("../assets/grass-no-materials.glb"));
  await asset.downloadAsync();

  return asset.localUri;
};

And then you can do:
  const url = usePromise(getUrl, []);
  const { nodes } = useLoader<GLTFResult>(GLTFLoader as any, url);

Just know that materials currently don't work because GLTFLoader can't make blob out of arrays. I get geometries from GLTFLoader and give them textures manually in the app. Here is related issue: https://github.com/expo/expo-three/issues/144


SxProps Type for sx property.

Actual Behaviour:
Accessing the type definition for the sx prop is convoluted. From my gathering, the only way is to use import { SxProps } from '@material-ui/system/styleFunctionSx/styleFunctionSx';.
Expected Behaviour:
Ideally, a solution like the one @rajzik suggested (import { SxProps } from '@material-ui/system';). Unfortunately, at the moment, this potential solution results in Module '"@material-ui/system"' has no exported member 'SxProps'.ts(2305).
Background:
There are several factors that lead to this issue:

I am using TypeScript (^4.0.3)
I am using Airbnb coding style for ESLint (^7.32.0)

Prevents usage of any type in TypeScript code.
Prevents usage of spread operator into props of components.

I am forced to explicitly write the types the component receives.

Code:
import * as React from 'react';
import { Link, Typography } from '@material-ui/core';
import { SxProps } from '@material-ui/system/styleFunctionSx/styleFunctionSx';

export type CopyrightProps = {
  sx: SxProps;
};

const Copyright: React.FunctionComponent<CopyrightProps> = ({ sx }) => (
  <Typography variant="body2" color="text.secondary" align="center" sx={sx}>
    {'Copyright © '}
    <Link color="inherit" href="/">
      Affinity Patient Portal
    </Link>
    {` ${new Date().getFullYear()}.`}
  </Typography>
);

export default Copyright;

You can use:
import { SxProps } from '@material-ui/system';

Thanks for the feedback.
It's not clear from the current description what the problem is. Please don't just discard the issue template you're prompted with but fill it out. Specifically, actual behavior and expected behavior.

You can use:
import { SxProps } from '@material-ui/system';

Unfortunately, this solution does not work. I get the following error even after adding @material-ui/system to my project's dependencies: Module '"@material-ui/system"' has no exported member 'SxProps'.ts(2305).

Thanks for the feedback.
It's not clear from the current description what the problem is. Please don't just discard the issue template you're prompted with but fill it out. Specifically, actual behavior and expected behavior.

Okay sounds good. Sorry about that. Updated.

@mahyarmirrashed please share a codesandbox illustrating the issue. The import proposed in https://github.com/mui-org/material-ui/issues/27564#issuecomment-890833860  should work.

You can use:
import { SxProps } from '@material-ui/system';

Unfortunately, this solution does not work. I get the following error even after adding @material-ui/system to my project's dependencies: Module '"@material-ui/system"' has no exported member 'SxProps'.ts(2305).

We are using airbnb and other strict rules according to type safety, and we are using import { SxProps } from '@material-ui/system'; with no problem.
This line should be the export. Did you try another version of typescript?

You can use:
import { SxProps } from '@material-ui/system';

Unfortunately, this solution does not work. I get the following error even after adding @material-ui/system to my project's dependencies: Module '"@material-ui/system"' has no exported member 'SxProps'.ts(2305).

We are using airbnb and other strict rules according to type safety, and we are using import { SxProps } from '@material-ui/system'; with no problem.
This line should be the export. Did you try another version of typescript?

Issue solved. This last comment about the TypeScript version is key. yarn create react-app with --typescript flag ships with TypeScript version ^4.0.3. Even adding @material-ui/system with yarn add @material-ui/system to the dependency list would not work with that version of TypeScript.
Solution:
yarn remove typescript and yarn add typescript ensuring that the TypeScript version is ^4.3.5 at the time of writing with the addition of yarn add @material-ui/system to the dependency list inside package.json.
Future steps:
Perhaps adding this as a caveat or note somewhere either in create-react-app or in the MaterialUI notes?

How do I import SxProps with the new @mui import?

@bkinsey808
import { SxProps } from '@mui/system';

@bkinsey808
import { SxProps } from '@mui/system';

this doesn't work anymore. Now you need to import from @mui/material

You can use:
import { SxProps } from '@material-ui/system';

@bkinsey808
import { SxProps } from '@mui/system';

this doesn't work anymore. Now you need to import from @mui/material

This helped me, thank you. Can I ask how you knew how to do this? I've found that to look up types for libraries like MUI, I have to google until I find an issue like this. I tried looking at the docs and of course following component definitions through the code itself, but in this case for example failed to find a type definition for SX or how to import it. How did you know?

For me this import seems like working:
import { SxProps } from '@mui/material';

Try:
import type { SxProps } from '@mui/material';

Пытаться:
import type { SxProps } from '@mui/material';

it's worrked

Using import type { SxProps } from '@mui/material'; slows down test runners like jest which do no tree shaking. All things imported in '@mui/material' are recursively imported before starting a test suite in case such an import is used.
Is there a better way to solve the barrel file import problem?

Try:
import { Theme, SxProps } from '@mui/material/styles';

export type Example = {
  sx: SxProps<Theme>;
};

This works with @mui/material ver5.15.0
import type { SxProps } from '@mui/material'


Change mosquitto: allow_connections_from to <IP_ADDRESS>/0

Hi,
First, thank you for making this great setup project :)
I tried it and couldn't connect to the MQTT broker, and found out that the iptables rules for the MQTT port (1883) had source address field to <IP_ADDRESS> instead of <IP_ADDRESS>/0 (everywhere). This lead to packet drops when attempting to connect to the MQTT broker.
This small change aims to fix that !

Good catch and thanks for fixing it!


Sponsor

Do research about who can sponsor our projects.

What about a christian church? They should be interested into fully translating the bible and they should have the money for it.
Can you make contact with a local priest?

Sponsorship is needed to buy a powerful computer, this is about > 10,000 $, I don't think the church would be interested in investing such amount to realize a machine translation.

What is the computer going to be used for? They will want to know.

Hello @Haleymiranda79
It will be used to further train the current NMT model (ab-ru language pair), and for more pairs.
Next to train STT as well.

Can Google Colab be used? Helsinki University is also interested in making Translation Software and could help.

Can Google Colab be used?

Yes, but it depends.
What we need is access to GPUs to train NMT models for various NLP tasks.
If you want, you could assign this task to yourself, try and talk to the university, see if they are interested in supporting us.


Seeking clickable link upon running server

Description
Web Server is available at //localhost:1313/ (bind address <IP_ADDRESS>)
Without the http: the link is not clickable on the Mac terminal.
Steps to reproduce
 % npm run start after cloning repo
Expected result
Web Server is available at http://localhost:1313/ (bind address <IP_ADDRESS>)
The http: enables the link to be clickable from the terminal.
I am able to get this result when I set: baseurl = "http://example.com"
Actual result
no http: so the link is not clickable.
Environment

><EMAIL_ADDRESS>precheck
> npm version

{
  'doks-child-theme': '0.2.3',
  npm: '7.7.6',
  node: '15.14.0',
  v8: '8.6.395.17-node.28',
  uv: '1.41.0',
  zlib: '1.2.11',
  brotli: '1.0.9',
  ares: '1.17.1',
  modules: '88',
  nghttp2: '1.42.0',
  napi: '8',
  llhttp: '2.1.3',
  openssl: '1.1.1k+quic',
  cldr: '38.1',
  icu: '68.2',
  tz: '2020d',
  unicode: '13.0',
  ngtcp2: '0.1.0-DEV',
  nghttp3: '0.1.0-DEV'
}

><EMAIL_ADDRESS>check
> hugo version

hugo v0.82.1-60618210+extended darwin/amd64 BuildDate=2021-04-20T11:02:50Z VendorInfo=gohugoio

Hey @mroswell, thanks again — I will update this w/ Doks v0.3. For now:
In package.json update the start script to:
"start": "hugo server  -b=\"http://localhost:1313/\" --bind=<IP_ADDRESS> --disableFastRender",

That should work.

On second thought, I won't update the start script. I'd like to keep one place only (less confusion that way) for updating the development baseurl, namely in ./config/_default/config.toml. Set baseurl = "/" (default, recommended), or baseurl = "http://localhost:1313/" (if you like).
Apart from that, I think it s/b fixed on terminal side (also applies to PowerShell btw). Note also that, //localhost:1313/ is clickable when e.g. viewing the page source in Chrome + Firefox.

Fixed w/ Doks v0.3.4


Add 3 Xs (XXX) to the mktemp command argument

To fix the following error:
mktemp: too few X's in template ‘codesigndoc’

Which macOS version did you have this error with?

The result of sw_vers:
ProductName:	Mac OS X
ProductVersion:	10.12.3
BuildVersion:	16D32

Thanks.

It works for me on 10.12.3 O_O
Viktors-MBP:tmp viktorbenei$ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.12.3
BuildVersion:	16D32

bash -l -c "$(curl -sfL https://raw.githubusercontent.com/bitrise-tools/codesigndoc/master/_scripts/install_wrap-xcode.sh)"
 => Creating a temporary directory for codesigndoc ...
 => Downloading version: 0.9.18
 => Downloading codesigndoc from (https://github.com/bitrise-tools/codesigndoc/releases/download/0.9.18/codesigndoc-Darwin-x86_64) to (/var/folders/p9/2zh1v8p545j5m09f47_w1tsr0000gn/T/codesigndoc.JBZQdC2m/codesigndoc) ...
######################################################################## 100.0%
 => Making it executable ...
 => codesigndoc version: 0.9.18
 => Running codesigndoc scan ...

Running mktemp manually also works:
mktemp -d -t codesigndoc
/var/folders/p9/2zh1v8p545j5m09f47_w1tsr0000gn/T/codesigndoc.4K8dEcwH

Ouch!!! Sorry for my carelessness.
I have installed coreutils with homebrew. So, my mktemp is /usr/local/opt/coreutils/libexec/gnubin/mktemp.
I could run mktemp command with full path:
$ /usr/bin/mktemp -d -t codesigningXXX
/var/folders/k4/yvdp21kn5sl81nf6zc_y0jbh0000gn/T/codesigningXXX.I8YNwEho

I'm very sorry. X-(

I'm very sorry. X-(

Don't be! Actually, reading e.g. https://github.com/fisherman/fisherman/pull/8/files I have to agree that including XXX would be a better solution, as that works everywhere, without any issue, while not including XXX can cause issues!
The only thing is, you should add this to the template "wrap" script instead (https://github.com/bitrise-tools/codesigndoc/blob/master/_scripts/install_wrap.sh.template), as that's the one we use to auto generate the other ones! ;)
Additionally, reading a couple of related docs, I'd suggest you to add 5 Xs instead of 3, that seems to be "universally accepted" on all platforms ;)

Or maybe 6 Xs, just to be really really sure :D http://www.gnu.org/software/libc/manual/html_node/Temporary-Files.html ("The last six characters of this string must be ‘XXXXXX’")

Okay, I'm sure. I create some commits again which add 5 Xs to the template file. Please wait a minute.

aahh... Yes, 6 Xs. I got it. Thanks for your advises!

Can I do any other things?
Squash 3 commits, change the title of this PR (3 Xs will make no sense in the future, I think) and etc. I do if you want.

Perfect!
Thank you, I'll re-generate the wrapper scripts in a sec! ;)

Re-generated and pushed, if you run the curl command again you should now get the updated scripts ;)
Thanks again, and Happy Building! :)

Thanks to merge! :smile:

Thanks for the PR! ;)


Add the OpenSSF Scorecard GitHub Action

Motivation
Hello, I'm working on behalf of Google and the Open Source Security Foundation to help essential open-source projects improve their supply-chain security. Given wasm-bindgen importance to integrate Rust Wasm modules and JavaScript, we want to offer our support to improve the project's security posture.
Proposed Solution
Would you consider adopting an OpenSSF tool called Scorecards? Scorecards runs dozens of automated security checks to help maintainers better understand their project's supply-chain security posture. It is developed by the OpenSSF, in partnership with GitHub.
I see wasm-bindgen already follows security best practices, such as release signing and CI testing. Adding Scorecards could help encourage other best practices, such as pinning dependencies and using fuzzing tools. It also checks for actions that increase risk, such as not setting least privilege permissions on tokens and using dangerous code patterns on GitHub Workflows. Overall, it could be a great tool to guide security decisions and show off your current hard work on security practices.
Would you be interested in a PR which adds this Action? Optionally, it can also publish your results to the OpenSSF REST API, which allows a badge with the project's score to be added to its README.
Additional Context
The Scorecard GitHub Action is very lightweight and runs on each change to the repository's main branch. The results of its checks are available on to the project's security dashboard, and include suggestions on how to solve any issues (see examples below). The Action does not run or interact with any workflows, but merely parses them to identify possible vulnerabilities. Over 1600 projects have added the action already.

Hey! Frindly ping here! If you'd like to try out Scorecard security tool, let me know! Otherwise we can close this as not planned for now :)

Personally I am in favor.
@hamza1311, @Liamolucko, what do you think?
Note: I found the relevant checks here: https://github.com/ossf/scorecard?tab=readme-ov-file#scorecard-checks

I'm in favor as well. We should add as many security checks as possible to return the chance of something bad slipping through.
@gabibguti, would like to send a PR? Happy to look at one

Sure!


NEXT-28685 - Move cheapestPrice to ProductDefinition

1. Why is this change necessary?
Dynamic product group with "Price" filter creates error when adding / searching the products in sales channel products.
2. What does this change do, exactly?
Move the cheapestPrice field from SalesChannelProductDefinition up to its parent ProductDefinition
3. Describe each step to reproduce the issue or behaviour.
Setup fresh shopware instance with dummy data
Create dynamic product group with these settings

Go to SalesChannel -> Products -> Add Products -> Product group selection -> search for created product group and select
Error Field "cheapestPrice" in entity "product" was not found

On ProductEntity as this only exists on SalesChannelProductEntity
4. Please link to the relevant issues (if any).
https://github.com/shopware/shopware/issues/4889
5. Checklist

[X] I have rebased my changes to remove merge conflicts
[X] I have written tests and verified that they fail without my change
[X] I have created a changelog file with all necessary information about my changes
[ ] I have written or adjusted the documentation according to my changes
[ ] This change has comments for package types, values, functions, and non-obvious lines of code
[X] I have read the contribution requirements and fulfill them.

The cheapest price was moved from the Product Definition to the Sales Channel Definition due to significant performance and memory consumption concerns.

Good point @OliverSkroblin - was thinking the same, but struggling to find a way to query SalesChannelProducts. Will investigate on that and most likely open a new PR

It's kinda an issue to use SalesChannelProduct here.
The product group selection of Add Products in Sales Channel, tries to filter products which are not yet added to the sales channel. So it normally uses the product api (using the regular product) and filtering for products NOT visible in the saleschannel.

https://github.com/shopware/shopware/blob/d9f22783c794eec466ab8fbcfba4a6185300e9f5/src/Core/Content/Product/SalesChannel/SalesChannelProductDefinition.php#L45
The SalesChannelProduct always adds a visibility filter to the criteria when using the SalesChannelProductRepository, making it hard to create the desired query for the products NOT visible in the saleschannel.


support for -cover

Is it possible to have tests run with -cover (if package is installed)?

Not at the moment, but that sounds like a good addition.
Would you always run with cover (if available) or just when you want to see test coverage?

I guess it would be better to have as an option.


Add support for a new class of "uniquely trackable" tokens

A new class of tokens would be created that supports addressing each token in the collection individually.
For example, Alice creates a new property "AliceToken", which receives property ID 77.  Alice grants 1000 tokens.  These tokens automatically receive identifiers 1 to 1000.  Each of these tokens is then uniquely addressable from a balance and movement perspective (eg 77_454).
Existing features would not support this new class of token, so the "how" of ensuring existing features only work with non-unique properties needs to be considered.
The nomenclature for this new type of property needs to be considered.
An initial proof of concept would provide:

The ability to create properties that allow the tokens to be addressed uniquely
The ability to track the owner (address) of each unique token
The ability to grant tokens of this new property type (^1)
The ability to move individual tokens of this new property type (^2)
The ability to move lists of tokens of this new property type
The ability to move ranges of tokens of this new property type

^1 Note, grants should return a range (eg start and end) noting the unique identifiers that were created (eg granting 10 to a property with 1000 existing tokens should return start:1001, end: 1011)
^2 Note, initially it is suggested that a simple send payload may contain two additional INT64 fields following the property id that denotes the range of token id (eg start: 5, end: 5 to send just token ID 5, or start: 5, end 10 to send range 5 to 10).  The question of lists is open.  The question of what to do by default with a simple send of just a token balance (no range) is also open..
A simple initial test would be:

Alice creates a property A
Alice grants some tokens 1-1000
Alice sends property A_123 and A_321 to Bob
Bob sends property A_123 to Charles

Alice should have a balance of 998 of property A, and listing her tokens should show that she owns A_1 to A_122, A_124 to A_320, and A_322 to A_1000.
Bob should have a balance of 1 of property A, and listing his tokens should show that he owns A_321.
Charles should have a balance of 1 of property A, and listing his tokens should show that he owns A_123.

Just some food for your thoughts:
We need to be smart in regards to the data representation. Consider the worst case: for each token/property there are 2^63-1 identifiers in theory. If each unqiue token/identifier is represented by a single bit, then that's more than 1 exabyte of data. x_x
@marv-engine suggested to use sparse arrays, but the worst case here (user owns MAX tokens) is similar bad, so I think we need to work with ranges right from the start. The downside would be: the more mixed all tokens are, the more space we need.

I see - so if someone granted 9 gazillion tokens, and then we stored those as 9 gazillion unique entities then we would be storing too much data.  I think the key is in working with ranges, and being efficient in how we store data.
I completely agree "Alice owns 1-998 and 1000-5000 & Bob owns 999" is far more feasible than "Alice owns 1, Alice owns 2, Alice owns 3, Alice owns 4, {Alice owns ...}, Bob owns 999".
Do you have much experience working with ranges, and any thoughts on what kind of data types we could use?

I completely agree "Alice owns 1-998 and 1000-5000 & Bob owns 999" is far more feasible than "Alice owns 1, Alice owns 2, Alice owns 3, Alice owns 4, {Alice owns ...}, Bob owns 999".

That's a great example.

Do you have much experience working with ranges, and any thoughts on what kind of data types we could use?

Not really. Just some thoughts: we could represent a range with two INT64 (start, end), and balances, on the lowest level, as set/collection of ranges. The actual balance then would be the sum of ranges. When adding tokens, a new range would be inserted, and when removing tokens, one or more ranges could be removed or replaced. Ideally after each update there would a merge/combination of adjoined ranges (if there are any).

The spec should state that "uniquely trackable" (UT) tokens are indivisible.

Existing features would not support this new class of token

@zathras-crypto are you referring to the current versions of existing features or future versions as well?

If we use the dollar bill metaphor, people pay with cash without any consideration for the serial numbers on each bill. At a minimum, the UI/UX should mimic this, and I don't see why the protocol shouldn't support it directly. If the user doesn't specify which individual tokens he's transacting, then I think the protocol just needs a deterministic mechanism to select the ones affected, e.g. lowest numbered or highest numbered in the affected balance.

We have to prevent double use of UT tokens, so any tx that reserves tokens would have to be enhanced to capture the serial numbers of those tokens so those serial numbers aren't part of the available balance of tokens.

As for initial implementation, I suggest we identify the max number (or order of magnitude) of UT tokens that could be supported in the code. If that number meets the initial application requirements, then we should go with that to start, keeping in mind that we don't want to make any decisions that preclude a larger number in the future.

Just like with blocks in files, over time the sequence of serial numbers for a UT token owned by an address is likely to become fragmented. We need to confirm the implementation can handle the worst case, maybe through a combination of protocol and application.

@zathras-crypto in your ^1 above, the "end" value should be 1010, not 1011.

naming suggestion - call them uniquely identifiable tokens, or UID tokens.

If the user doesn't specify which individual tokens he's transacting, then I think the protocol just needs a deterministic mechanism to select the ones affected, e.g. lowest numbered or highest numbered in the affected balance.

Based on the minimum requirements we don't need to support the normal transaction types, but we are fine with one transaction type to grant/create tokens, and one to move tokens.
Other than that I fully agree, and more to this point soon.

We have to prevent double use of UT tokens, so any tx that reserves tokens would have to be enhanced to capture the serial numbers of those tokens so those serial numbers aren't part of the available balance of tokens.

Internally we use different "buckets" for each balance type (i.e. "available", "reserved", ...), and if we enhance the balance handling for unique tokens, then we'd basically move a bunch of UT from one bucket to another, without running into the risk of some fuzzyness.

As for initial implementation, I suggest we identify the max number (or order of magnitude) of UT tokens that could be supported in the code.

How would you tackle this? It's hard for me to determine which amount is too much.

Just like with blocks in files, over time the sequence of serial numbers for a UT token owned by an address is likely to become fragmented. We need to confirm the implementation can handle the worst case, maybe through a combination of protocol and application.

We can't handle the worst case, see: https://github.com/OmniLayer/omnicore/issues/340#issuecomment-176280069

A few notes about what I had in mind: I believe we might support UID tokens with all transaction types with moderate adjustments:
Currently all balance/tally related updates go through update_tally_map, so this is the layer we need to tackle. As Marv already proposed: if we come up with a deterministic rule set for movements of unspecified batches of tokens, we could support normal transaction types as well.
For example, Alice owns [0-9, 100-125, 300-350] and wants to send 15 UID tokens via a normal simple send to Bob, then we could take 5 tokens either from the beginning or the end, so Alice ends up with [0-9, 100-125, 300-345] and Bob ends up with [345-350].
1) Refine balance operations
We'd need more specific and atomic balance operations to:

Create new tokens (e.g. when creating properties, granting tokens, ...)
Destroy tokens (e.g. when burning STO fees, revoking tokens, ...)
Transfer tokens (e.g. when sending tokens, ...)
Change the state of tokens (e.g. when reserving tokens, ...)

This could be adopted right now without tackling unique tokens at all, simply by wrapping update_tally_map with something like this.
Furthermore, to support sending a single unique token or a specific range of unique tokens, we'd also need a new balance operation to:

Transfer specific collection/range of tokens

2) Update global tally map to support unique tokens
The backend needs to be adjusted to handle unique tokens. I'm not yet sure how exactly, and it's probably not so simple. @zathras-crypto pushed already a new balance backend, which looks very, very handy. It could be thinkable to handle all balances as if they were uniquely trackable, and it would be a change on the lowest levels (i.e. CMPTally, mp_tally_map, ...).
3) Introduce a new token type
Currently we have the token types:

1: indivisible
2: divisible

And I propose to add:

5: trackable, indivisible
6: trackable, divisible

The normal transaction types would only care whether tokens are indivisible or divisible and use the deterministic rule set for moving tokens, while a new transaction type to move a specific range of tokens would require the trackable property.
4) Add new transaction type to move unique tokens
As previously discussed: there should be one or more new transaction types to move a specific collection of unique tokens. This new transaction type only works with trackable tokens.
5) Add unique token specific balance retrievals
Only for trackable tokens, it should be possible to retrieve balances on a unique token level, while retrieving balances of non-trackable tokens should not show exactly who owns which tokens.

These are just some ideas, and I'm curious what you guys think. :)

I believe we might support UID tokens with all transaction types with moderate adjustments:

I'm concerned about "taking on too much" here.  The principle of unique tokens is that "each token in a collection is different".  I don't see how we can support all existing transaction types fundamentally when those existing transaction types were designed to operate on "each token in a collection is the same" principle.  They seem quite different principles.
Let's take MetaDEx for example, say I want to trade 5 ZathrasCoins, but ZathrasCoin[1] is worth ten times as much as ZathrasCoin[2] - it just doesn't seem to 'fit' with existing code.  With 'default' rules about automatically using the top or bottom tokens from the collection, doesn't that open the door to "I didn't mean to do that"?  I've always liked the fact Omni requires you to be explicit about what you want to do :)

Update global tally map to support unique tokens

This is where I'm really having a hard time, because I don't see how we can use something like the tally map for unique tokens.  A tally by definition is just a sum (eg "tally up the votes", or "tally up the income" etc - it's just an amount, not a collection of unique items).  The requirement to use ranges to keep the addressable space down is another complicating factor.

Introduce a new token type:
5: trackable, indivisible
6: trackable, divisible

I support this, sure :)

Add new transaction type to move unique tokens

I was actually kind of thinking just a new version of simple send, except this time it supports a start and end token ID instead of an amount field (eg send {100,200} of prop X).  So v0 simple sends would be used for existing token types, and v1 simple sends would be used for unique token types.
Just a thought, we could have an entirely new send type also - but I'm not sure the best approach.

Only for trackable tokens, it should be possible to retrieve balances on a unique token level, while retrieving balances of non-trackable tokens should not show exactly who owns which tokens.

Agreed here - the levelDB stuff I did had a helper "GetUniqueTokenOwner" which would find the address that owns a particular token.  The reverse also needs to be available (listing which tokens an address owns).

These are just some ideas, and I'm curious what you guys think. :)

Sure sure, and please don't think I'm being negative - I'm just really struggling to find a model to move towards implementation.  I know @CraigSellars is really keen on a PoC soon, and whilst I've played with a few ideas, the levelDB stuff was the only tangible thing that I came up with to work with unique tokens - not to say that it's the right way to go - just that it's all I've been able to come up with so far hehe :)
Thanks dude

Fair enough, I see your points. I'm probably overthinking this, and since we're just tackling a PoC, the simpler the better, otherwise we probably open Pandora's box. :)
My only concern with this is that I assume we might start to reinvent the wheel at some point, because more and more transaction types are needed, and we then end up with two systems within one.

[MetaDEx] just doesn't seem to 'fit' with existing code. With 'default' rules about automatically using the top or bottom tokens from the collection, doesn't that open the door to "I didn't mean to do that"?

Yeah, the idea would be to take the first/last, and in this case it would be really unhandy. For the creation transactions (e.g. crowdsales, creating fixed supply tokens, creating managed tokens) it would be a nice way to distribute the initial "blank" tokens though.

Just a thought, we could have an entirely new send type also - but I'm not sure the best approach.

I tend to prefer a new transaction type, especially if the normal and unique token systems are completely seperate.
On the other hand, this raises also the question about issuance: initially I thought we might extend the creation transactions and issue trackable tokens, if they are created with type 5 or 6, but now I'm not longer sure.

Sure sure, and please don't think I'm being negative - I'm just really struggling to find a model to move towards implementation.

Don't worry, and thanks for your feedback. I agree with you here now, and we should prioritize a proof of concept first. We can always refine and update everything later, if we needed. :)

It would be nice, if we could define what we need and are going to do.
In particular, what we need to decide: do we want to use:

the old transaction types with a version pump (e.g. simple send v1, issue managed v1, ...), or rather new transaction types (e.g. send unique, issue unique managed, ...)?
are we going with new property types (e.g. unqiue and divisible, ...)?

I think with @zathras-crypto great basis, which looks awesome, with a few potential optimiziations here and there (and reorg support!), getting a PoC out is pretty straight forward once those questions are answered.
As a blue print for adding new transaction types, the send all PR could be used: https://github.com/OmniLayer/omnicore/pull/155/files

Sorry for delayed response bud, I've been away this weekend...

My only concern with this is that I assume we might start to reinvent the wheel at some point, because more and more transaction types are needed, and we then end up with two systems within one.

That's a really good point, and I completely agree.  We do not want to end up with two systems in one.

Yeah, the idea would be to take the first/last, and in this case it would be really unhandy. For the creation transactions (e.g. crowdsales, creating fixed supply tokens, creating managed tokens) it would be a nice way to distribute the initial "blank" tokens though.

My understanding at this stage is that the requirements are based on token management, so for the time being only managed issuance is required (though fixed would be simple enough too).  Crowdsales we could look at later I think.

I tend to prefer a new transaction type, especially if the normal and unique token systems are completely seperate.

I'm happy with a separate transaction type for sending unique tokens, though as per your earlier comment about systems within systems I'm not sure the normal and unique systems should be entirely separate.  Though if we did separate them perhaps the simplest model is to have an additional ecosystem.

It would be nice, if we could define what we need and are going to do.

Sure sure, I've just kind of been playing with ideas - we'll need to go to something structured at some point I'd wholeheartedly agree!  Basically I've just been trying to deliver the items for the PoC I listed in the OP in some form of functional manner, which we can then work on refining.

In particular, what we need to decide: do we want to use: t
the old transaction types with a version pump (e.g. simple send v1, issue managed v1, ...), or rather new transaction types (e.g. send unique, issue unique managed, ...)?
are we going with new property types (e.g. unqiue and divisible, ...)?

My votes in this area are:

New types (5 and 6) for unique tokens
Create managed transaction needs no changes to packet structure to support unique tokens so doesn't really need a version change & thus should be re-used
Grant transaction needs no changes to packet structure to support unique tokens so as above
Simple send I am ambivalent on whether we go a new version or a new transaction type

I think with @zathras-crypto great basis, which looks awesome, with a few potential optimiziations here and there (and reorg support!), getting a PoC out is pretty straight forward once those questions are answered.

Thanks dude...  FYI I'm getting close to a functional PoC based on my levelDB stuff which I can push up to your repo when it's functional over the next day or two.  I figured if I at least gave us a starting point we could rip and replace pieces as we saw fit (even replace it all if we need to) but at least we'd have a base to collaborate from :)
What do you think?

***** Copy-OfflineCharts - BackupRH080216.ps1

Foreach ($Location in $Content)
***** COPY-OFFLINECHARTS.PS1

$IgnoreProcessingList = @()

Foreach ($Location in $Content)
*****

***** Copy-OfflineCharts - BackupRH080216.ps1

$copyState = 0
$deleteState = 0

Foreach ($Chart in $ChartList)
***** COPY-OFFLINECHARTS.PS1

Foreach ($Chart in $ChartList)
*****

***** Copy-OfflineCharts - BackupRH080216.ps1
{
    $filename = $Chart.FullName
***** COPY-OFFLINECHARTS.PS1
{
    $copyState = 0
    $deleteState = 0
    $filename = $Chart.FullName
*****

***** Copy-OfflineCharts - BackupRH080216.ps1

    Write-Host "$filename will be copied to $Destination"
***** COPY-OFFLINECHARTS.PS1

    if ($IgnoreProcessingList.Contains($Ward))
    {
        Write-Host "$filename is being skipped due to failure with $Ward NUM PC.
"
        continue
    }

    Write-Host "$filename will be copied to $Destination"
*****

***** Copy-OfflineCharts - BackupRH080216.ps1
            Remove-Item $filename
            $remoteItem = $Destination + "\" + $Chart.BaseName
            Remove-Item -Path $remoteItem
***** COPY-OFFLINECHARTS.PS1
            Remove-Item $filename
            $remoteItem = $Destination + "\" + $Chart.Name
            Remove-Item -Path $remoteItem
*****

***** Copy-OfflineCharts - BackupRH080216.ps1

}
***** COPY-OFFLINECHARTS.PS1

    If($copyState -gt 0)
    {
        $ticketNo = Log-Ticket("Unable to transfer offline mediaction charts to
share on $Ward NUM PC")
        Send-SMSMessage("$Ward NUM PC is down, ticketno $ticketNo")
        $IgnoreProcessingList += $Ward
    }

    If($deleteState -gt 0)
    {
        $ticketNo = Log-Ticket("Unable to delete offline mediaction charts to sh
are on $Ward NUM PC")
        Send-SMSMessage("Offline Chart delete failed on $Ward computer, ticketno
 $ticketNo")
    }
}
*****

***** Copy-OfflineCharts - BackupRH080216.ps1
If($copyState -gt 0)
{
    $ticketNo = Log-Ticket("Unable to transfer offline mediaction charts to shar
e on $Ward NUM PC")
    Send-SMSMessage("$Ward NUM PC is down, ticketno $ticketNo")
}

If($deleteState -gt 0)
{

    $ticketNo = Log-Ticket("Unable to delete offline mediaction charts to share
on $Ward NUM PC")
    Send-SMSMessage("Offline Chart delete failed on $Ward computer, ticketno $ti
cketNo")
}
***** COPY-OFFLINECHARTS.PS1
*****

Bad copy/paste, apologies.

Trying to help out an old colleague, don't think he would be too happy about me pasting his scripts into GitHub hehehe :)

Just a note here, while working this some more I wonder whether supporting divisible types is even feasible when working with unique tokens?  If a unique token is divisible, can two parties own "half" a unique token each?
For now I'm just going to focus on indivisible.

I'm happy with a separate transaction type for sending unique tokens, though as per your earlier comment about systems within systems I'm not sure the normal and unique systems should be entirely separate.

Our logic often checks only the transaction type, but not the transaction version (e.g. in the UI). This is mostly the reason I suggested a new type instead of a version bump. I agree that the systems should not be completely seperate (if possible).

Bad copy/paste, apologies.

No worries. :)

Just a note here, while working this some more I wonder whether supporting divisible types is even feasible when working with unique tokens? If a unique token is divisible, can two parties own "half" a unique token each?

That's a fair point. I always considered divisibility as hint for the UI, but it really makes no sense. :)

What do you think?

Sounds perfect!

@zathras-crypto @dexX7 @achamely and anyone else, here's my proposal for representing each range of UID tokens in transaction messages. It's based on the chat conversation during yesterday's All Hands mtg. The goal is to use as few bytes as possible to represent the range. Each unsigned integer value is represented in the fewest number of bytes possible.
Notes:

A range can include 1 or more valid UIDs for the referenced UID token, e.g. UID 3, UIDs 27-1234.
A UID and the number of UIDs in the range can be represented as an unsigned integer in 1 to 8 bytes.

The first byte contains two 4-bit values:

the lower 4 bits are the number of bytes, SB, (1 to 8) required to represent the starting UID
the upper 4 bits are the number of bytes, IB, (0 to 8) required to represent the incremental number of UIDs in the range, beyond the starting UID.

The next SB number of bytes contains the starting UID.
The next IB number of bytes contains the incremental number of UIDs to reach the end of the range. If just a single UID is in this range, then IB is 0 and this incremental number field is not in the message.
Examples:

1 to 1

byte 1: 1 (IB = 0, SB = 1)
byte 2: 1 (starting UID)
no bytes for the increment because IB = 0

256 to 256

byte 1: 2 (IB = 0, SB = 2)
bytes 2,3: 256 (starting UID, a 2-byte integer)
no bytes for the increment because IB = 0

251 to 255

byte 1: 17 (IB = 1, SB = 1)
byte 2: 251 (starting UID, a 1-byte integer)
byte 3: 4 (incremental UIDs to reach the end of the range, a 1-byte integer)

50,000 to 2,000,000

byte 1: 50 (IB = 3, SB = 2)
bytes 2,3: 50000  (starting UID, a 2-byte integer)
bytes 4,5,6: 1950000 (incremental UIDs to reach the end of the range, a 3-byte integer)

50,001 to 50,200

byte 1: 18 (IB = 1, SB = 2)
bytes 2,3: 50001 (starting UID, a 2-byte integer)
byte 4: 199 (incremental UIDs to reach the end of the range, a 1-byte integer)

I feel like these are two different topics:

how does the message format look like?
can we compress numbers?

The latter question is a broad one and I'd really like the idea of hovering the water, e.g. with the new "send list of unique ranges" transaction type. It would be great, if we use something that could be adopted for other transaction types as well.

Payload compression I would also like to attack "holistically" rather than per message.  We should have a helper function that any payload can be pushed into regardless of content which returns a compressed payload for attaching to a transaction.
Basically a "Class D" that supports payload compression (and potentially multiple payloads per transaction, but that's another topic) I think is the way to go to maximize efficiency of data storage within transactions.
I do agree that we also need to be efficient within the payload but if we stay with fixed size integers (like we have been doing before), any compression we add in holistically is going to have no problems compressing a bunch of leading zeros.
Note, since UT stuff is all testnet only, we can add in compression and use it as a test case for sure :)

To summarize, we basically want either one or two new transaction types:

send a list of unique tokens: [1, 3, 5, 6, 7, ...]
send a list of unique token ranges: [1-1, 3-3, 5-7, ...]

For the message format, without anything facy, we probably need to push the number of elements to come:
[version] [tx type] [elements to come] [element 1] [element 2] [element 3] ...

Whereby an element in the first case would be a unique token identifier, and in the second case a range (consisting of two numbers).
For the first case, it could look like this:
UINT16 UINT16 UINTxx INT64 INT64 INT64 ...
^      ^      ^      ^     ^     ^ element 3
|      |      |      |     | element 2
|      |      |      | element 1
|      |      | elements to come
|      | tx type
| version

And for the second:
UINT16 UINT16 UINTxx INT64 INT64 ...
^      ^      ^      ^     ^
|      |      |      |     | range end/...
|      |      |      | range start
|      |      | elements to come
|      | tx type
| version

Marv proposed a new approach to encode numbers, by prepending the number of bytes of a number to come for each range.
My point was just: instead of using a new approach tailored to these transaction types, it would be great, if we could use something that might be used for other transaction types in the future as well. Just some examples:
Compress the whole payload:
COMPRESS(UINT16 UINT16 UINTxx INT64 INT64 ...)

Or use some variable length encoding approach:
VARINT(UINT16) VARINT(UINT16) VARINT(UINTxx) VARINT(INT64) VARINT(INT64) ...`

My other point was: compression/smarter number representation and the UIT stuff seem like two different topics, and while I'd welcome to tackle this, I'm not really a fan of a tailored approach, so to speak, which is only useful for the "send list of ..." transaction types. :)
But as Z mentioned: it's testnet, and this is a proof of concept, so we can test anything, including Marv's scheme. :)

Here is a method to reduce the number of bytes required to represent the number of tokens and a token UID in a transaction. This method can be used wherever the number of tokens or a token UID is represented.
This approach takes advantage of the fact that both the number of tokens and a token UID is never < 0 (and maybe never < 1). So we can use the sign bit as an indicator that what would be the high order byte of an 8-byte signed integer instead contains a count of the number of following bytes that represent the value.

Integer Value
Indicator Byte
# Value Bytes
# Bytes Used
# Bytes Saved

0
1000 0000
0
1
7

1 to 255
1000 0001
1
2
6

256 to 65,535
1000 0010
2
3
5

65,536 to 16,777,215
1000 0011
3
4
4

16,777,216 to 4,294,967,295
1000 0100
4
5
3

4,294,967,296 to 1,099,511,627,775
1000 0101
5
6
2

1,099,511,627,776 to 281,474,976,710,655
1000 0110
6
7
1

= 281,474,976,710,656 | none | 8 | 8 | 0

To Read a Numeric Value
Examine the high order bit in what would be the high order byte of an 8-byte signed integer. That's the sign bit. For the number of tokens or a token UID, negative numbers are not allowed.
If that bit = 1 then the low order 3 bits in that byte contain the number (0-6) of following bytes that represent the numeric value.
Else if that bit = 0 then that byte is the high order byte of an 8-byte signed integer representing the numeric value.

There are 4 unused bits in the Indicator Byte, but I haven't thought of a way to make use of them.

Thanks guys...
I think I may be in the minority here, but I am concerned that we're making this overly complex for the sake of saving a byte here and there (overly complex for the user and overly complex from a coding standpoint).
I agree there are minor efficiencies we can achieve if we engineer the payload for the exact specific use cases, but I disagree this is the right thing to do...  To be max efficient we would need to structure the packet to the case, having to provide for example:

omni_senduniquesingletoken (a single token ID)
omni_senduniquesingletokenrange (a start and end token ID)
omni_senduniquelistofsingletokens (a list of single token IDs)
omni_senduniquelistoftokenranges (a list of start and end token IDs)

On the opposite side, I believe it is simpler to simply have:

omni_sendunique (which would do everything)

This may cost us an extra byte or two here and there, but avoids lots of over-engineering IMO.  The packet structure could simply be:
UINT16 UINT16 UINTxx INT64 INT64 INT64 ...
^      ^      ^      ^     ^     ^ second range start
|      |      |      |     | first range end
|      |      |      | first range start
|      |      | number of ranges
|      | tx type
| version

This covers all use cases:

Send unique token X:
number of ranges = 1, first range start & end = X
Send unique token range X-Y:
number of ranges = 1, first range start = X, first range end = Y
Send list of N tokens:
number of ranges = N, first range start & end = X1, second range start & end = X2, third range start & end = X3 and so on,,,
Send list of N token ranges:
number of ranges = N, first range start = X1, first range end = Y1, second range start = X2, second range end = Y2, third range start = X3, third range end = Y3 and so on...

I do agree that at first look we may not necessarily be fully optimized here, but we keep things simple (which I believe is a good engineering principle hehe).
Of course there are arguments against, for example when sending single tokens we are setting start and end to the same value, thus duplicating data we are storing in the packet - but since we're going to start compressing payloads (all payloads I vote) anyway that duplicated data would be removed in the compression stage.

Compress the whole payload

This is where my vote is - payload compression should be done across all payloads, I am quite against doing payload encoding/decoding differently depending on the type of transaction - it's just going to get very messy - I view the next transaction class as having compressed paylaod full stop (not special payloads for special tx types).
Bit of a ramble hehe, and the ideas here are great ideas, but I just think we're heading down a path (a bunch of stuff that's unique to a single transaction type) that I don't think is in our long term interests...
Just my 2 cents :)

If payload compression does the trick, then I'm all for it, as long as we understand what the limits are and what happens if we exceed those limits.

To be max efficient we would need to structure the packet to the case, having to provide for example: ...

Actually I think having very specific transactions should be the end goal, and I'd welcome, if we untangle the protocol like this in the long run. It's similar to the overhauled MetaDEx transactions, which were split into more than one.
However, this (and also compression and alike) are low level details, something we can change under the hood without letting the end user notice (well, mostly), and for a proof-of-concept we can of course stick to a simple "one for all" transaction. :)

Did any UITs get pushed out in 2016 or were the first ones deployed in 2019/20?
I'm doing a little Historical NFT digging, and @CraigSellars was kind enough to point me here.

Hey @adamamcbride,
thanks for your research!
While there was a lot of discussion about the topic, nothing was pushed into production.


Support loading ensemble from pandas and dask dataframes

Adds the loader functions Ensemble.from_pandas and Ensemble.from_dask_dataframe. This allows the user to provide TAPE with their own Pandas or Dask dataframes constructed via their own preferred sources and I/O as outlined in issue #223.
Ensemble.from_source_dict and Ensemble.from_parquet were simplified with some of their functionality moved into Ensemble.from_dask_dataframe.

Will add an additional test to bump up the code coverage, but I wanted it to be ready for a first review pass.


Add --id to "kontena master token current"

Fixes #3072
(Perhaps could wait for #2763)

Maybe also write some specs for the kontena master token commands?

Maybe also write some specs for the kontena master token commands?

Issue: #3151


Support multi-threading

Even though it sounds nice, I would consider internal threading as an anti-feature. Threads don't scale modern applications use a task-based method for concurrency. That said, creating a library that can be acted upon in a concurrent fashion is very useful. But the mechanism shouldn't be threads. That should be up to the user. It would feel like jab from the side if a library spawns threads internally and I'm already running a thread pool that optimally pins its threads to cores.
For example, our use case of building a telemetry engine over structured data involves hundreds of filters being created and queried in parallel. The bottleneck is not making insertion or querying faster of each single filter instance. Given this embarrassingly parallel setup, much more important to us is a merge operation, so that we can roll up multiple filters—LSM-style. If this would be possible with Taffy filters, then you'd solve the number one challenge with tree-structured hierarchical filter structures, which is provisioning the size of the non-leaf instances.

Thanks for your thoughts!
Taffy cuckoo filters do support a Union operation - is that not what you're looking for?

Taffy cuckoo filters do support a Union operation - is that not what you're looking for?

That's great, I wasn't aware of it. Looking at the implementation, it seems also that there are no strict requirements in terms of sizing, as it would be the case for Bloom filters.
Mechanically, TBFs should support union as well by performing it piece-wise, no? I would only be worried about the FP-rate getting out of control due to oversubscribing of bits in each component that would otherwise be prevented by "organic" growth.
And MTCFs? Would it be possible to support Union also? Or are the subtables making this impossible?

With TBFs it should be possible by doing bitwise OR on the levels, just like you surmise.
With MTCFs Union should also be possible. The key operation if iterating over an MTCF, performing the permutations in reverse, to reconstruct as many bits of the original key as possible.


how to use Distiller?

hi, I read the paper 'DeepAID: Interpreting and Improving Deep Learning-based
Anomaly Detection in Security Applications'. It is a wonderful work!
I try to use the Distiller, however, I did not find any code about Distiller.
Thank you so much if any advance!!!

Hello, I have the same problem. Have you solved the problem of the distiller？

Distiller is preliminary and only supports tabular data now. We are planning to expand our work to the journal version, including the expansion of the applicability of Distiller. At that time, the code of istiller will be updated in another repo.
If you really need help, please email me, I'd be happy to help.


[Question] Is DisplayPort bandwidth monitoring possible?

I have 3 displays running through MST over USB C (Steam Deck). Doing raw calculations this would be 24.87 Gbps before compression.
DisplayStream Compression is most definitely being used to lower this requirement.
Are there any metrics available to see in real-time what the actual bandwidth is?

I see this mention in the kernel docs but doesn't seems to exist on the running system
https://www.kernel.org/doc/html/v6.6/gpu/amdgpu/driver-misc.html?highlight=pcie_bw#pcie-bw

Are there any metrics available to see in real-time what the actual bandwidth is?

AFAIK, there are no such metrics.
I recommend opening an issue about it on drm/amd.


[Bug] fileReplacements configuration can not replace html files

Versions
Angular CLI: 6.0.0
Node: 8.11.1
Angular: 6.0.0
Windows 7

Repro steps
Reproduce git: https://github.com/idododu/ng6-filereplacements-bug

Step 1: add fileReplacements config in configurations block of angular.json

           "project-b": {
              "fileReplacements": [
                {
                  "replace": "src/environments/environment.ts",
                  "with": "src/environments/environment.project-b.ts"
                },
                {
                  "replace": "src/app/home-a/home-a.component.html",
                  "with":    "src/app/home-b/home-b.component.html"
                },
                {
                  "replace": "src/app/home-a/home-a.component.ts",
                  "with":    "src/app/home-b/home-b.component.ts"
                }
              ]
            }

Step 2: npm run project-b to start dev server

Observed behavior
home-a.component.ts was replaced correctly.

home-a.component.ts in chrome source

home-a.component.ts in my source code

home-a.component.ts was not replaced

home-a.component.html in chrome source

module.exports = "
\r\n  home-a works!\r\n
\r\n"

home-a.component.html in my source code

  home-a works!

home-b.component.html in my source code

  home-b works!

Desired behavior

Expected to see
content of  home-a.component.html should be replaced by home-b.component.html since i have configed in angular.json

{
                  "replace": "src/app/home-a/home-a.component.html",
                  "with":    "src/app/home-b/home-b.component.html"
                },

Usecase
Suppose we got two projects: Project A and Project B.
They share the most of my source codes except HomeComponent.
And I would like to put the codes in the same repo, and build different projects through environments configuration.

Mention any other details that might be useful (optional)
Another point: can we support glob expressions in fileReplacements configuration?
{
  "replace": "src/app/home-a/*/**",
  "with": "src/app/home-b/*/**"
}

Confirmed. This doesn't work for me either:
"fileReplacements": [
                {
                  "replace": "src/app/app.component.html",
                  "with": "src/app/som-other.html"
                }
              ]

FYI, this has been addressed and will be part of the 6.1 release.
To look at that change, please refer to this PR: angular/devkit#887

Just for reference: https://github.com/angular/devkit/issues/885 explains:

Currently, the file replacement functionality works only for TS files. More specifically - it works only for files that are part of the webpack compiler host.

The above-mentioned https://github.com/angular/devkit/pull/887 apparently fixes that; can't wait for 6.1 now ;-)
(And indeed, for me .scss files are not copied either.)

scss files are indeed not copied. Also not working with 6.1.0-beta.2. Any idea if it will be?

FYI, not working in 6.1.0-rc.0 either

Issue still happening in this version: "@angular/cli": "6.1.5".

Issue still happening in this version: "@angular/cli": "6.1.5".
"configurations": {
"production": {
"optimization": true,
"outputHashing": "all",
"sourceMap": false,
"extractCss": true,
"namedChunks": false,
"aot": true,
"extractLicenses": true,
"vendorChunk": false,
"buildOptimizer": true,
"fileReplacements": [
{
"replace": "src/environments/environment.ts",
"with": "src/environments/environment.prod.ts"
}
]
}
}

@luillyfee @georgefam did you make sure you've updated @angular-devkit/build-angular as well ?

@suau I can no longer reproduce this issue. Also I cannot remember how I did make it work, maybe is how you are pointing out.

Any updates on this issue? Should it be fixed in 6.1.5?
Im trying to replace a constants.scss (with constants.newconfig.scss) file that I import at the top of my root styles file. But none of the newconfig styles are pulling through.
The strange thing is if I use this method with a component specific scss file it works perfectly. From what I can see its only with the root style files.
Here's a stackoverflow link to my question.
Im using 6.1.5

@developcodeza-matthew  there is not link in your answer!

@luillyfe My bad 🙈 Should be there now

I'm facing the same issue with CLI 6.2.2

@iki789 You save my life . Thanks Bro.

Is it possible to do dynamic file replacements?
I'm trying to build two different UI's from the same App. So i have two different html files.
          "fileReplacements": [
            {
              "replace": "**.template1.ts",
              "with": "**.template2.ts"
            }

It's throwing me an error. Is this possible?

using angular ^6.0.9 & angular/cli ^6.0.8, updated @angular-devkit/build-angular in devDependencies in package.json to ^0.8.9. seems to work as expected with environments and all

with @angular/cli 7.3.0 html files are not replaced.

Replacing html files is good feature to have. Very strange that the angular team hasn't addressed this issue.

Has anyone got a fix for this yet?

UPDATE: if updating an existing project it's not enough to update the global and project-local @angular/cli version, you also need to update the @angular-devkit/build-angular package to 0.7.0-rc.0.
It is a separate package.
npm i -g<EMAIL_ADDRESS>cd your-project-folder
npm i -D<EMAIL_ADDRESS><EMAIL_ADDRESS>
@robindijkhof

@AntonisFK @MattCMcNabb did you make sure yo update to the newest version as well ? works fine for me.

Has anyone tried to replace app.module.ts?

Have tried to replace app.module.ts with app.module.promoter.ts, but get error, that AppComponent is in 2 modules.
ERROR in : Type AppComponent in C:/Users/santa/WebstormProjects/theBox/src/app/app.component.ts is part of the declarations of 2 modules: AppModule in C:/Users/santa/WebstormProjects/theBox/src/app/app.module.ts and AppModule in C:/Users/santa/WebstormProjects/theBox/src/app/app.module.promoter.ts! Please consider moving AppComponent in C:/Users/santa/WebstormProjects/theBox/src/app/app.component.ts to a higher module that imports AppModule in C:/Users/santa/WebstormProjects/theBox/src/app/app.module.ts and AppModule in C:/Users/santa/WebstormProjects/theBox/src/app/app.module.promoter.ts. You can also create a new NgModule that exports and includes AppComponent in C:/Users/santa/WebstormProjects/theBox/src/app/app.component.ts then import that NgModule in AppModule in C:/Users/santa/WebstormProjects/theBox/src/app/app.module.ts and AppModule in C:/Users/santa/WebstormProjects/theBox/src/app/app.module.promoter.ts.
And there are no references to the app.module.promoter.ts in code at all.

I never had this problem, but since I updated my project to Angular 8, this is not working.
The replacement of the index.html does not occur.

I never had this problem, but since I updated my project to Angular 8, this is not working anymore.
The replacement of the index.html does not occur.

For later reference, in #14599, @clydin said:

The file replacements option was never intended to replace files outside of the script bundles. That it worked previously was more of a defect itself than anything else.

See this ticket for temporary workarounds (disable es2015 targetting or rename index file after building).

even .ts not worked for me
//angular.json -> projects.myProject.architect.build.configurations.dev :
//also in architect.server section

"dev": {
              "fileReplacements": [
                {
                  "replace": "src/environments/index.ts",
                  "with": "src/environments/dev.ts"
                }
              ]
            }

//src/environments/index.ts
console.log("index")

//src/environments/dev.ts
console.log("dev")

> ng run myProject:build:dev && ng run myProject:server:dev  && npx webpack --config webpack.server.config.js --progress --colors && nodemon dist/server --inspaect --open --watch
import env from "../src/environments/index.ts" //<--- should be replaced with dev.ts
it prints the console.log of index.ts not dev.ts
repo: https://github.com/xxyyzz2050/eldeebCMS

Can confirm this bug with Angular 8.1.3 (build-angular at 0.801.3). I've worked around it for now by having the following build script. I simply copy over the correct index.html at the end:
"build:stg": "ng build --configuration=staging && cp src/index.staging.html dist/index.html",

also failed for me  in version 8.0.1

Failed for me on build but works on serve.
Config is:
            {
              "replace": "src/index.html",
              "with": "src/prod/index.html"
            }

Angular CLI: 8.0.3
Node: 10.16.0
OS: win32 x64
Angular: 8.0.1
... animations, cdk, common, compiler, compiler-cli, core, forms
... language-service, material, platform-browser
... platform-browser-dynamic, router
Package                           Version
@angular-devkit/architect         0.800.3
@angular-devkit/build-angular     0.800.3
@angular-devkit/build-optimizer   0.800.3
@angular-devkit/build-webpack     0.800.3
@angular-devkit/core              8.0.3
@angular-devkit/schematics        8.0.3
@angular/cli                      8.0.3
@angular/flex-layout              8.0.0-beta.26
@ngtools/webpack                  8.0.3
@schematics/angular               8.0.3
@schematics/update                0.800.3
rxjs                              6.5.2
typescript                        3.4.5
webpack                           4.30.0

@vaughnmuirhead Exact same case with me.

Can confirm this bug with Angular 8.1.3 (build-angular at 0.801.3). I've worked around it for now by having the following build script. I simply copy over the correct index.html at the end:
"build:stg": "ng build --configuration=staging && cp src/index.staging.html dist/index.html"

how do the scripts get injected into the  index staging?
i have resorted to this  should work in the azure  pipeline
    "build:en": "copy src\\index.en.html src\\index.html && ng build --configuration=production-en",
    "build:fr": "copy src\\index.fr.html src\\index.html && ng build --configuration=production-fr",

Can confirm it does not work yet in 0.803.2
Trying to reference the index directly (not replacing it) seems to be another solution meanwhile:
https://stackoverflow.com/a/57274333/3719922

This is working in my angular client but not in my angular universal ssr project

"fileReplacements": [                 {                   "replace": "src/environments/environment.ts",                   "with": "src/environments/environment.prod.ts"                 },                 {                   "replace": "src/index.html",                   "with": "src/index.prod.html"                 }               ]
This is still not replacing the index.html in the build.
@angular/cdk 8.2.0
@angular/cli 8.3.4
@angular/core 8.2.6
@angular-devkit/build-angular 0.803.4
Can anyone tell me if I am using the file replacements correctly?

@racerhere as stated above fileReplacements only works with certain filetypes.
The most simple alternative is to use a shell script to replace those file in the build process.


Make bridgeless build with use_frameworks

Summary:
This changes allow React Native to build with Use_Frameworks on iOS.
The major problem we encountered was that objective-c headers should not import C++ files. C++ headers must only be used by .mm files.
Changelog:
[iOS][Fixed] - Make bridgeless works with static frameworks
Differential Revision: D47023664

Closing as abandoned internally


documentation updated to be consistent with last changes

Fixes #1211

Documentation only, no need to wait for the CI.


gfx1100. Why is this a thing?

Problem Description
I find it disappointing that get full backend support for a GPU I paid alot of money for is still, today, lacking the support it needs.  I am building LLVM-17 with clang but I am almost certain that everything will be enabled for my system except the single thing I need to get my gpu to perform as advertized.  Why?  I am not much of a coder and pouring through the config files and hoping I don't change the wrong thing is not support.  I tried returning it because despite my efforts I've never enabled it.  They basically told me tough s***. So now I'm stuck with it so I went ahead and deleted ubuntu (cuz it sucks) and reloaded the OS I use for work.  I've been able through advice and trial and error get it to function almost corect.  I still have issues with vbox and  hashcat won't use it.  Considering the amount of time and the lack of solutions surrounding this elusive gfx11(00) I thinkn AMD and possibly Mesa and  LLVM need to explain this.  Every other version of gfx builds easily an d by default.  Yet gfx11 doesn't, and if you do get it built it sounds like one patch after the next.  Unacceptable.  Full stop.  You'te gonna charge that kind of money, it better work.  And if you're going to support Linux, then support it.  3 distros isn't support, that's just being lazy. If you focused more on compatibility with "debian based" and "arch based" compatibility y you'd probably sell more and have less problems. 9 months I've been waiting for you to figure it out.  Times up, time to put our money to work and fix this.
Operating System
4 distros, 3 versions of one of those
CPU
Intel i9 13th gen
GPU
RX7900XTX
ROCm Version
I don't think so, I use a combo of mesa, vulkan, khronos and python3
ROCm Component
Is there one that isn't full of bugs
Steps to Reproduce
Turning on my computer.
Output of /opt/rocm/bin/rocminfo --support
Like I said.  Ironically my  3 yr old laptop with nvidia and linux work just great.  In fact that old GeForce 16 series is I think faster than my $1100 cooling system, yeah, the fans work fine.

I find it disappointing that get full backend support for a GPU I paid alot of money for is still, today, lacking the support it needs. I am building LLVM-17 with clang but I am almost certain that everything will be enabled for my system except the single thing I need to get my gpu to perform as advertized. Why? I am not much of a coder and pouring through the config files and hoping I don't change the wrong thing is not support. I tried returning it because despite my efforts I've never enabled it. They basically told me tough s***. So now I'm stuck with it so I went ahead and deleted ubuntu (cuz it sucks) and reloaded the OS I use for work. I've been able through advice and trial and error get it to function almost corect. I still have issues with vbox and hashcat won't use it. Considering the amount of time and the lack of solutions surrounding this elusive gfx11(00) I thinkn AMD and possibly Mesa and LLVM need to explain this. Every other version of gfx builds easily an d by default. Yet gfx11 doesn't, and if you do get it built it sounds like one patch after the next. Unacceptable. Full stop. You'te gonna charge that kind of money, it better work. And if you're going to support Linux, then support it. 3 distros isn't support, that's just being lazy. If you focused more on compatibility with "debian based" and "arch based" compatibility y you'd probably sell more and have less problems. 9 months I've been waiting for you to figure it out. Times up, time to put our money to work and fix this.

So you just have to wait, because they have been working hard for the last 6 months. The company's direction on AI has totally changed. Wait for rocm6 they are refactoring all code. They are working very hard since AMD decided to go all in one on AI.

Vbox and hashcat (separately) were working with LLVM 15. They stopped working on the latest version. Try downgrading to the versions at this time of year last year when they were working.

I find it disappointing that get full backend support for a GPU I paid alot of money for is still, today, lacking the support it needs. I am building LLVM-17 with clang but I am almost certain that everything will be enabled for my system except the single thing I need to get my gpu to perform as advertized. Why? I am not much of a coder and pouring through the config files and hoping I don't change the wrong thing is not support. I tried returning it because despite my efforts I've never enabled it. They basically told me tough s***. So now I'm stuck with it so I went ahead and deleted ubuntu (cuz it sucks) and reloaded the OS I use for work. I've been able through advice and trial and error get it to function almost corect. I still have issues with vbox and hashcat won't use it. Considering the amount of time and the lack of solutions surrounding this elusive gfx11(00) I thinkn AMD and possibly Mesa and LLVM need to explain this. Every other version of gfx builds easily an d by default. Yet gfx11 doesn't, and if you do get it built it sounds like one patch after the next. Unacceptable. Full stop. You'te gonna charge that kind of money, it better work. And if you're going to support Linux, then support it. 3 distros isn't support, that's just being lazy. If you focused more on compatibility with "debian based" and "arch based" compatibility y you'd probably sell more and have less problems. 9 months I've been waiting for you to figure it out. Times up, time to put our money to work and fix this.

So you just have to wait, because they have been working hard for the last 6 months. The company's direction on AI has totally changed. Wait for rocm6 they are refactoring all code. They are working very hard since AMD decided to go all in one on AI and support all type of gpus, including risc-v, ffpga and ryzen

Well, I have a (still very good) R5600XT card, mainly for gaming. Also have a (pretty good) work laptop with Ryzen 7 5700u and a pretty decent iGPU.
Very recently got sucked into machine learning stuff. And I'm just... speechless that AMD, in all it's glory, supports 2 (in words: two!) GPUs on Linux?! They really expect anybody to use this... thing?
I love AMD, but I have to say I'm extremely disappointed by their complete lack of support in this. I'm very close to upgrading going the Green team again. Their GPU 'just work' on Linux (and have been for years now!) :hankey:

Wait 6th December

Wait 6th December

I really hope that we - ordinary Radeon endusers - will finally, after all those years, get something more than just a flashy PR presentation...

Ok, so:

No mention of any kind of RDNA compute support
No mention of (and currently no) support of Ryzen AI for Linux
To say this is massively frustrating is just a brutal understatement... :angry:

Expanded Radeon support was mentioned briefly at 26:30.

Wait 6th December

It's December 9th.  Still waiting

Nah, they said it, just have to go to their website from the 2 hour video which doesn't have a section on consumer hardware and check out something like the huggingface link on the ROCm 6 page you'll eventually find.  Not a ROCm 6 download page, mind you, just a page saying that the version number of ROCm will be 6 at some unforseen future date in case you were just born and don't understand how versions work.  Then you get this:
"On the GPU side, AMD and Hugging Face will first collaborate on the enterprise-grade Instinct MI2xx and MI3xx families, then on the customer-grade Radeon Navi3x family. In initial testing, AMD recently reported that the MI250 trains BERT-Large 1.2x faster and GPT2-Large 1.4x faster than its direct competitor."
I'm on Windows and read this as "we're never going to finish MIOpen / ROCm for Windows, we'll put it off for Linux until everybody forgets or the next generation of cards comes out and they don't care, but buy an MI300X ($40,000 per) because we're making some really nice promises about how good that will be at something rather than how good it is at that thing just like RDNA3."
The MI300X has been in the pipeline for 2 years and should have already been working with every major AI model / framework a year ago and entered into the optimizing the heck out of it phase 6 months ago unless no functional parts were created until very recently which is perhaps worse.  The fact that this incredibly recent announcement is saying "will first collaborate" is one of the more disturbing things I've read about a new piece of probably $30-40k hardware in years.  It would be like a car company announcing the day before it went on sale, "we're collaborating with the electrical engineers to make sure the car steers and brakes when you operate the controls."

They obviously aren't going to pay any attention to their consumers.  Going this weekend to get a 4090.

I did so last year since they failed to deliver on two major features (and still haven't delivered).  You won't regret it if you're planning on touching any GPU rendering software or ML related stuff.


Async persistance

Addressing #484 to have an async persistence interface.
Changes

Adds base classes for saving/loading/running persistor in application via async
Adds async initializer/persister to ApplicationBuilder and lets us build async Application via .abuild()
Adds async validation to Application to warn about async hook being ignored in sync runs and raises error if you try to do sync run with an async Application.
Adds create_async_app in parallelism to create application via AsyncApplicationBuilder (in case of legacy use of sync initializer/persister reverts to old sync Application)
TBD: database specific implementations (only AsyncDevNullPersister for testing)

To think about: (1) fire-and-forget or (2) blocking/transactional options --
Async persistors are naturally (1) and sync persistors are (2). If we want to have both options for both cases, we need to:

effectively block the async save to turn (1)->(2) and
create an event loop / another tread and make sync save a coroutine executing there to go from (2) -> (1).
My initial idea was to have that option as an attribute of the persistor class and handle it within PersisterHook/PersisterHookAsync, but still checking if there is a more natural place for this.

How I tested this

Unit and E2E

Notes
This part is very much for discussion if we want to have another AsyncApplicationBuilder or put this into the existing ApplicationBuilder. My arguments for having two:

It is cleaner to separate async and sync (and maybe less error-prone to use the wrong one?).
There are 3 methods we need to define async: with_state_persister, _load_from_persister, and build.
Related to the above, if with_state_persister is async it gets a bit hairy how to do method chaining -- what I did is to overwrite the original method to just store the state_persister (similar to what initialize_from does) and then pushed all the logic into an async helper function __with_async_state_persister that gets awaited in build to manually chain coroutines.
Having another builder class made it clearer also in the Application class to raise an error when run is used instead of arun.

Having said all of that, I also can push those methods into the existing builder with an abuild() to follow the pattern in the Application class where both sync and async are side-by-side.
Checklist

[x] PR has an informative and human-readable title (this will be pulled into the release notes)
[x] Changes are limited to a single goal (no scope creep)
[x] Code passed the pre-commit check & code is left cleaner/nicer than when first encountered.
[x] Any change in functionality is tested
[] New functions are documented (with a description, list of inputs, and expected output)
[ ] Placeholder code is flagged / future TODOs are captured in comments
[ ] Project documentation has been updated if adding/changing functionality.

[!IMPORTANT]
Add asynchronous persistence support with AsyncApplicationBuilder and related async classes and tests.

Behavior:

Introduces AsyncApplicationBuilder to support asynchronous state persistence.
Adds create_async_app in parallelism.py for async application creation.
Implements AsyncDevNullPersister for testing async persistence.

Persistence:

Adds AsyncBaseStateLoader and AsyncBaseStateSaver in persistence.py for async state operations.
Introduces PersisterHookAsync for async lifecycle hooks.

Testing:

Adds tests for async persistence in test_application.py, including test_async_save_and_load_from_persister_end_to_end.

Misc:

Updates imports in __init__.py and application.py to include async classes.

This description was created by  for 9e233c887482dd04ee96662b3cd8bc739f8f1a72. It will automatically update as commits are pushed.

Looks really good -- as noted in a DM I think we have to align on behavior. E.G. when using .build() versus .abuild(), sync versus async persister, amethod versus method -- which configurations break/log warning/work? Cartesian product/table is a clean way to think of it.

This is a current working proposition of which cases should allow what. I think we can replace persister with hooks and put it in the builder docs as well to have clarity on how to use .abuild().

Looks really good -- as noted in a DM I think we have to align on behavior. E.G. when using .build() versus .abuild(), sync versus async persister, amethod versus method -- which configurations break/log warning/work? Cartesian product/table is a clean way to think of it.

This is a current working proposition of which cases should allow what. I think we can replace persister with hooks and put it in the builder docs as well to have clarity on how to use .abuild().

Looks really good -- as noted in a DM I think we have to align on behavior. E.G. when using .build() versus .abuild(), sync versus async persister, amethod versus method -- which configurations break/log warning/work? Cartesian product/table is a clean way to think of it.

This is a current working proposition of which cases should allow what. I think we can replace persister with hooks and put it in the builder docs as well to have clarity on how to use .abuild().

OK, yeah, I think this is the right way to do it! Nice work enumerating it, you have me convinced. The only case (and I'm not sure how this works in the one above) is when you have a synchronous hook in an async .abuild() method. Not a persister (so not fully covered by the above), but I think it should work. That said, we should have appropriate documentation about it.

OK, yeah, I think this is the right way to do it! Nice work enumerating it, you have me convinced. The only case (and I'm not sure how this works in the one above) is when you have a synchronous hook in an async .abuild() method. Not a persister (so not fully covered by the above), but I think it should work. That said, we should have appropriate documentation about it.

This works as before. You run both sync and async hooks --> sync ones are blocking.  But if you use the new .abuild() method, it complains in there about using sync persisters and later when you have the application it complains that you are trying to run it sync.


test inlining enter/exit node

#4288  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @overlookmotel and the rest of your teammates on  Graphite


Add once option to automatically unhook after the first call

Hello @sindresorhus !
I hope this is what you where looking for with #1. I wasn't sure so I set its default behavior as false.
I also replaced new Buffer() with Buffer.from() within the test cases, because that was giving me deprecated errors and breaking the tests. Let me know if that should have been made in a separated PR.
Fixes #1

Looks great. Thank you @ruimarques :)

Cool!


iPhone app crashes when registering ESP-Box (AEGHB-245)

When trying to register the ESP-Box with the iPhone app, the app scans the code, says "connecting" and then just crashes. When I open the app again, the device is not registered. Trying to register again crashes the app.
The iPhone app itself has a link for support, but the website crashes, too:

Error message
PDOException: SQLSTATE[HY000] [2003] Can't connect to MySQL server on '<IP_ADDRESS>' (111) in lock_may_be_available() (line 167 of /var/www/html/includes/lock.inc).

Thanks for your feedback. What's your iPhone model and iOS version?

iPhone SE3, it was the latest version before 16.5.
Over night the iPhone was updated to 16.5 and I now have CircuitPython installed on the ESP32-Box.

We will debug and solve this issue.

We solved and will update the APP, but now is blocked by an issue of Apple Developer, please track the progress through this issue https://github.com/espressif/esp-box/issues/80

The bug has been fixed, please update your ESP-BOX from App Store.

Resolved by https://github.com/espressif/esp-box/issues/72. I will now close this issue. If you have any further questions or concerns, feel free to reopen it.


templates: moved angular templates to static

angular templates, need to be under static so they can be collected and accessed from `/static/templates/invenio_search_ur/{template}.html
resolves inveniosoftware/invenio#4061

Quick question, would a webpack alias work? mainly to avoid having templates spread around.

@topless Quick question, would a webpack alias work? mainly to avoid having templates spread around.

I am not really sure we can do this with aliases in first place, so few of my thoughts

These files take no part in build time of the application, they are static resources and https://flask.palletsprojects.com/en/1.1.x/tutorial/static/ static files for flask is a special folder that gets served
if we put it in assets we have to somehow copy it in the static directory so it can be collected from flask-collect  which runs before webpack actions
webpack aliases are used just to simplify import paths (to my understanding), like shortening long names lets say

I couldn't agree more with you that we scatter template folders around, but they are all different and serve different conventions from invenio
templates/ <- jinja templates
static/templates <- stuff that need be accessed with requests i.e. (my-site.com/static/template.html) (lazily loaded for angular, fetched only when they need to be rendered)
assets/{framework}/templates <- templates that are part of our javascript application etc..


Missing MERGE command

Would you accept a PR to add support for MERGE

@davisford RedisGraph now supports MERGE clauses since:
https://github.com/RedisGraph/RedisGraph/pull/714
You can clone from master branch of use edge docker version