TypeError: can't concat str to bytes

Everytime I try to dump an iOS app, I get several "TypeErrors" relating to "can't concat str to bytes". How do I resolve this issue? An example:
Start the target app com.apple.mobilenotes
Dumping Notities to /tmp
[frida-ios-dump]: Load NotesAnalytics.framework success.
start dump /private/var/containers/Bundle/Application/251DD56D-F079-499C-B66E-0C950D564DEE/MobileNotes.app/MobileNotes
0.00B [00:00, ?B/s]Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/frida/core.py", line 383, in _on_message
callback(message, data)
File "dump.py", line 119, in on_message
scp.get(scp_from, scp_to)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scp.py", line 238, in get
self._recv_all()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scp.py", line 389, in _recv_all
commandcode
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scp.py", line 451, in _recv_file
self._progress(path, size, pos, self.peername)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scp.py", line 116, in
self._progress = lambda *a: progress(*a[:3])
File "dump.py", line 106, in progress
t.update(sent - last_sent[0])
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tqdm/std.py", line 1207, in update
self.refresh(lock_args=self.lock_args)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tqdm/std.py", line 1311, in refresh
self.display()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tqdm/std.py", line 1435, in display
self.sp(self.repr() if msg is None else msg)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tqdm/std.py", line 1065, in repr
return self.format_meter(**self.format_dict)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tqdm/std.py", line 424, in format_meter
l_bar = prefix if bool_prefix_colon_already else prefix + ": "
TypeError: can't concat str to bytes
Exception ignored in: <function tqdm.del at 0x104c5ce60>
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tqdm/std.py", line 1062, in del
self.close()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tqdm/std.py", line 1268, in close
self.display(pos=0)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tqdm/std.py", line 1435, in display
self.sp(self.repr() if msg is None else msg)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tqdm/std.py", line 1065, in repr
return self.format_meter(**self.format_dict)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tqdm/std.py", line 424, in format_meter
l_bar = prefix if bool_prefix_colon_already else prefix + ": "
TypeError: can't concat str to bytes
start dump /private/var/containers/Bundle/Application/251DD56D-F079-499C-B66E-0C950D564DEE/MobileNotes.app/Frameworks/NotesAnalytics.framework/NotesAnalytics
0.00B [00:00, ?B/s]Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/frida/core.py", line 383, in _on_message
callback(message, data)
File "dump.py", line 150, in on_message
t.close()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tqdm/std.py", line 1268, in close
self.display(pos=0)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tqdm/std.py", line 1435, in display
self.sp(self.repr() if msg is None else msg)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tqdm/std.py", line 1065, in repr
return self.format_meter(**self.format_dict)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tqdm/std.py", line 424, in format_meter
l_bar = prefix if bool_prefix_colon_already else prefix + ": "
TypeError: can't concat str to bytes
0.00B [00:00, ?B/s]Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/frida/core.py", line 383, in _on_message
callback(message, data)
File "dump.py", line 137, in on_message
scp.get(scp_from, scp_to, recursive=True)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scp.py", line 238, in get
self._recv_all()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scp.py", line 389, in _recv_all
commandcode
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scp.py", line 451, in _recv_file
self._progress(path, size, pos, self.peername)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scp.py", line 116, in
self._progress = lambda *a: progress(*a[:3])
File "dump.py", line 106, in progress
t.update(sent - last_sent[0])
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tqdm/std.py", line 1207, in update
self.refresh(lock_args=self.lock_args)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tqdm/std.py", line 1311, in refresh
self.display()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tqdm/std.py", line 1435, in display
self.sp(self.repr() if msg is None else msg)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tqdm/std.py", line 1065, in repr
return self.format_meter(**self.format_dict)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tqdm/std.py", line 424, in format_meter
l_bar = prefix if bool_prefix_colon_already else prefix + ": "
TypeError: can't concat str to bytes
Exception ignored in: <function tqdm.del at 0x104c5ce60>
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tqdm/std.py", line 1062, in del
self.close()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tqdm/std.py", line 1268, in close
self.display(pos=0)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tqdm/std.py", line 1435, in display
self.sp(self.repr() if msg is None else msg)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tqdm/std.py", line 1065, in repr
return self.format_meter(**self.format_dict)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tqdm/std.py", line 424, in format_meter
l_bar = prefix if bool_prefix_colon_already else prefix + ": "
TypeError: can't concat str to bytes
0.00B [00:00, ?B/s]Generating "Notities.ipa"
0.00B [00:00, ?B/s]
'app'

diff --git a/dump.py b/dump.py
index fd4ba95..a20c7ce 100755
--- a/dump.py
+++ b/dump.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # -*- coding: utf-8 -*-

 # Author : AloneMonkey
@@ -98,7 +98,7 @@ def on_message(message, data):

     def progress(filename, size, sent):
         baseName = os.path.basename(filename)
-        if IS_PY2:
+        if IS_PY2 or isinstance(baseName, bytes):
             t.desc = baseName.decode("utf-8")
         else:
             t.desc = baseName

This patch #98 works for me

diff --git a/dump.py b/dump.py
index fd4ba95..a20c7ce 100755
--- a/dump.py
+++ b/dump.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # -*- coding: utf-8 -*-

 # Author : AloneMonkey
@@ -98,7 +98,7 @@ def on_message(message, data):

     def progress(filename, size, sent):
         baseName = os.path.basename(filename)
-        if IS_PY2:
+        if IS_PY2 or isinstance(baseName, bytes):
             t.desc = baseName.decode("utf-8")
         else:
             t.desc = baseName

This patch #98 works for me

works great, thx


[backport] Fix no_std MSRV

Backport of #690 to 0.28

Oops, it looks like #985 beat me to the punch ... @tcharding should we close this one or yours? I believe they are identical except for the name of the branch that we're merging into.
Why is yours marked DO NOT MERGE?

Why is yours marked DO NOT MERGE?

I wrote that when the target branch was master before I realised I could target 0.28.x.

Merged, tagged and published.

used in https://github.com/lightningdevkit/rust-lightning/pull/1389


time: Parse does not accept year (YYYY) following an underscore

The time we want to parse is of the format -> HH:MM_YYYYMMDD. Go's time.Parse seems to barf on this format.
time, err := time.Parse("15:04_20060102", "14:38_20150618")

err is not nil:
&time.ParseError{Layout:"15:04_20060102", Value:"14:38_20150618", LayoutElem:"_2", ValueElem:"_20150618", Message:""}

The trouble seems to be here.  The code is assuming that _2 is a date example.  So, the next token it finds is "006".  That's unrecognized by the zero pattern, so the error is thrown.
Not sure how to fix this.  The whole model of specification-by-example seems curious to me.  Suppose I (oddly enough, admittedly) wanted to use the sequence "02" as my separator - in an example it would look like this: Mon02Jan022022006 which is unparsable.

@dtertman sufficiently exotic timestamp formats require custom parsing routines. (Not saying that's the case for the original issue, but it could be.)

@cespare fair enough.  The question then becomes whether _yyyy is "sufficiently exotic".

Proposed fix : Allow \ to indicate that the next character is not part of the example, but text.  Submitter's pattern could then be written as `15:04\_20060102` and mine from above might even be workable as `Mon\0\2Jan\0\22\0\22006`

FYI, I have been bypassing this bug using this specification replacement method:
func bypassTimeParseBug(s string) string {
        // NB: Looks like there is a bug in Golang's time.Parse when handing format strings
        // with _2 in the format string. Here is a snippet that exhibits this issue:
        //     t, e := time.Parse("15:04_20060102", "14:38_20150618")
        // We replace underscores with space to bypass this bug.
        return strings.Replace(s, "_", " ", -1)
}

I would posit that the format I specified is not exotic at all. At least, I didn't invent it. It's the format used by Graphite to specify absolute timestamps.

Pretty much any change made is likely to break someone's time format string. But time format strings are probably almost always in ASCII. So would a unicode character that's unlikely to be part of a format string be considered as an escape character?  Or _ could be special cased so that ‗ (U+2017; double low line) means a literal _.

I think _2006 should be parsed as _ followed by 2006, not as _2 followed by 006.
That is, there's no need for new API or notation here; just fix the current notation.
/cc @robpike

@rsc I was over thinking the issue, my bad. Please see my CL.


PARSE_ERROR

Preflight Checklist

[X] I have installed the latest version of Storage Explorer.
[X] I have checked existing resources, including the troubleshooting guide and the release notes.
[X] I have searched for similar issues.

Storage Explorer Version
1.28.1
Regression From
No response
Architecture
x64
Storage Explorer Build Number
20230309.1
Platform
Windows
OS Version
Windows 11
Bug Description
{
"name": "RestError",
"message": "Error reading response as text: aborted",
"stack": "RestError: Error reading response as text: aborted\n    at IncomingMessage. (C:\Program Files (x86)\Microsoft Azure Storage Explorer\resources\app\node_modules\se-table-extension\dist\src\Jobs\TableJobsHandler.js:2:152007)\n    at IncomingMessage.emit (node:events:513:28)\n    at emitErrorNT (node:internal/streams/destroy:157:8)\n    at emitErrorCloseNT (node:internal/streams/destroy:122:3)\n    at process.processTicksAndRejections (node:internal/process/task_queues:83:21)",
"code": "PARSE_ERROR"
}
Steps to Reproduce

Launch Azure storage
Connect to your storage
choose tables
open a table
click "Export" -> Export all
the scanning process stops after a while with this error

Actual Experience
We are trying to export a table from the Azur Storage Explorer. But we face an error after a while in the middle of the scanning process before the download starts.
Expected Experience
An exported file with the data from the table
Additional Context
No response

Closing due to lack of response. If you still have this problem, please gather the additional information and open a new issue.


chore: upgrade charts

updated chart jxgh/jxboot-helmfile-resources from 1.1.111 to 1.1.112

/close


Block-DX and XLite terminated with return code 127

I setup a new Xubuntu 24 VM in VirtualBox and installed the latest AIO. It downloaded everything OK, downloaded the bootstrap, started and synced Blocknet OK. When I tried to start Block-DX and XLite, both failed with the following messages in the console....
INFO:root:xbridgeconfpath: /home/mark/.blocknet/xbridge.conf
WARNING:root:/home/mark/.config/BLOCK-DX/app-meta.json doesn't exist
DEBUG:root:Updated 'user' in meta_data
DEBUG:root:Updated 'password' in meta_data
DEBUG:root:Updated 'xbridgeConfPath' in meta_data
DEBUG:root:Initialized 'selectedWallets' with 'blocknet--v4.2.0' in meta_data
INFO:root:Updated Blockdx config with new data.
INFO:root:Started Blockdx process with PID 362125: /home/mark/.AIO_Blocknet/BLOCK-DX-1.9.5-linux-x64/block-dx
INFO:root:Block-DX process has terminated with return code 127
INFO:root:Started Xlite process with PID 363217: /home/mark/.AIO_Blocknet/XLite-1.0.7-linux/xlite
INFO:root:XLite process has terminated with return code 127

Problem seems to be related to ubuntu 24.X introducing new security feature, this break most Electron apps:
https://github.com/electron/electron/issues/41066

The temporary solution is disabling the kernel.apparmor_restrict_unprivileged_userns:
sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0

until we get something better, that will fix the problem.


Url to uri

Changed all occurrence of URL to URI

This change is

Coverage decreased (-0.02%) to 54.561% when pulling 33fa785f6406debe0442d584cb6072e1d8ba0b00 on URLToURI into 89b95ec56ddecad920b3c2dd1611e648b030c799 on master.

@kunsingh You need to merge the master branch in this branch.

Coverage decreased (-0.02%) to 54.584% when pulling c579cccc1c91c1a47390aabc2fc17e2d257ae149 on URLToURI into f179d36fbd20fc527c120e73ef8c251681d4e14c on master.

fix for https://github.com/canoo/dolphin-platform/issues/714


Still active?

Is this repo still active? Looks like it's using old versions of k8s, etcd, docker, etc., and hasn't changed since February. If this is still active, we've been doing updates at https://github.com/18f/kubernetes-release that we can break up into PRs. If not, I'm wondering if this could be transferred to https://github.com/cloudfoundry-community so that it can be maintained.

Ping @acrmp @cghsystems: any interest in reviewing PRs and/or transferring to cloudfoundry-community? Thanks!

Hi Joshua, sorry for the delay in getting back to you.
Chris may know better than I do, but I'm not aware of plans to maintain this BOSH release going forwards.
However if you're finding it useful (and making what seem like fairly substantial improvements) it would be great to see it actively maintained elsewhere.
Thanks,
Andrew.

Hi Joshua
Sadly we have no plans to maintain this at the minute. I would love to maintain it in my own time but sadly I do not have time.
I would love to transfer it to the community. Let me look into this and see if we can make it happen.
Thanks
Chris

@cghsystems: did you have any luck on transferring the repo? Thanks!


Fix codeAsInSource

Let's see if this works better now...
It took a day but I think I've found a somewhat non-chaotic implementation that seems to work. I added a few tests but I will add a few more and manually go through the docs to see if it looks OK.
fixes #97
TODO

[ ] Handle comments at end of code blocks
[ ] Go through the docs and check for errors

Ok, that seems to have fixed it but why did it work locally without any errors... 🤷

Nice! Can you try also to remove the when defined(nimibCodeFromAst) in t nimib.nim?

Something smells weird here, when the defined(nimibCodeFromAst) was included no errors. But when it was removed it gives an error at the very end (after the last test has been run with OK). I read somewhere that a plain Error: execution of an external program failed: means that the program returned a non-zero error code but I have idea how that could be dependent on this when-statement :/

To make it even stranger, running it with nim r gives this error but doing ``nim c` and then running the executable doesn't give any errors

Oh wait I'm just blind, there was a failed test 🤦

ah, right, it is to be expected! In nim multiline string remove the first newline so CodeFromAst does not see it, while CodeAsInSource sees it as it is written. They are both right! Probably the easieast way to fix it would be a short template/proc used only in tests and added to the multiline strings involved here:
template normalizeMultilineStringLiteral(s: string) =
  when not defined(CodeFromAst):
    if s[0] == "\n":
      return s[1 ..< s.len]

didn't see you already had the fix. fine as it is for me!

ah, right, it is to be expected! In nim multiline string literal remove the first newline so CodeFromAst does not see it, while CodeAsInSource sees it as it is written. They are both right! Probably the easieast way to fix it would be a short template/proc used only in tests and added to the multiline strings involved here:

TIL :D

didn't see you already had the fix. fine as it is for me!

It is the only case where it matters thus far so making a template only feels warranted when we have a few test that needs it 👍

I've gone over the docs now and to my eyes, it looks like it's working but I would appreciate a second opinion on that 😅


-slim suffix removed when updating Dockerfile base image from python:3.10.5 to python:3.10.6

In a multistage build with two python base images:
python:3.10.5 as base
python:3.10.5-slim as production

the -slim suffix wasn't preserved in an update from 3.10.5 to 3.10.6

Package ecosystem
docker
Manifest location and content before the Dependabot update
https://github.com/mozilla/jira-bugzilla-integration/blob/main/Dockerfile
dependabot.yml content
https://github.com/mozilla/jira-bugzilla-integration/blob/main/.github/dependabot.yml
Updated dependency
python:3.10.5-slim
What you expected to see, versus what you actually saw
What I expected to see:
python:3.10.5-slim as production -> python:3.10.6-slim as production
What I saw:
python:3.10.5-slim as production -> python:3.10.6 as production
Images of the diff or a link to the PR, issue, or logs
diff --git a/Dockerfile b/Dockerfile
index 4324ead..d64ace4 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -1,5 +1,5 @@
 # Creating a python base with shared environment variables
-FROM python:3.10.5 as base
+FROM python:3.10.6 as base
 ENV PIP_NO_CACHE_DIR=off \
     PIP_DEFAULT_TIMEOUT=100 \
     PIP_DISABLE_PIP_VERSION_CHECK=on \
@@ -21,7 +21,7 @@ COPY ./poetry.lock ./pyproject.toml ./
 RUN poetry install --no-dev --no-root

 # `production` stage uses the dependencies downloaded in the `base` stage
-FROM python:3.10.5-slim as production
+FROM python:3.10.6 as production
 ENV PORT=8000 \
     PYTHONUNBUFFERED=1 \
     PYTHONDONTWRITEBYTECODE=1 \

There's a chance we may have to revert the fix for this given the discussion in https://github.com/dependabot/dependabot-core/issues/3261 and the fact that it included similar PR which then got reverted... a few more details.
But even if #5560 gets reverted, this should still stay closed as a duplicate of

https://github.com/dependabot/dependabot-core/issues/3261


get_parameters_dict() method possibility

I was wondering, can we have an option to draw a param dict from the cosmology object?
cosmology = ccl.Cosmology(Omega_m=0.3, h=0.67...)
params = cosmology.get_parameters_dict()
params = {
'Omega_c': cosmology['Omega_c'],
'Omega_b': cosmology['Omega_b'],
'h': cosmology['h'],
'sigma8': cosmology['sigma8'],
'n_s': cosmology['n_s'],
# Add any other relevant parameters here
}
I would find this beyond useful as I work with dicts a lot

@hsinfan1996

For now you can retrieve the two private attributes cosmo._params_init_kwargs and cosmo._config_init_kwargs.

This kind of exists but it's ocult:
If cosmo is a ccl.Cosmology object you can do:
cosmo.__dict__['_params_init_kwargs']

Oh, @hsinfan1996 beat me to it :)

@hsinfan1996  @fjaviersanchez brill! much obliged!!

Cosmology.write_yaml does that internally. We should probably factor that out into a to_dict method or similar.

@tilmantroester ah I see you also prefer to have a method rather then calling cosmology._params_init_kwargs
yeah actually makes sense (going back to my initial request)

You shouldn't use private APIs (eg anything with a leading underscore) outside of CCL, since they can change at any point. Better to add a proper API than hack something together.

You shouldn't use private APIs (eg anything with a leading underscore) outside of CCL, since they can change at any point. Better to add a proper API than hack something together.

Yeah totally makes sense, you are right. And this is not some elaborate code that requires 3 days (it's a few min in Python at least) and it is beyond useful to have. I have my own version of it and use it constantly.

We should look into how to make to_dict and from_dict methods that allow round trip generation of Cosmology objects. That's useful for serialisation (#1159), as well as for the use case where we want the same cosmology setup but with a few parameters changed, e.g. for blinding (@arthurmloureiro).
We might be able to factor this functionality out of the __repr__ machinery in https://github.com/LSSTDESC/CCL/blob/master/pyccl/core/repr.py#L103.

@tilmantroester @nikosarcevic
I am having precisely that difficulty with the blinding library. I found a very dirty temporary solution that is similar to what has been suggested in this issue:
https://github.com/LSSTDESC/Blinding/blob/1920d195ca182f3625d5bab0018dea0db925bffc/src/blinding/smokescreen.py#L156
    def _create_blinded_cosmo(self):
        """
        Creates a blinded cosmology object with the shifts applied.

        FIXME: Unsure this is the best way of doing this but it is similar to what is done in Augur.
        """
        blinded_cosmo_dict = deepcopy(self.cosmo._params_init_kwargs)
        # sometimes we have this extra paramters that can cause problems:
        try:
            del blinded_cosmo_dict['extra_parameters']
        except KeyError:
            pass
        for k in self.__shifts.keys():
            blinded_cosmo_dict[k] = self.__shifts[k]
        blinded_cosmo = ccl.Cosmology(**blinded_cosmo_dict)
        return blinded_cosmo

A safer, more "off the shelf" solution would be great. I am not super happy with the solution I found.
(Note that HORRIBLE, inexcusable try/except which is begging to cause trouble... I couldn't find a better way of doing this...)


Prohibit mirroring to internal repositories

Motivation:
We should prohibit mirroring to internal repositories which can cause a security incident.
Modification:

Raise an exception if the localRepo of mirroring setting is one of meta and dogma which are internal repositories.

Result:

You cannot set up mirroring to internal repositories anymore.

Thanks for reviewing. 😉


Not able to quickstart datahub via docker

I am trying to run datahub in my Vm via docker, it was running fine few weeks ago but now when I try to start it again it gives me an error for the datahub-gms container.
Screenshots

Error Log file
datahub-gms.txt

Same


HTM doesn't work

I've created example project with deno run -A -r https://fresh.deno.dev my-project and tried to replace JSX with HTM. It renders fine but there is no interactivity. As I can see no JS related to Counter is sending to browser (e.g. no related <source> tag). I suppose it's a wrong behaviour.
Here are my changes:
// islands/Counter.ts (not tsx!)

import { html } from "htm/preact";
import { useState } from "preact/hooks";
import { IS_BROWSER } from "$fresh/runtime.ts";
import { tw } from "@twind";

interface CounterProps {
  start: number;
}

export default function Counter(props: CounterProps) {
  const [count, setCount] = useState(props.start);
  const btn = tw`px-2 py-1 border(gray-100 1) hover:bg-gray-200`;
  return html`
    <div class=${tw`flex gap-2 w-full`}>
      <p class=${tw`flex-grow-1 font-bold text-xl`}>${count}

      <button
        class=${btn}
        onClick=${() => setCount(count - 1)}
        disabled=${!IS_BROWSER}
      >
        -1
      </button>
      <button
        class=${btn}
        onClick=${() => setCount(count + 1)}
        disabled=${!IS_BROWSER}
      >
        +1
      </button>

  `;
}

// routes/index.ts (not tsx!)

import { html } from "htm/preact";
import { tw } from "@twind";
import Counter from "../islands/Counter.ts";

export default function Home() {
  return html`
    <div class=${tw`p-4 mx-auto max-w-screen-md`}>

      <p class=${tw`my-6`}>
        Welcome to \`fresh\`. Try update this message in the ./routes/index.tsx
        file, and refresh.

      <${Counter} start=${3} />

  `;
}

Thanks for the clue! Looks like HTM really has problems with hooks.
So, if get it right, this issue can be closed - Fresh doesn't relate to this.

Actually, I think that issue gave me an idea of something you can try. Can you try import htm like this: https://esm.sh/htm<EMAIL_ADDRESS>

Yes, it's working now!
// import_map.json
...
    "htm"<EMAIL_ADDRESS>    "htm/preact": "https://esm.sh/htm<EMAIL_ADDRESS>...

Then I'll close issue. Solution is using same version of preact.


Cutoff/spacing issues w/ ether send

text box is too close to button

text is cutoff & button again

text is cutoff & button again

Widened the send modals in 7b9efaba6c17377f4cd080346aef9de9d12218b1.


useSignalEffect not activated, effect triggers fine

[x] Check if updating to the latest Preact version resolves the issue

This is probably some issue on my side, but I would be very glad if anyone could help me fixing it.
To Reproduce
I have a global state
const state = {
  data: signal([0])
}

export default state;

When I reassign the data state.data.value = [0, 1, 2] the event is propagated only partially. If I have a functional component,  effect works.
export default function SceneTable() {
    let l = 0;
    effect(() => console.log(appstate.data.value.length));
    return <p>{appstate.data.value.length}+</p>;
}

But when I use a class Component instead, then useSignalEffect never triggers. Why so? Did I forget about something?
//a props interface IProps

export default class SceneTable extends Component<IProps> {
  useSignalEffect(){
    console.log(appstate.data.value.length);
    //never hits here
  }

  //some other methods
}

Maybe an important info: the class component is created as tableRef = createRef<SceneTable>() in a parent class component, while the my test with the functional component was stand-alone.

Are you using React or Preact?
You should be using the hook versions (useX) in function components, and the non-hooks in class components. React limits hooks to function components, you cannot use them in class components. Preact does allow you to use hooks in class components, but it's quite unidiomatic and I'm not sure if support is guaranteed long-term.

Ah that was not clear to me. I thought it is a lifecycle method! I wrongly interpreted the following docs line

When responding to signal changes within a component, use the hook variant: useSignalEffect(fn).

as being a lifecycle method in class components.
So just out of curiosity - is there any concept that I could use within a class component to handle updates in signal values?

I rather thought of something like:
const data = signal([1,2,3]);

class CheckBox extends Component {
    process = effect(() => {
      //some heavy processing specific for this component
      this,state.data = this.process(data.value);
    });
}

I need to process data before rendering them and a computed signal makes not much sense as

the data view is specific for the component instance
it does not explicitly appear in the render() function but instead it is fed as scene data to a THREE.js scene

but that is my specific use case. The above concept with process = effect(() => { /*...-* }) seems to work in class components and I anyway switched to functional in the meanwhile :)


Operator: Support JSON-formatted Vault policies

Q
A

Bug fix?
no

New feature?
yes

API breaks?
no

Deprecations?
no

Related tickets
fixes #979

License
Apache 2.0

What's in this PR?
This adds support for Vault Policies to be provided as JSON strings, in addition to HCL-formatted strings. Current behaviour tries to use an HCL formatter to make it pretty, which won't work for JSON!
This change makes the operator attempt to parse the policy (with a Vault function which parses both JSON and HCL) and if it succeeds, applies it.
Why?
Vault supports policies in both HCL and JSON format so the operator should too!
Additional context
Our policies were becoming boilerplate-y and less maintainable so we decided to generate them via Jsonnet, where it's significantly easier to template JSON than HCL.
This has been tested with valid JSON and HCL policies and both are created successfully.
Checklist

[x] Error handling code meets the guideline

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Good Job @chrlwrd , i won't approve since i work with him but i am glad to bring him on board on this project!

Ah geat, nice job! Welcome!


Storage: Fixes LVM VM snapshot list

This bug was introduced in d782daf202cda0de0a88886e21aa056775024139 due to the way that VM snapshot LVs are named differently than container snapshot LVs.
Signed-off-by: Thomas Parrott<EMAIL_ADDRESS>

@stgraber I found this whilst doing the backup feature for VMs.


Undefined behavior (null pointer passed as argument 1, which is declared to never be null) in pkcs12kdf_derive

Hello.
I am using latest OpenSSL on ubuntu 20.04. I compiled with clang++ and UBSan.
It seems that funtion pkcs12kdf_derive in providers/implementations/kdfs/pkcs12kdf.c doesn't do validation check on param *out. When *out is nullptr, UBSan detects a runtime error:

Do you have a concrete code example which invokes this code with NULL *out?

Try this:
void test(){
    int keySize = 0;
    EVP_KDF_CTX* kctx = nullptr;
    const EVP_MD* md = nullptr;
    OSSL_PARAM params[7], *p = params;
    uint8_t* out = nullptr;

    std::vector<uint8_t> password;
    password.push_back(0xcc);
    password.push_back(0xd2);
    *p++ = OSSL_PARAM_construct_octet_string(OSSL_KDF_PARAM_PASSWORD, password.data(), password.size());

    std::vector<uint8_t> salt;
    *p++ = OSSL_PARAM_construct_octet_string(OSSL_KDF_PARAM_SALT, salt.data(), salt.size());

    unsigned int iterations = 2;
    *p++ = OSSL_PARAM_construct_uint(OSSL_KDF_PARAM_ITER, &iterations);

    unsigned int id = 1;
    *p++ = OSSL_PARAM_construct_uint(OSSL_KDF_PARAM_PKCS12_ID, &id);

    md = EVP_blake2b512();
    std::string name(EVP_MD_name(md));
    *p++ = OSSL_PARAM_construct_utf8_string(OSSL_KDF_PARAM_DIGEST, name.data(), name.size() + 1);

    int pkcs5 = 0;
    *p++ = OSSL_PARAM_construct_int(OSSL_KDF_PARAM_PKCS5, &pkcs5);

    *p = OSSL_PARAM_construct_end();

    EVP_KDF* kdf = EVP_KDF_fetch(nullptr, "PKCS12KDF", nullptr);
    kctx = EVP_KDF_CTX_new(kdf);
    EVP_KDF_free(kdf);
    EVP_KDF_derive(kctx, out, keySize, params);    //trigger bug

    EVP_KDF_CTX_free(kctx);
    free(out);
}

This is not a bug. Using NULL pointer for out parameter of EVP_KDF_derive() is invalid use of the function. We do not add NULL pointer checks to such function calls. It would be a bug only if you could reproduce the crash with valid use of the public API calls.

Okay. Thanks a lot.


Inquiry Regarding Missing "layout.mat" File in the Codebase

Hello, I've been encountering an issue while running the code, specifically a missing "layout.mat" file.  Despite searching through the file system, I couldn't locate it. In the code, it is referenced as an input, and its  location is within the "outputs" folder. Does this mean that this is the output of a submodule?
I observed that during the project clone, I faced network issues, and as a result, the "third_party" submodule appears to be empty.    Could this be the reason behind the submodule-related problems and the absence of the "layout.   mat" file as its output?
I appreciate any insights or guidance you can provide to help resolve this issue.

Hi @yhhxt ,
Thanks for your question. You need to also clone the submodules as well. In this error, it seems that third_party/Total3D is not cloned successfully so my script cannot generate the files under data/total3d/outputs. Please make sure all the submodules are cloned completely. Thank you.

Thank you very much for your response. Yesterday, I attempted to clone all the submodules again, and now it can produce partial results. However, some outputs are still missing. I noticed that the three folders (uvCoord, normal, partid)  located under preprocess/renderIni are all empty. In the code, these three components are implemented by OptiXRender, and their output results are empty. I'm unsure whether this is a compilation issue with OptiXRender or an installation problem with OptiX. The error message is as shown in the attached image.

Yes, the graphics card I'm currently using is the RTX 3090.   This might be one of the reasons.   Thank you very much for your response.


Twobody states

This issue is related to the ModifiedEquinoctialState class in twobody/states.py and its computation in core/elements.py.

I tried to take a look at the reference of the method coe2mee i.e. Walker 1985, which tells that there is a singularity if the inclination of the orbit is 180 degrees, although it removes any singularity problems with equatorial and/or circular orbits.
I think, due to the way np.tan works, it doesn't return any "singularity!" return value if the angle is 90 degrees, for example, but
returns a large value (of the order, say, 1e16).
So the question is, would it be a good choice the raise a warning or error if the inclination is 180 degrees instead of returning
unnecessary large values for some equinoctial elements? It also seems GMAT doesn't support modified equinoctial elements
for true retrograde equatorial orbits. But maybe such orbits could also be handled by having a retrograde factor?

This line also has a "TODO" to check for the polar case.

TODO: Check polar case (see [Walker, 1985])

Did it mean to add a test that checks for polar orbits (inclination = 90 degrees) or something else?

It seems, currently there are methods to convert from any two of the state pairs {classical, position-velocity, equinoctial}, except to convert from equinoctial to position-velocity. Would it be a good choice to add a method that does this? It looks like the description of this is given in this paper: https://doi.org/10.1007/BF01228432 under the section "5. Conversion from Equinoctial Elements to Position-Velocity ".

In this line, p is passed as a parameter but is returned unchanged. Would there be a choice to instead pass a and then calculate p = a * (1 - ecc ** 2) inside the method? Maybe p was passed intentionally for consistency reasons?

Thanks!

See also #282, maybe we should continue the conversation there.


Request捕获异常的顺序错误

https://github.com/denisneuf/python-amazon-ad-api/blob/f0cf04163551494c3637c86a4531d52168593d80/ad_api/base/client.py#L68
https://github.com/denisneuf/python-amazon-ad-api/blob/f0cf04163551494c3637c86a4531d52168593d80/ad_api/base/client.py#L76
RequestException是ConnectionError的基类，我认为ConnectionError应该在RequestException的前面被捕获。

I have altered the order as it makes sense. It will appear in the next release as I have to add some other functions. Thank you.


Create a config fileIt should include a title for the application (to go in h1) and a message to display when there are are no storage options available (e.g. contact so and so, whatever). The message should be able to be HTML from a string

It should include a title for the application (to go in h1) and a message to display when there are are no storage options available (e.g. contact so and so, whatever). The message should be able to be HTML from a string


Chain-independent script to generate setImplementation operations

Description
Adds a template to the fp-recovery directory with tools to create operations that set the implementation for fault dispute games.  Multiple implementations can be set in a single batch (e.g. setting CANNON and PERMISSIONED game type as part of a hard fork upgrade.
This would replace #354 - it generates essentially the same task but in a much more automated way and with the ability to make multiple changes in a single operation.

Tested this on mainnet and it works with a simulation that looks right now that the livenessGuard is automatically determined.

@mdehoog Could you help me with updating this one to handle the changes in https://github.com/ethereum-optimism/superchain-ops/pull/356 please? Getting the code to compile is easy, but now that it's rebased when I call getOwners on 0x1Eb2fFc903729a0F03966B917003800b145F56E2 (on sepolia L1) which should be the 2-of-2, it only gives me 1 address.  The actual contract on L1 still has two owners but it seems the simulation is now removing the council safe as an owner.
It seems when SIMULATE_WITHOUT_LEDGER=true is set, the owners of the 2-of-2 is set to the first owner account of the council safe.  But the council safe is still accessed somehow as part of the execution so it fails with "Unallowed Storage access". Was that deliberate? It seems like a weird configuration to be trying to simulate with.

To reproduce the problem I'm hitting run:
just clean prep sepolia op set-implementation 0 0xaA511096675013995A2D58275af6a8880AC39F44
cd out
SIMULATE_WITHOUT_LEDGER=true just --dotenv-path $(pwd)/.env --justfile ../../../../../nested.just simulate council

I get the same problem without SIMULATE_WITHOUT_LEDGER=true as well and when using the foundation safe.

Worked around this by going back to using the hard coded council and foundation safe addresses (https://github.com/ethereum-optimism/superchain-ops/pull/363/commits/aa9543e9f7e4d81141f7b2c44b8beb18c01a6a3b) but it's a shame to have to embed that into the script as it means the script itself will have to be adjusted to deal with 3-of-3 safes and other variants.
Overriding the safe owners still seems kind of suspicious though, especially since it doesn't seem to completely eliminate the original owner safes.

@ajsutton bug fix is here: base-org/contracts#105, apologies for the regression.

Oh nice. Thanks.


Implement simple test via GitAction / GitRunner

Implement simple test via the gitAction/Runner to ensure the implementation is working correctly. Task 6 from https://github.com/opendatahub-io/model-registry/issues/187
Definition of Done
When the simple test runs and passes successfully.

Done


Allow to install version 1 of jms/serializer

Hi @norkunas thanks for your contribution, and sorry for my late response!
I've been backtracking on the other PRs, one of which already solved this issue (requiring ~1.0 from now on) so I will close this now.

@cleentfaar thanks for solving this at last ;)


WIP: Are the derive(Accounts) to_account_infos and to_account_metas implementations dead code?

After reading through the #2094 PR it seems like the derive(Accounts) to_account_infos and to_account_metas implementations might be dead code, since __client_accounts_mod and __cpi_client_accounts_mod seem to do everything that's needed? Just going to run the tests and see what breaks once they're removed from the macro generation.

the derive traits for to_account_infos/metas are used but it seems like they're mainly used by

CPI accounts in the SPL CPI library
nested / composite accounts, to return a list of their account infos


Fixes 2717: add config to turn off notifs

Summary
so we can disable it in prod
Testing steps
in one terminal:
make kafka-topic-consume KAFKA_TOPIC=platform.notifications.ingress

in another:
go run cmd/content-sources/main.go api consumer
Then create a repo for introspection:
curl -X POST --location "http://localhost:8000/api/content-sources/v1.0/repositories/" \
    -H "Content-Type: application/json" \
    -H "x-rh-identity: eyJpZGVudGl0eSI6eyJ0eXBlIjoiVXNlciIsInVzZXIiOnsidXNlcm5hbWUiOiIxIn0sImFjY291bnRfbnVtYmVyIjoiMSIsImludGVybmFsIjp7Im9yZ19pZCI6IjEifX19Cg==" \
    -H "x-Rh-Insights-Request-Id: 9876" \
    -d "{
          \"name\": \"pulp3.16-8\",
          \"url\": \"http://yum.theforeman.org/pulpcore/3.16/el8/x86_64/\",
          \"snapshot\": false
        }"

You should see no messages comes through.   Finally, turn it on:
OPTIONS_ENABLE_NOTIFICATIONS=true  go run cmd/content-sources/main.go api consumer
create and introspect a new repo, now see messages

we want to merge this https://gitlab.cee.redhat.com/service/app-interface/-/merge_requests/83144 before this PR

https://issues.redhat.com/browse/HMS-2717

/retest

MR to app-interface is merged, merging this

(as no qe is needed as per discussion)


Simplify API: Cleans-up some API details around Model and Module

Changes

elegy.States now implements Mapping and __getattr__  so you can add arbitrary keys but still use states.{field} to access values. By not enforcing any particular set of fields States can be more flexibly used with the low-level API, however, the user must know what the are the standard names (rng, net_params, net_states, metrics_states, etc) and their behavior, this will be properly documented.
Removed all names previously in States (rng, net_params, net_states, ...) from the Dependency Injection lists, now the users must request states and access the field of interest (e.g. states.rng), this is both simpler and more explicit.
elegy.Module.init and elegy.Module.apply now return (y_pred, parameters, collections) where parameters = collections.pop("parameters", None). apply now expects parameters and collections as arguments.
Added base_step to the low-level API, defines the initial set of elegy.States.
Simplified the definition of PredStep, it is now only (y_pred, states).
Removed the Uninitialized type.
Introduces summary_step and moves summary to ModelCore.

Codecov Report

Merging #158 (a828d02) into master (cb70cf5) will decrease coverage by 0.16%.
The diff coverage is 97.41%.

@@            Coverage Diff             @@
##           master     #158      +/-   ##
==========================================
- Coverage   84.90%   84.73%   -0.17%
==========================================
  Files         131      131
  Lines        6815     6785      -30
==========================================
- Hits         5786     5749      -37
- Misses       1029     1036       +7

Impacted Files
Coverage Δ

elegy/generalized_module/generalized_module.py
66.15% <0.00%> (ø)

elegy/model/model_base.py
83.47% <ø> (ø)

elegy/nn/moving_averages.py
85.71% <ø> (ø)

elegy/nets/resnet.py
93.39% <87.50%> (-0.67%)
:arrow_down:

elegy/types.py
94.73% <94.23%> (-1.90%)
:arrow_down:

elegy/model/model_core.py
87.62% <95.78%> (+2.71%)
:arrow_up:

elegy/module.py
95.18% <97.61%> (-0.14%)
:arrow_down:

elegy/generalized_module/elegy_module.py
91.17% <100.00%> (-4.75%)
:arrow_down:

elegy/generalized_module/haiku_module.py
90.62% <100.00%> (-0.56%)
:arrow_down:

elegy/generalized_module/linen_module.py
90.62% <100.00%> (-0.15%)
:arrow_down:

... and 23 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cb70cf5...a828d02. Read the comment docs.


FireFox on OS X pixelization

This is not quite an issue, just a note that various browsers display different results with the proposed technique.
FireFox 43 under OS X displays significant pixelization for scaled up images. Here's how the front page looks in FF / Mac OS X:

A little further research shown differences of interpolation algorithms utilized by various browsers. Sample 9x5px image (white bg, 9 black pixels)

was scaled up to 900x500 in several browsers / operating systems.
Chrome 47, Mac OS X  (best-looking one)

Firefox 43.0.4, Mac OS X

Firefox 43.0.4, Windows 8

Should it be reported to FF as a bug/feature request? At least it should respect the image-rendering CSS property, as it's a standard.
https://drafts.csswg.org/css-images-3/#the-image-rendering

I don't see a connection to image-rendering. The property only suggests three values: auto | crisp-edges | pixelated, none of which specifies which interpolation algorithm specifically to use for scaled-up image blur. They only mention "bilinear interpolation" (FF/Win above implements it if I'm not wrong) as an example.

It does specify algos: in my understanding only pixelated options should be pixeled :) Auto should use bilinear which is quite specific.
Anyways, FF now handles this thing inconsistently across platforms, I'll file a bug report to them.


Project Task_1

Log Message using Arguments and Asset creation.

{
"NoFile":  [
{
"Your organization requires your project to have an Automation Hub URL defined.":  "Use "Project Settings" to add the missing information. "
},
{
"Dependency package UiPath.Excel.Activities is not used.":  "Remove unused packages in order to improve process execution time. "
},
{
"Dependency package UiPath.Mail.Activities is not used.":  "Remove unused packages in order to improve process execution time. "
}
],
"Main.xaml":  [
{
"WriteLine activity usage at Write Line.":  "Write Line activity can be replaced with tracepoints during debug. They should be removed before publishing. Consider also changing WriteLine to Log Message if it is really needed for debugging purposes. "
},
{
"Argument str_Name does not respect the set pattern ^in_(dt_)?([A-Z]|[a-z])+([0-9])*$":  "Make sure all the arguments follow the naming convention. "
},
{
"Activity Write Line has a default name.":  "Check if it could be easier to understand what the activity does by adding a more descriptive title. "
}
]
}


Fix UART RX character dropping

This is a continuation of #50, but also see pull request #97. That was a temporary improvement to reduce the frequency at which received characters get dropped, but this problem needs to be solved for good.
Refer to the 32M1 datasheet, Section 21 on UART: https://ww1.microchip.com/downloads/en/DeviceDoc/Atmel-8209-8-bit AVR ATmega16M1-32M1-64M1_Datasheet.pdf

Results from transceiver testing on 2018-12-27, note character dropping in most responses after the first couple of commands. On the left is what OBC sends and receives; on the right is the transceiver's actual responses as read by the laptop.

Not sure what changed but this is working properly now.


No ability to message yourself :(

All devices have the ability to SHARE to other platforms.  Often its necessary when not on your main device to SEND (share) yourself a message or image.  Riot does not allow you to send (share) a message to YOURSELF.
The opportunity to write yourself is also VERY handy to take notes so that you don't need to LEAVE the riot platform to take notes.

You could create a room with just yourself in it, that should accomplish what you're asking for :)

Perfect!  Thanks.


Added several logistics pages

We need these changes pushed to production so Tim is going to resubmit this to the master branch


Avaya dmcc with asp.net core 2.1

This is not a bug report or any feature request.
My issue is that I have been asked to enable cti Intigration with avaya one x softphone. I have found an application (.net framework 2.0) that enables(provides) all the telephony features. But my concern is that, I have to communicate to the avaya one x using my web portal which is created using angular 5 and asp.net core2.1. So how should I go about this .. I have red about web sockets, WCF etc..
My situation is, I have a web deployed on a server. The dmcc application will be running on all client pc. And whenever a client opens the web app and login, he/she must be able to contrll the avaya softphone on that PC using that web portal.
Any help would be highly appreciated..I only need the structural view of this solution..

@AhmadWaheedQadri

One-X softphone does have a REST API that you can talk to over HTTP. So the softphone runs a local HTTP server that you can in theory talk to right from your client code: https://www.devconnectprogram.com/site/global/products_resources/avaya_one_x_agent/interfaces/one_x_agent_api/overview/index.gsp. One issue we found with this API is that it doesn't support CORS headers and the browser may complain about security violation:
XMLHttpRequest cannot load
http://localhost:60000/onexagent/api/registerclient?name=test.
'Access-Control-Allow-Origin' header is present on the requested
resource.
Origin http://<IP_ADDRESS>:8000 is therefore not allowed access.

This has nothing to do with DMCC API though.

Use DMCC API to integrate your .Net backend with AVAYA server and then provide endpoints in your backend to integrate your client with. You can in theory try to run dmcc-ws from this repository and interact with this service instead.

What do you mean by "The dmcc application"?

Thank you so much for the response, I was beginning to loose hope.
a) By "The dmcc application" I mean the sample application provided by avaya (popup Applicaiton).
b) Should I not go for the javascript softphone client provided by avaya? which Im finding hard to understand.
c) "Use DMCC API to integrate your .Net backend with AVAYA server and then provide endpoints in your backend to integrate your client code with" by this do you mean the telephony logic will reside on the server (StartApplicationSession, getDeviceId and so on).
Ill now check the link..

by this do you mean the telephony logic will reside on the server (StartApplicationSession, getDeviceId

Yes – DMCC API is intended to be used by server apps.


Can't find Antiforgery package

When using FluentEmail it seems to work find for text emails, but when I try to use Templates. I get the following error when using .Net Core 3.1
Cannot find reference assembly 'Microsoft.AspNetCore.Antiforgery.dll' file for package Microsoft.AspNetCore.Antiforgery
Is this something that's already known about? Is there a workaround?

Out of curiosity, why does this fix the problem?

I only needed <PreserveCompilationReferences>true</PreserveCompilationReferences> in my .NET 5 web application. Is an issue with RazorLight rather than FluentEmail afaik.

Could this be highlighted in the documentation? This took some time to find the solution.

This does not work in .NET 6

<PropertyGroup>
    <PreserveCompilationReferences>true</PreserveCompilationReferences>
    <PreserveCompilationContext>true</PreserveCompilationContext>
 </PropertyGroup>

Add .csproj file

For net 6 use:
<PropertyGroup>
	<PreserveCompilationReferences>true</PreserveCompilationReferences>
	<PreserveCompilationContext>true</PreserveCompilationContext>
	<MvcRazorCompileOnPublish>false</MvcRazorCompileOnPublish>
	<MvcRazorExcludeRefAssembliesFromPublish>false</MvcRazorExcludeRefAssembliesFromPublish>
</PropertyGroup>

For net 6 use:
<PropertyGroup>
	<PreserveCompilationReferences>true</PreserveCompilationReferences>
	<PreserveCompilationContext>true</PreserveCompilationContext>
	<MvcRazorCompileOnPublish>false</MvcRazorCompileOnPublish>
	<MvcRazorExcludeRefAssembliesFromPublish>false</MvcRazorExcludeRefAssembliesFromPublish>
</PropertyGroup>


Published JSII images don't build JSII packages without further initialization

Describe the bug
Late in the day on 4/30/2024 all our builds started failing:
 Error: Cannot find module 'jsii-rosetta'

We ensured we were using the most recently published version of 1-bookworm-slim-node20 (also published yesterday). Looking at yesterday's JSII release, it's clear that this is driven by this PR. Specifically the PR says that users of yarn v1 will need to install jsii-rosetta manually. The JSII image comes with yarn v1.22.19. There are also npm requirements posted, but it appears that the current images meet those requirements.
Since the latest version of yarn is still 1.22, I'm not sure what can be done to address the yarn requirement listed in the PR. The 2.x versions of yarn appear to still be beta.
It looks like it will be straightforward to update our build process to either update yarn or install jsii-rosetta, but it would be nice if the image was preconfigured with versions of the toolset required to build without additional installation.
Expected Behavior
That our JSII build steps continued to work after a minor release of JSII.
Current Behavior
An additional library jsii-rosetta is required which is not automatically installed by the versions of the package managers installed on the most recent JSII docker images.
Reproduction Steps
yarn run jsii-pacmak
Possible Solution
No response
Additional Information/Context
No response
SDK version used
CDK v2.138.0
Environment details (OS name and version, etc.)
jsii/superchain:1-bullseye-slim-node20

Unfortunately yarn 1 aka Classic is outdated software that does not receive any updates anymore. It has significantly diverged from the rest of the ecosystem.
Modern Yarn is currently at v4: https://yarnpkg.com/blog/release/4.0
Installation for modern Yarn is quite a bit different; it used a new Node tool called corepack as described here:
https://yarnpkg.com/getting-started/install
So the superchain image does indeed ship with modern yarn, it's just not enabled by default (yet). Maybe we should!

It appears that adding the line
    "jsii-rosetta": "^1.92.0",

to our package.json file was all that was required.
FWIW - We need to use 1.92 to match major releases with our outdated version of JSII itself, you should match your version of JSII. And, yes, upon discovering this we will be updating our JSII packages shortly.


ctDNA TSO500 workflow: copy several temp files into final Results

Keeping track of this here before I forget again: if possible copy the following files that get output into the (temporary) run_tso_ctdna_workflow_step folder over to the final Results folder:

fastqc_metrics.csv
replay.json
mapping_metrics.csv
sv_metrics.csv
time_metrics.csv
trimmer_metrics.csv
umi_metrics.csv
wgs_contig_mean_cov.csv
wgs_coverage_metrics.csv
wgs_fine_hist.csv
wgs_hist.csv
wgs_overall_mean_cov.csv
normalizedBinCount.tsv (part of CnvCaller output)
rawBinCount.tsv (part of CnvCaller output)

The post-processing pipeline dumps 6 of those into the AlignCollapseFusionCaller_metrics.json.gz (which makes it into Results), but the rest only come via the MultiQC JSON or not at all.
I don't mind if those get zipped or not - they're pretty small (only the last two are ~1 MB each).

Relevant code: https://github.com/umccr/cwl-ica/blob/main/workflows/tso500-ctdna-post-processing-pipeline/1.0.0/tso500-ctdna-post-processing-pipeline__1.0.0.cwl
Relevant Slack threads:

https://umccr.slack.com/archives/C025TLC7D/p1658388222817319
https://umccr.slack.com/archives/CCC5J2NM6/p1683884902594149
https://umccr.slack.com/archives/CCC5J2NM6/p1675984921930139

Sorry for the delay on this one, the post processing pipeline is a bit of a beast with the way that ICAv1 requires CWL tools to copy data around (doesn't support inline directory creation).
I've completed the following in PR #452

created an extra tool to convert the bin files into a compressed json (I know it's overkill but at least it's then consistent with all of the other outputs),
The missing metric files are now added into the aligncollapsefusioncaller json metrics file
The replay json is added to the top directory of the results.

Currently running through development with wfr.5f9309023fcc4d888230da4a79776067.
Steps to reproduce
# Template generated from autorun wfr.5169dc713dc44aa4afc99fc4ac79fe27 (which is a prod run)
ica-context-switcher --project-name production  # Read-only

# Generate template
cwl-ica create-workflow-submission-template \
  --workflow-path "${CWL_ICA_REPO_PATH}/workflows/tso500-ctdna-with-post-processing-pipeline/1.2.0--1.0.0/tso500-ctdna-with-post-processing-pipeline__1.2.0--1.0.0.cwl" \
  --prefix "test-cttso-with-cnv-bin-files-post-processing" \
  --project "development_workflows" \
  --launch-project "development" \
  --ica-workflow-run-instance-id "wfr.5169dc713dc44aa4afc99fc4ac79fe27" \
  --gds-prefix "gds://development/temp/alexis/cttso-cnv-bin-test" \
  --ignore-workflow-id-mismatch

# Presign files (not accessible in dev otherwise)
# And update resources directory from prod to dev
cp "test-cttso-with-cnv-bin-files-post-processing.template.yaml" "test-cttso-with-cnv-bin-files-post-processing.template.yaml.bak"
yq --output-format=json \
  '
    .input.resources_dir.location = "gds://development/reference-data/dragen_tso_ctdna/ruo-<IP_ADDRESS>/" |
    .engineParameters.maxScatter = 8
  ' \
  < "test-cttso-with-cnv-bin-files-post-processing.template.yaml.bak" | \
gds-presign-my-cwl-inputs \
  --input-json - \
  --output-json - | \
yq \
  --prettyPrint \
  --unwrapScalar > "test-cttso-with-cnv-bin-files-post-processing.template.yaml"

# Launch now we have presigned inputs
bash "test-cttso-with-cnv-bin-files-post-processing.launch.sh"

@alexiswl thanks so much, this will solve a few headaches.
Is the AlignCollapseFusionCaller_metrics.json.gz JSON created for convenience instead of copying over those individual files? It's adding a bit more complexity to the dracarys parser, since first you need to read that JSON into R as a list, then iterate through each different metric group, handle edge cases, pivot, rename columns, then write. It also doesn't help that the original files have differing numbers of fields spread throughout the file (e.g. you'll get rows with an extra comma for some of the metrics in some of the files).
I already have parsers for nearly all those individual source files, since those are standard DRAGEN outputs.
If this needs to stay as-is, I can probably read it in as a list and write each individual sublist as a separate file in tmp, then re-read back with the available parsers. But it becomes slightly hacky.

Is the AlignCollapseFusionCaller_metrics.json.gz JSON created for convenience instead of copying over those individual files?

It stems from https://umccr.slack.com/archives/C05PPTXRCPR/p1624322034007800?thread_ts=1624321089.006400&cid=C05PPTXRCPR
And then from https://umccr.slack.com/archives/C025TLC7D/p1620340847308500 (very long slack thread, tl;dr jsons were decided because it would be easier to read in through something like Athena), is that fair recap @brainstorm? This was decided well-before the days of dracarys or DataBricks.
I think in hindsight with the difficulties of file-handling in ICAv1 through CWL (ICAv2 would be much easier for this on a local disk space), we should have just gone with the csvs as is and a post-processor could have imported them as need be.

I already have parsers for nearly all those individual source files, since those are standard DRAGEN outputs.
If this needs to stay as-is, I can probably read it in as a list and write each individual sublist as a separate file in tmp, then re-read back with the available parsers. But it becomes slightly hacky.

I'll add these csvs in as their own file to the top of the results directory. That'll just be an extra step for CWL.
Pushed again with https://github.com/umccr/cwl-ica/pull/452/commits/d967615eb942d00f158d3d106b3248f9b7ae1707, which will put the csvs and bin count tsvs in the top directory.

Thanks so much, that was quick! And that's a massive, all-encompassing thread btw!
I'll keep a lookout for the wfr.4ecb97d885a44d5bb3b5b473ced88a92 results and ping here if any issues.

gds-ls gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/

gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/dsdm.json.gz
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/evidence.PRJ210166_L2101400.bam
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/evidence.PRJ210166_L2101400.bam.bai
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400.AlignCollapseFusionCaller_metrics.json.gz
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400.bam
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400.bam.bai
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400.bam.md5sum
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400.cleaned.stitched.bam
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400.cleaned.stitched.bam.bai
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400.CnvCallerBinCounts.json.gz
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400_CombinedVariantOutput.tsv
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400_CopyNumberVariants.vcf.gz
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400_CopyNumberVariants.vcf.gz.tbi
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400_ctDNA_alignment_collapse_fusion_caller_multiqc/
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400_Failed_Exon_coverage_QC.json.gz
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400_Failed_Exon_coverage_QC.txt
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400.fastqc_metrics.csv
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400.fragment_length_hist.csv
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400.fragment_length_hist.json.gz
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400_Fusions.csv
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400_Fusions.json.gz
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400.mapping_metrics.csv
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400_MergedSmallVariantsAnnotated.json.gz
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400_MergedSmallVariantsAnnotated.json.gz.jsi
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400_MergedSmallVariants.genome.vcf.gz
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400_MergedSmallVariants.genome.vcf.gz.tbi
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400_MergedSmallVariants.vcf.gz
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400_MergedSmallVariants.vcf.gz.tbi
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400.msi.json.gz
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400_normalizedBinCount.tsv
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400_rawBinCount.tsv
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400-replay.json.gz
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400.sv_metrics.csv
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400.TargetRegionCoverage.json.gz
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400.time_metrics.csv
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400.tmb.json.gz
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400_TMB_Trace.json.gz
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400_TMB_Trace.tsv
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400.trimmer_metrics.csv
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400.umi_metrics.csv
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400.wgs_contig_mean_cov.csv
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400.wgs_coverage_metrics.csv
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400.wgs_fine_hist.csv
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400.wgs_hist.csv
gds://development/temp/alexis/test-post-processing-pipeline/output/20231219_102859/PRJ210166_L2101400/PRJ210166_L2101400.wgs_overall_mean_cov.csv

Epic, thanks so much!


933110: Add .phar and .phps to executable php files

This is a report from the BugBounty event.
Report id: 33574D9R
Payload:
POST /juiceshop/file-upload HTTP/1.1
Host: sandbox.coreruleset.org
Content-Length: 176
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryRGM420pTQRvLu0HE
Connection: close
x-crs-paranoia-level: 4
x-format-output: txt-matched-rules
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.41 Safari/537.36

------WebKitFormBoundaryRGM420pTQRvLu0HE
Content-Disposition: form-data; name="file"; filename=shell.phar

<?php echo '123'; ?>
------WebKitFormBoundaryRGM420pTQRvLu0HE--

Explanation
Since PHP 7 the following extensions are recognized as exectuable PHP scripts:
<FilesMatch ".+\.ph(ar|p|tml)$">
    SetHandler application/x-httpd-php
</FilesMatch>

The rule doesn't recognize .phar nor .phps as executable PHP files. Although .phps is not considered executable now, it is for lower version of PHP.

CRS Bug Bounty PR assessment

Rules affected (list rules): FIXME
Paranoia Level addressed (1, 2, 3, 4, full or explain): FIXME
FTW passes (yes or no) : FIXME
Rule(s) picked for solution (correct or not-correct or explain) : FIXME
Risk for false positives (irrelevant, adequate, substantial or explain) : FIXME
Regular expression quality (inspirational, decent base, needs work, adequate or explain) : FIXME
Documentation (needs work, adequate or explain) : FIXME
Tests (none or some or adequate) : FIXME

Verdict (Unusable, inspirational, usable, almost perfect or perfect) : FIXME

This is not meant to be final. As a CRS dev, feel free to comment below and edit this form directly. As committer or observer, feel free to comment below with feedback and we will think about updating the assessment accordingly.

In the extension list https://github.com/swisskyrepo/PayloadsAllTheThings/tree/master/Upload Insecure Files/Extension PHP there are some extensions not yet caught by this rule: .pht , .phpt.
If this could be added it would be perfect.

@lifeforms Added the two mentioned extensions. Additionally refactored descriptions of previous tests and added missing multipart tests.
IMO there are too many tests that don't add anything to the test bundle and could be compressed to a smaller number of rules. I think it should be sufficient to refactor tests to:

all extensions (.php, .php123, .phar, ...)
1 positive and 1 negative test to test paths (maybe include something like 'aaa.txt/dangerous.php` as well)
1 test to test php... gets blocked correctly
3 multipart tests as they are now
all headers with a chosen extension (x-file-name, x-filename, x_filename, ...)
1-2 other negative tests

Once this PR gets merged I could offer my help in refactoring those.

Thank you @terjanq.
We have updated our assessment. Regex is now adequate, as are the tests and the verdict is now Perfect.
Thank you for your offer to cut down on the excessive tests after merging this. I agree that would make a lot of sense.
Please open an issue and name yourself as a volunteer if you want to follow through. That way we do not lose track of this.


bug

VS Code Version:
OS Version:

Steps to Reproduce:

Does this issue occur when all extensions are disabled?: Yes/No

/needsMoreInfo


Add Matias Zabala to Rookie list

Description
Welcome! We are excited to have you on board. This is your first formal step in making it official that you are a part of the Polkadot Heroes program as either a hero or a rookie. Please fill out your profile details in this pull request. Remember to place your profile in alphabetical order in the list, add your profile image to the ./images subfolder of either the Heroes/ or Rookies/ folder, and fill out all the informational fields in your card. You can copy and paste someone else's card and replace their information with your own.
Make sure to preview your additions in the Heroes/README.md or the Rookies/README.md file to confirm it looks as expected.
Checklist:

[x] My name is alphabetically ordered by first name in the README
[x] My profile picture is added to the /images folder of either the Heroes/ or the Rookies/ directory
[x] All the fields are filled out for my profile (see the other profiles for examples)

Hi @0xMatiz! Welcome! We are so delighted you're here. 🎉
Just leaving one small comment to fix and then we're ready to merge. Please see the review comment below.


H.264 in filename incorrectly interpreted as season 2, episode 64

# Detected series: Connections - 10 - Yesterday, Tomorrow and You - H (season: 2, episode: 64)
ShowNotFound(u'Show Connections - 10 - Yesterday, Tomorrow and You - H not found on www.thetvdb.com',)
####################
Old filename: Connections - 10 - Yesterday, Tomorrow and You - H.264.m4v
New filename: Connections - 10 - Yesterday, Tomorrow and You - H - [02x64].m4v

There is a filename pattern to handle show.123.avi as season 1, episode 23 - I'm tempted to remove this from the default patterns as it is very ambiguous, like in this case
In this case you'd have to add a custom pattern to handle that filename, as filenames with no season number aren't handled (once "fixed", that file will just be skipped over with a Invalid filename: Cannot parse ... warning)

Thank you for this!


BUG: multiplying CSR matrix by a constant instantiates huge dense index arrays

Reproducing Code Example
from scipy.sparse import coo_matrix

A = coo_matrix(([5.0,5.0], ([0, 1000_000], [0, 1000_000])))
A = A.tocsr()
A[:,0:10000] *= -1

Error message
Traceback (most recent call last):
  File "/tmp/test_sparse_mul.py", line 5, in <module>
    A[:,0:10000] *= -1
  File "/home/s/.local/lib/python3.9/site-packages/scipy/sparse/_index.py", line 116, in __setitem__
    self._set_arrayXarray_sparse(i, j, x)
  File "/home/s/.local/lib/python3.9/site-packages/scipy/sparse/compressed.py", line 808, in _set_arrayXarray_sparse
    self._zero_many(*self._swap((row, col)))
  File "/home/s/.local/lib/python3.9/site-packages/scipy/sparse/compressed.py", line 924, in _zero_many
    i, j, M, N = self._prepare_indices(i, j)
  File "/home/s/.local/lib/python3.9/site-packages/scipy/sparse/compressed.py", line 874, in _prepare_indices
    i = np.array(i, dtype=self.indices.dtype, copy=False, ndmin=1).ravel()
numpy.core._exceptions.MemoryError: Unable to allocate 37.3 GiB for an array with shape (1000001, 10000) and data type int32

SciPy/NumPy/Python version information
1.7.1 1.19.5 sys.version_info(major=3, minor=9, micro=5, releaselevel='final', serial=0)

This only happens when the slice is unassigned.  I.e., the following works:
B = A[:,0:10000]
B *= -1

Thanks for looking into this.  The code above works, but doing
A[:,0:10000] = B
results in a similar problem.  So there is no way to change sign
of selected elements.  Actually the slice sign change operation
is used in scipy.optimize.linprog.
The following works:
A.data[A.indices < 10000] *= -1
but is a hack.


Add logo for Red Hat Registry

I'd love to see the Red Hat logo in the Registries menu after a successful login.  As shown below, there is a visual difference between the preset registries and the one added by the extension.  I'd like to fill this gap.
I am also looking into figuring out an alternative name to replace the URL registry.redhat.io with a more human-friendly name.

Ignore until https://github.com/redhat-developer/podman-desktop-redhat-account-ext/pull/35 has merged.
Cc: @dgolovin @benoitf @jeffmaury @slemeur

I would just avoid to make this extension bigger and bigger and keep a separation of concerns.
IMHO this extension should only manage the authentication and delegate/require to other extensions
For example stuff for registries ==> redhat registry extension
or this extension is no longer fitting in a pure 'identity provider' and more a 'all-in-one'

we should be able to update logos or add new Red Hat registries, etc without needing to release a new version of the redhat-account extension

I guess I was thinking of this as a 'RH accounts' extension. I'm ok moving to another podman-desktop-redhat-* extension, which would need to depend on this one so you can actually login to the registry.

BTW icon and name for the registry does not survive podman desktop restart (see https://github.com/containers/podman-desktop/issues/6218)


Add Sources to API

Details

Adds an ADR document describing adding Sources as richer objects on the API
Adds proposed implementation

Pivotal Story (if relevant)
Story URL: https://www.pivotaltracker.com/story/show/186711160
Related PRs
Followed by https://github.com/bbc/tams/pull/19
Submitter PR Checks
(tick as appropriate)

[x] Added bbc/rd-apmm-cloudfit team as a reviewer
[X] PR completes task/fixes bug
[X] API version has been incremented if necessary
[ ] ADR status has been updated, and ADR implementation has been recorded
[X] Documentation updated (README, etc.)
[X] PR added to Pivotal story (if relevant)
[ ] Follow-up stories added to Pivotal

Reviewer PR Checks
(tick as appropriate)

[ ] PR completes task/fixes bug
[ ] Design makes sense, and fits with our current code base
[ ] Code is easy to follow
[ ] PR size is sensible
[ ] Commit history is sensible and tidy

Info on PRs
The checks above are guidelines. They don't all have to be ticked, but they should all have been considered.

This implementation diverges from NMOS in some ways such as the exclusion of the high level media type (e.g. audio/video/mux/data) from the Source metadata. I'd suggest that unless we have a specific reason to diverge, we should build on previous work and include that metadata.

ADR document updated with discussion outcomes from Tuesday's session - I'll request re-review before merge


[Experiment] Prototype of multi-version models

Initial prototype of simple multi-version client described in https://github.com/Azure/autorest.csharp/issues/2362
At present, it covers only the case where an optional property is added to a round-trip model across versions.
TODO

[ ] Add convenience API
[ ] Generate model
[ ] Address serialization
[ ] Add test cases for back-compat
[ ] Test cases for forward compat?

The Cadl file describing this client is here: main-v2client.cadl, and is discussed in this README.

API change check
APIView has identified API level changes in this PR and created following API reviews.
Azure.Template


Node.js 10.x support

Amazon announced that they drop support for Node8 - Switching the Node Version for the Lambda function to 10.x results in a response
{
message: "Internal server error"
}

Amazons Notice:
We are contacting you as we have identified that your AWS Account currently has one or more Lambda functions using Node.js 8.10, which will reach its EOL at the end of 2019.

> What’s happening?

The Node community has decided to end support for Node.js 8.x on December 31, 2019 [1]. From this date forward, Node.js 8.x will stop receiving bug fixes, security updates, and/or performance improvements. To ensure that your new and existing functions run on a supported and secure runtime, language runtimes that have reached their EOL are deprecated in AWS [2].

For Node.js 8.x, there will be 2 stages to the runtime deprecation process:

1. Disable Function Create – Beginning January 6, 2020, customers will no longer be able to create functions using Node.js 8.10

2. Disable Function Update – Beginning February 3, 2020, customers will no longer be able to update functions using Node.js 8.10

After this period, both function creation and updates will be disabled permanently. However, existing Node 8.x functions will still be available to process invocation events.

> What do I need to do?

We encourage you to update all of your Node.js 8.10 functions to the newer available runtime version, Node.js 10.x[3]. You should test your functions for compatibility with the Node.js 10.x language version before applying changes to your production functions.

> What if I have issues/What if I need help?

Please contact us through AWS Support [4] or the AWS Developer Forums [5] should you have any questions or concerns.

It looks like the CustomResourceFunction is fine but that there is an issue with the image handler function. Looking at the error logs it appears that it is probably that sharp-node needs to be compiled for a specific version of Node.
I am going to need this so I will take a look and see if I can get it working and if I can I will create a PR.

I created a pull request for this. #157
To make use of it before the changes are applied to the main build you will have to follow the example for a custom deploy in the readme. Of note make sure when you build that you do it with node 10 as the problem with sharp was that it was built for NodeJS 8, I am not entirely sure if minor version differences are an issue.

I tried compiling it for node10, and it seems to work well then.

We are trying to update the solution in this year to support Node.js 10.x.

Do you have any news ? (because it is almost the end of the year...)

Hi @lmontier,
Sorry for the delay in the response.
We are still working on the update, and we’re planning to release it before December 31st.
We’re also going to use node12 (instead of node10), since this version will be supported until April 2022 (https://aws.amazon.com/about-aws/whats-new/2019/11/aws-lambda-supports-node-js-12/).

Will the update come before Christmas? Changes at the end of the year are generally a bad idea. It would be good if the solution is ready soon so that we can use it safely and tested.

Hi @martin,
Yes, we're planning to release it before Christmas.

@dscpinheiro , any update christmas is coming up soon

HI All - The latest commit contains the Node updates. Please let us know if you need additional assistance.

@georgebearden what is the recommended way to upgrade without any downtime?

Hi @andidev,
You should be able to update an existing stack with the latest template (https://s3.amazonaws.com/solutions-reference/serverless-image-handler/latest/serverless-image-handler.template). That will require minimum downtime since the lambda function runtime has changed.

@andidev it depends on what sharpJS features you are using, because 0.23.3 doesn't really compatible with 0.21.3.

@dscpinheiro can you dumb it down a little bit for me. I cant find anything on updating templates for Lambdas on AWS.

@brad13x, here's what I meant:

Go to your existing stack and click update:

On the Update stack screen, select "Replace current template" and specify the latest template (https://s3.amazonaws.com/solutions-reference/serverless-image-handler/latest/serverless-image-handler.template):

@dscpinheiro Thank you so much! This helped a lot. Did not know about this area of the site.

Is anyone else getting this error? 1 validation error detected: Value 'serverlessrepo-image-resizer-serviceImageHandlerFunctionRole-us-west-2' at 'roleName' failed to satisfy constraint: Member must have length less than or equal to 64 (Service: AmazonIdentityManagement; Status Code: 400; Error Code: ValidationError;

any advices?
CloudWatch Logs role ARN must be set in account settings to enable logging (Service: AmazonApiGateway; Status Code: 400; Error Code: BadRequestException;

@brad13x When you launch the CloudFormation stack under the hood it creates IAM::Role with the name of the format {CloudFormationName}ImageHandlerFunctionRole-{CloudFormationRegion}. In your case it is serverlessrepo-image-resizer-serviceImageHandlerFunctionRole-us-west-2.
The problem is that the resulting IAM::Role name should be less than or equal to 64 characters. But in your case, there are 70 characters.
To solve this issue assign the CloudFormation shorter name with no more than 26-30 characters, depending on the region where you deploy the stack.
Unfortunately, you can not update the CloudFormation name, so you have to delete the current stack and create the new one. Here is the field where you have to set the name with the proper length:

I get some errors when i update the Stack with the new template file:
my_stackname, UPDATE_ROLLBACK_IN_PROGRESS: The following resource(s) failed to create: [Logs, ApiLoggingRole]. The following resource(s) failed to update: [ImageHandlerApiDeployment, CustomResourceCopyS3, CustomResourceAnonymousMetric].
and
ImageHandlerApiDeployment, UPDATE_FAILED: CloudWatch Logs role ARN must be set in account settings to enable logging (Service: AmazonApiGateway; Status Code: 400; Error Code: BadRequestException; Request ID: 87a64f3-xxxx-xxx-…)
and
Resource creation cancelled for CustomResourceAnonymousMetric, CustomResourceCopyS3, ApiLoggingRole and Logs with status UPDATE_FAILED
In the update view Configure stack options i select nothing for Permissions -> IAM role - optional (like before). And for Advanced options -> Stack policy i have select  No stack policy like before.
In the Review view with Change set preview i have 16 changes (4 "new" and 12 "Modify" actions).
Any ideas? @georgebearden, @dscpinheiro

Hi @martin,
It looks like it's the same issue as https://github.com/awslabs/serverless-image-handler/issues/172.
I'll investigate and reply there.

@martin, @andidev
Yesterday I wrongly said that updating was possible, but it turns out that is not the case.
I apologize for the mistake, and I listed some alternatives on https://github.com/awslabs/serverless-image-handler/issues/172#issuecomment-566583834, and I'll make sure this information is included on the implementation guide.
Again, sorry for the inconvenience.

Hi everyone,
Before I've updated the image handler to NodeJS 12.x it was possible to enter a URL like:
_https://xxxx.cloudfront.net/fit-in/1500x0/image.jpg_
The response was the image with a width of 150px and the height according to the aspect ratio.
Now, after the update, I get the following Error in CloudWatch:
Expected positive integer for height but received 0 of type number
at Object.invalidParameterError (/var/task/node_modules/sharp/lib/is.js:101:10)
at Sharp.resize (/var/task/node_modules/sharp/lib/resize.js:225:18)
at ImageHandler.applyEdits (/var/task/image-handler.js:72:27)
at ImageHandler.process (/var/task/image-handler.js:27:46)
at Runtime.exports.handler (/var/task/index.js:24:53)
at processTicksAndRejections (internal/process/task_queues.js:93:5)
Is it still possible to enter only a width and get the image back with the height according to the aspect ratio?
Thanks in advance!

Hi @marco910,
We had to update the version of sharp, and I believe the issue you mentioned is the same one as this: #171
Could you try this approach (https://github.com/awslabs/serverless-image-handler/issues/171#issuecomment-566656678) and check if the problem still happens?

Hi @dscpinheiro,
Yes that's the same issue as in #171 . @mberneis is requesting the image with JSON but in my case the request is with a "normal" URL. How can I make a valid request with a URL?

Hi @marco910,
So I just tested and it doesn't look like it's possible to do the request you want via URL on v4.1. Unfortunately, as I mentioned on the other issue, we had to update sharp for compatibility with Amazon Linux 2.
The alternative is to do the request via JSON, and I'll update the documentation to make that clear.

@dscpinheiro Can't you make the handler convert a 0 passed on the url into a null?

@dscpinheiro Can't you make the handler convert a 0 passed on the url into a null?

Yes, that's definitely an option. I'll add this change to our backlog for the next release.

@dscpinheiro Can't you make the handler convert a 0 passed on the url into a null?

Yes, that's definitely an option. I'll add this change to our backlog for the next release.

Hi, When can we expect the feature back in the image handler?


New Sample script to resubmit & cancel flow runs

Script to Resubmit failed & Cancel running flow runs

Linked Issue
Resubmit all failed flow runs for a flow in an environment. Closes #2440
Cancel all running flow runs for a flow in an environment. Closes #2441

New sample scripts = Two Files

docs/docs/sample-scripts/flow/resubmit-all-failed-flow-runs.md
docs/docs/sample-scripts/flow/cancel-all-running-flow-runs.md

Main Branch Only

Awesome! Thank you! We'll review it in the coming days 👍

Merged manually. Thank you and welcome to the crew! 👏

Thank you so much @waldekmastykarz.


Implementing setSpeed for android

This uses the feature of the andoid media player to implement setSpeed.
The patch from https://github.com/zmxv/react-native-sound/pull/162 was need to actually get the latest master running on android.
Some initial ref: http://stackoverflow.com/a/37094353/2297277
I'm new to the project and the react native world so let me know if something need to be reworked in this PR.

@twitwi Looks great - thank you. It looks like you've combined #162 in here, so if we merge this one we can close #162.
Code changes look good - one small thing, can you update the feature matrix on the main readme? Then I'll merge this. Thanks!

Awesome, could this be merged and released? The current 0.10.0 cannot play any sound on Android since a exception is thrown where that should not happen for local files


Recognise cursor column for completion result

First of, THANK YOU 🙏  this project is amazing!

monacopilot version: 0.15.0
monaco-editor version: 0.52.0
node version: 22.11.0
pnpm version: 9.4.0

Relevant code or config
const copilot = new Copilot(process.env.ANTHROPIC_API_KEY!, {
  provider: 'anthropic',
  model: 'claude-3-5-sonnet',
})
const completion = await copilot.complete({ body })

What you did: everything according to the docs.
What happened: The completion metadata looks as following:
{
   completionMetadata: {
     language: 'json',
     textBeforeCursor: '{\n' +
       '  "tags": [\n' +
       '    "Contacts"\n' +
       '  ],\n' +
       '  "summary": "Get Contact List",\n' +
       '  "parameters": [\n' +
       '    {\n' +
       '      "name": "Authorization",\n' +
       '      "in": "header",\n' +
       '      "schema": {\n' +
       '        "type": "string",\n' +
       '        "description": "Bearer authentication token"\n' +
       '      },\n' +
       '      "example": "Bearer {{token}}"\n' +
       '    }\n' +
       '  ],\n' +
       '  "responses": {\n' +
       '    "200": {\n' +
       '      "description": "OK",\n' +
       '      "content": {\n' +
       '        "text/plain": {\n' +
       '          "schema": {\n' +
       '            "type": "array",\n' +
       '            "items": {\n' +
       '              "type": "object",\n' +
       '              "pro',

Problem description: the cursor may not be at the beginning of the editor, however the completion result returns a result that assumes that.
Suggested solution: as a workaround I currently have the following code:
  const initialSpaces = body.completionMetadata.textBeforeCursor.split('\n').pop()?.match(/^\s+/)?.[0] || ''
  if (initialSpaces.length > 0 && completion.completion) {
    completion.completion = completion.completion.replace(/\n/g, `\n${initialSpaces}`)
  }

I am sure this may be useful for anyone using the package so I wonder if it makes sense to make this the default behavior.
Thanks again!

Thank you for pointing out this, Will fix and let you know!

Could you please share a screen record for clarity if possible?

@christian-bromann Fixed it in v0.15.2 🚀

Awesome, thanks a ton!


generate_counterfactuals call results in error on Tensorflow v2.9

Hello,
We recently updated to Tensorflow v2.9. Since the upgrade, we've had trouble with the generate_counterfactuals function call. Below is my code and error. Note: The same code was working on Tensorflow 1.x.
import numpy as np
import pandas as pd
import tensorflow as tf

dataset = pd.read_csv('data_NN_for_classification_binary.csv')

categorical_names = ['Geography','Gender']
target_name = ['Exited']

from sklearn.preprocessing import OneHotEncoder
encoder = OneHotEncoder(handle_unknown='ignore')
encoded_df = pd.DataFrame(encoder.fit_transform(dataset[categorical_names]).toarray())
encoded_df.columns = encoder.get_feature_names(categorical_names)
dataset = pd.concat([dataset,encoded_df], axis='columns')
dataset = dataset.drop(categorical_names, axis='columns')
encoded_df = None

target_df = dataset[target_name]
dataset = dataset.drop(target_name, axis='columns')
dataset = pd.concat([dataset,target_df], axis='columns')
target_df = None

features = dataset.columns.values[:-1]

X = dataset.iloc[:, :-1].values
y = dataset.iloc[:, -1].values
class_names = dataset["Exited"].unique()

from keras.utils import np_utils
y = np_utils.to_categorical(y)

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 0)

import keras
from keras.models import Sequential
from keras.layers import Dense
from keras.models import model_from_json

ann = Sequential()

ann.add(Dense(units = len(features), kernel_initializer = 'uniform', activation = 'relu'))

ann.add(Dense(units = 12, kernel_initializer = 'uniform', activation = 'relu'))

ann.add(Dense(units = 2, kernel_initializer = 'uniform', activation = 'softmax'))

ann.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])

ann.fit(X_train, y_train, batch_size = 32, epochs = 10)

y_pred = ann.predict (X_test[0:1])

import dice_ml
dice_data = dice_ml.Data(dataframe=dataset,
                         continuous_features = features.tolist(),
                         outcome_name = 'Exited')

dice_model = dice_ml.Model(model=ann,
                           backend="TF2")

dice_explainer = dice_ml.Dice(dice_data,
                        dice_model,
                        method="random")

query_instance = {'CreditScore':714,
                'Age':45,
                'Tenure':8,
                'Balance':150900,
                'NumOfProducts':2,
                'HasCrCard': 0,
                'IsActiveMember':1,
                'EstimatedSalary': 139889,
                'Geography_France': 0,
                'Geography_Germany': 0,
                'Geography_Spain': 1,
                'Gender_Female': 0,
                'Gender_Male': 1}

counterfactuals = dice_explainer.generate_counterfactuals(query_instance,
                                       total_CFs = 1,
                                       desired_class = 'opposite',
                                       features_to_vary=['CreditScore', 'Tenure', 'EstimatedSalary'],
                                       permitted_range={'CreditScore': [400, 800],
                                                        'Tenure': [1, 7],
                                                        'EstimatedSalary': [5000, 999999]})

counterfactuals.visualize_as_dataframe(show_only_changes=True)

Executing this results in the following error from the generate_counterfactuals function call
Traceback (most recent call last):

  File "C:\Users\test\.conda\envs\dice-env\lib\site-packages\spyder_kernels\py3compat.py", line 356, in compat_exec
    exec(code, globals, locals)

  File "d:\my files\dice\ai\code\sample\dice_test\dice_test.py", line 188, in <module>
    counterfactuals = dice_explainer.generate_counterfactuals(query_instance,

  File "C:\Users\test\.conda\envs\dice-env\lib\site-packages\dice_ml\explainer_interfaces\dice_tensorflow2.py", line 117, in generate_counterfactuals
    self.minx, self.maxx = self.data_interface.get_minx_maxx(normalized=True)

  File "C:\Users\test\.conda\envs\dice-env\lib\site-packages\dice_ml\data_interfaces\public_data_interface.py", line 302, in get_minx_maxx
    minx[0][idx] = (self.permitted_range[feature_name]

KeyError: 'Age'

Age is not one of the features we expect dice to vary. Again, we never had this issue with an older version of TF.

I'd like to add that if I remove the line specifying permitted_range, that is if I change the generate_counterfactuals line to below
counterfactuals = dice_explainer.generate_counterfactuals(query_instance,
                                       total_CFs = 2,
                                       desired_class = 'opposite',
                                       features_to_vary=['CreditScore', 'Tenure', 'EstimatedSalary'])

then the execution is initiated, but after several seconds I receive the error
Traceback (most recent call last):

  File "C:\Users\test\.conda\envs\dice-env\lib\site-packages\spyder_kernels\py3compat.py", line 356, in compat_exec
    exec(code, globals, locals)

  File "d:\my files\dice\ai\code\sample\dice_test\dice_test.py", line 188, in <module>
    counterfactuals = dice_explainer.generate_counterfactuals(query_instance,

  File "C:\Users\test\.conda\envs\dice-env\lib\site-packages\dice_ml\explainer_interfaces\dice_tensorflow2.py", line 132, in generate_counterfactuals
    self.find_counterfactuals(query_instance, desired_class, optimizer,

  File "C:\Users\test\.conda\envs\dice-env\lib\site-packages\dice_ml\explainer_interfaces\dice_tensorflow2.py", line 567, in find_counterfactuals
    final_cfs_df_sparse = self.do_posthoc_sparsity_enhancement(

  File "C:\Users\test\.conda\envs\dice-env\lib\site-packages\dice_ml\explainer_interfaces\explainer_base.py", line 537, in do_posthoc_sparsity_enhancement
    final_cfs_sparse = self.do_linear_search(diff, decimal_prec, query_instance, cf_ix,

  File "C:\Users\test\.conda\envs\dice-env\lib\site-packages\dice_ml\explainer_interfaces\explainer_base.py", line 559, in do_linear_search
    while((abs(diff) > 10e-4) and (np.sign(diff*old_diff) > 0) and self.is_cf_valid(current_pred)):

  File "C:\Users\test\.conda\envs\dice-env\lib\site-packages\dice_ml\explainer_interfaces\explainer_base.py", line 724, in is_cf_valid
    pred_1 = model_score[self.num_output_nodes-1]

IndexError: index 1 is out of bounds for axis 0 with size 1

@nagainfosolutions the tensorflow code was updated in the latest version v0.9. Can you try using the latest version from pip? Let me know if the error still persists.

This issue is fixed now. Thank you for the quick resolution. Much appreciated.


cairo: add X11 support

Hello,
The cairo formula currently can't be built on Linux with the --with-x11 option.
This patch adds xlib support.

Curious, I have done this before as well. Do you have to fake the x11 packages? Because the pkgconfig files for e.g. xcb cannot be found otherwise.

What error message are you seeing? Please report brew gist-logs cairo
Does it help to sudo apt-get install libx11-xcb-dev?

I don't get any error message. Without my patch, cairo is simply built without xlib support (the option is not available).
Then I get this kind of error when I try to build formulae like gtk+:
No package 'cairo-xlib' found

Ah, I see. Does installing libx11-xcb-dev help?

It is already installed.

I'd prefer to figure out why `xcb

Here it is: https://gist.github.com/Rudloff/ebd6baa383b0e64de13f

Sorry, I meant to say include the first chunk depends_on :x11 => :optional if MacOS.version > :leopard || OS.linux? but not the second chunk --enable-xcb=no and report brew install cairo --with-x11; brew gist-logs cairo

Oh, OK.
In this case, I do get an error:
checking for cairo's XCB surface backend feature...
checking for xcb... no
checking whether cairo's XCB surface backend feature could be enabled... no (requires xcb >= 1.6 xcb-render >= 1.6 http://xcb.freedesktop.org)
configure: error:  XCB surface backend feature could not be enabled

Here are the logs: https://gist.github.com/Rudloff/229344610460955b98fd

Does sudo apt-get install libxcb-render0-dev help?
What version of xcb and xcb-render do you have installed?

It is already installed (1.10-3+b1).

https://gist.github.com/Rudloff/229344610460955b98fd#file-config-log-L2356-L2357
Package xcb was not found in the pkg-config search path.
Perhaps you should add the directory containing `xcb.pc'

Set PKG_CONFIG_PATH to the directory containing xcb.pc.
See also #47

Yeah, that is why I ask my question in my reply. I don't think linuxbrew supports the ":x11" dependencies or am I wrong? In the past what I have to do is to change PKG_CONFIG_PATH / PKG_CONFIG_LIB in the formula, but it seems in this PR adding the dependency of ":x11" will work just fine.

Correct. The :x11 dependency has no effect on Linuxbrew.

Thanks for the clarification @sjackman .
@Rudloff In that case I am wondering how this PR works as the pkgconfig for xcb will not be found from linuxbrew, unless it is manually symlinked somewhere in linuxbrew's pkgconfig search path (is that what you did?), or additional paths are added in PKG_CONFIG_PATH in the formula (which is not the case in this PR)

I've merged the first chunk of this patch.
-  depends_on :x11 => :optional if MacOS.version > :leopard
+  depends_on :x11 => :optional if MacOS.version > :leopard || !OS.mac?

Thank you for your contribution, Pierre.