Migrate the Single Cell Eng wiki page to how-we-work/

Currently our "how we work" documentation is distributed across Github and the CZI Wiki. Move all the information in the wiki to pages in the how-we-work folder, update the contents to reflect present day, de-duplicate information, remove all the contents from the Wiki, and update the Wiki with a link to our how-we-work folder in Github.

Closing -- we are moving our docs primarily to Google Drive and the wiki page has been deleted.


Account not issued by...

Error using api gateway access.
Added namespace when running micro.
micro api --namespace=go.micro.wPan
Account not issued by go.micro.wPan
why?
thank！

Which release are you using?

go 1.13
github.com/micro/go-micro v1.18.0
github.com/micro/go-plugins v1.5.1

What version of micro? This is the micro api

cmd micro --version
==>micro version latest

the actual release version? how are you downloading micro?
if you are running an old version please get the latest. Otherwise if you plan to stay with go-micro v1.18 please downgrade micro

I have the same problem,  go-micro version:
github.com/micro/cli/v2 v2.1.2
github.com/micro/cli/v2 v2.1.2/go.mod
github.com/micro/go-micro v1.18.0
github.com/micro/go-micro/v2 v2.9.0
github.com/micro/go-micro/v2 v2.9.0/go.mod

Hello, I have the same issue...
go version go1.14.2 darwin/amd64
micro version latest

micro  --auth_namespace=go.username  api --namespace=go.username

Me too,
go 1.14
go.mod
...
github.com/micro/go-micro/v2 v2.8.0
replace github.com/micro/go-micro/v2 => github.com/micro/go-micro/v2 v2.8.1-0.20200604083217-bd3ef6732811
docker-compose:
micro:
command: api --handler=api --namespace com.dinamonetworks
image: micro/micro:latest
ports:
- "9090:8080"
Message:
Account not issued by com.dinamonetworks

hi,all:
Me too

micro --auth_namespace=go.username api --namespace=go.username

This works for me!

Looks like there is a resolution, otherwise please file issue in micro/micro for any further problems


Fixes #3370 .Fix and change some code structure

Fixes #3370 .
Changes proposed in this pull request:

use jdbc api to fix oracle difference and fix other jdbc compatibility
Fix jdbc getIndexInfo method and NullPointerException.
Fix and change catalog and schema method parameter error.
Fix and change resultSet.getString(IS_NULLABLE) and
resultSet.getString(IS_AUTOINCREMENT) return type error.(at MySQL Connector/J 8.0 + not work)
Change some code structure for current compatibility
Change and add  TEST unit

Pull Request Test Coverage Report for Build 922

120 of 170 (70.59%)  changed or added relevant lines in 24 files are covered.
4 unchanged lines in 2 files lost coverage.
Overall coverage decreased (-0.04%) to 68.461%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

sharding-core/sharding-core-common/src/main/java/org/apache/shardingsphere/core/metadata/datasource/dialect/H2DataSourceMetaData.java
12
13
92.31%

sharding-core/sharding-core-common/src/main/java/org/apache/shardingsphere/core/metadata/datasource/dialect/MySQLDataSourceMetaData.java
12
13
92.31%

sharding-core/sharding-core-common/src/main/java/org/apache/shardingsphere/core/metadata/datasource/dialect/OracleDataSourceMetaData.java
12
13
92.31%

sharding-core/sharding-core-common/src/main/java/org/apache/shardingsphere/core/metadata/datasource/dialect/PostgreSQLDataSourceMetaData.java
12
13
92.31%

sharding-core/sharding-core-common/src/main/java/org/apache/shardingsphere/core/metadata/datasource/dialect/SQL92DataSourceMetaData.java
12
13
92.31%

sharding-core/sharding-core-common/src/main/java/org/apache/shardingsphere/core/metadata/datasource/dialect/SQLServerDataSourceMetaData.java
12
13
92.31%

sharding-core/sharding-core-common/src/main/java/org/apache/shardingsphere/spi/database/H2DatabaseType.java
0
1
0.0%

sharding-core/sharding-core-common/src/main/java/org/apache/shardingsphere/spi/database/MariaDBDatabaseType.java
0
1
0.0%

sharding-core/sharding-core-common/src/main/java/org/apache/shardingsphere/spi/database/MySQLDatabaseType.java
0
1
0.0%

sharding-core/sharding-core-common/src/main/java/org/apache/shardingsphere/spi/database/OracleDatabaseType.java
0
1
0.0%

Files with Coverage Reduction
New Missed Lines
%

sharding-core/sharding-core-common/src/main/java/org/apache/shardingsphere/core/metadata/datasource/dialect/MariaDBDataSourceMetaData.java
2
0.0%

sharding-core/sharding-core-execute/src/main/java/org/apache/shardingsphere/core/execute/metadata/TableMetaDataLoader.java
2
0.0%

Totals

Change from base Build 555:
-0.04%

Covered Lines:
11609

Relevant Lines:
16957

💛  - Coveralls


Any way to customize the derivation path format?

Currently polkadot portal only allow derivation path in //0~19//0~19 format
But most people may come from parity signer, which use //polkadot as the default derivation path
I think it would be great if there's a way to use a user-specified derivation format
so that those who came from parity signer can use the same seed on a ledger, rather than unbonding funds and wait for a month then transfer to the new wallet.

The derivation path itself is not really the problem.
For security reasons, Ledger apps do not have access to the mnemonic. In a Ledger device, all apps share keys derived from the same mnemonic and key derivation is restricted by the SDK.
Ledger devices follow the de-facto standard BIP39 and BIP44. Substrate unfortunately is not BIP39/BIP44 compatible. You can find parity's decision here: https://github.com/paritytech/substrate-bip39#substrate-bip39
We will discuss this with Ledger as this will require a  complete firmware update but we cannot promise when/if this will happen.


Failed to set (leftSide) user defined inspected property on (UINavigationController): [<UINavigationController  setValue:forUndefinedKey:]: this class is not key value coding-compliant for the key leftSide.

New Issue Checklist

I have read the guidelines for contributing and I understand:

[ ] My issue is happening in the latest version of SideMenu.
[x] My issue was not solved in the README.
[ ] My issue can not be answered on stackoverflow.com.
[ ] My issue is not a request for new functionality that I am unwilling to build and contribute with a pull request.
[ ] My issue is reproducible in the demo project.

Issue Description

Please follow the README carefully.

I read but didn't get idea. Thanks


fix(webrtc): expose ice servers on cors requests

The link header was getting ignored by browser fetch as it was not exposed by Access-Control-Expose-Headers
fixes #2480

merged, thanks!

added in v1.2.0


Add Snackbar

Is your feature request related to a problem? Please describe.
No.

@Madonahs Is there a problem here? I'm here to help

@Emmanuel it's a suggestion. You can add some if you have time. 😀


Azure's userinfo endpoint not working in v2

When using v2 of the Azure API, in profile() it tries to call /openid/userinfo.
However, according to the passport-azure-ad node package (sorry, this is the best documentation I could find) userinfo does not work with v2, and one needs to extract that information from id_token.

These are the adaptations that need to be made for v2:

JWK_SET_URL should be '{}{}/discovery/v2.0/keys' instead of '{}{}/discovery/keys'.
The profile needs to be fetched by callingparse_openid, which extracts it from the id_token.
For the above to work, iss claim validation needs to be disabled by calling create_azure_backend with claims_options={}. This is because Microsoft reports specifies its issuer is "https://login.microsoftonline.com/{tenantid}/v2.0", and expects you replace {tenantid} with the value they provide in the tid claim. Without this replacement, validation fails. For additional reference, here's a blog post that covers this issue.

@salty-horse can you send a PR for this issue?

Is it OK to remove the issuer validation? I'm not sure how to replace the tenant ID without modifying authlib code.

@salty-horse You can disable it by setting claims_options:
    "iss": {"essential": False},

However, according to your description, it seems you can use claims_options:
def validate_iss(claims, value):
    iss = "https://login.microsoftonline.com/{}/v2.0".format(claims['tid'])
    return iss == value

claims_options = {
    "iss": {"essential":  True, "validate": validate_iss},
}


Phantom go

Add the game of Phantom Go based on Go Tagging @michalsustr for review.
Creating new request beacuse of failing CLA in #743 pull request.

Hi @Syor,
Do you see this panel?

If so, can you click on "Details" to the right of cla/google "X Missing CLA from one or more contributors" and follow the instructions at that link?

Hello @lanctot,
Thank you for helping me with the CLA. I found out that I made this pull request also with my school e-mail. Following the instructions, I signed it also on my school e-mail. Should I write some comment tagging the bot to refresh the CLA check? I am not sure if something has changed.

Great, yes now try tagging the bot like last time?

@googlebot I signed it

Seems to have not worked. Is it possible to open a PR using your github account directly?

A ha! I figured it out. Things have changed in how the CLA works and it looks like now I have to manually re-run it a different way (the googlebot seems to no longer exist). I will start a run of the tests now.

Ok pyspiel_test failed.
When adding a new game, you need to add it to the list here: https://github.com/deepmind/open_spiel/blob/c3d50a3fb3de7846bcba632619649b9f5d2f284c/open_spiel/python/tests/pyspiel_test.py#L30

Hello @lanctot,
Thank you for your help with the CLA.
About the pyspiel_test, I have added the game in my local project, but I forgot to add it to the commited files, I will fix it in next commit.

Hi @Syor, I made a pass through the code.

@Syor can you let me know when you are finished implementing this and when it's ready to import?

Hello @lanctot, I will let you know as soon as I finish fully implementing this game. I have to implement ResampleFromInfostate to be able to use is-mcts on this game, and to reimplement few methods to match the imperfect information.

 I reverted the files into their state before my accidental push of my local implementations.

You will probably also need to pull in changes from master (from an hour ago).

Hi @Syor @michalsustr, we're discussing a (v1.1) release mid-Feb to early March. Wondering if this game will be ready to be imported before then? Would be cool to include it in the release, if possible.

Hello @lanctot. If everything goes right, I should finish my winter semester exams by 2. February. There are two "things" that are to be implemented, first and easier is to add Observer, the harder one is to implement Information State sampling. Right now, I would say that the resampling method is about 70% done, right now I am stuck on some enemy stone placements. So if the remaining part of the resampling isn't too difficult, I think I can finish it until the release.

Hi @Syor, just an update: we are planning a release for next week, Feb 28th.
If you do not finish the game before that, I will do another minor release soon after it is finished and merged.

Alternatively, we can import the game without the ResampleFromInfoState function (which you can add later) if the rest is ready; that way it can make it into the 1.1 release.

Hello @lanctot. With @michalsustr we have decided to use OT-Tools CSP solver to make the resampling function. Because of that, the Phantom Go game should be ready to be released in the upcoming release. In the few next days, I will get a code review according to which I will refactor the code to releasable form. As soon as the final touches are commited, I will let you know.

Ok cool. This means you need OR-Tools installed to use it?
I would love to have the resampling function conditional on the ORTOOLS build macro: if it is included, then it uses the fancy CSP solver. Otherwise, it is just empty and throws an error.
That way we can still include it in the list of games by default.

Hello @lanctot, I am sorry that I forgot to add one important information, which is that I will make another pull request for the resampling function, so the game will be released without it. The problem is pretty big and I am still working on it.

Hi @Syor, the CI tests failed. Can you look into it? I cannot import it until they pass. Thanks.

Hello @lanctot , I am, unfortunately, completely clueless on how to fix these errors, but I will try to fix them somehow. I am sorry if I am not able to make the checks pass until the release. If I will be unsuccessful, I will hopefully fix them with some help from @michalsustr , who helps me with this project.

No rush. will probably release tomorrow or in the next few dsys regardless, and we can do another minor release when this is merged.

From what I can tell from the output it looks like there are change markers (from git merges) still in is_mcts_test.cc. Are you able to find the CI tests output?

These are the two error messages.

I have commited few changes by accident, that also included the is_mcts_test.cc file, but I thought I have fixed it. Unfortunately I am not aware of anything that could cause a problem, since I reverted the file into the original form.

As I am looking at it, there are many changes in the file. I will make a commit with the original file, hopefully it will fix it.

Just to clarify the previous commit comment, the "residual" code is the Infostate resampling method, that is not finished, but I would like to keep it in the file, in case of possible reuse or completion-

Ok now you need to generate a playthrough: https://github.com/deepmind/open_spiel/blob/master/open_spiel/scripts/generate_new_playthrough.sh

We released this morning, btw, so no rush on this. We have another game internally in a similar state, so when this is merged we can do a 1.1.1 release for those two games.

Hello @lanctot, after the current push, I hope that all of the tests should pass. If all of the tests will be ok, I will probably make a final push after a code review in less than a week.

Great! Looks like it worked. Will take a look soon.

Just to clarify, by the code review, I meant a review by @michalsustr, who is a supervisor for my project based on Phantom Go, and makes a code review after every major push to help me maintain the code in publishable form.

No problem, we'll take a look after him. Either way, we'll have to look at it before it's merged, but it definitely saves us time to have others helping with that. Thanks @michalsustr !

Closing this pull request. I will create a new one, which will have a better push history.


python310Packages.furo: 2022.1.2 -> 2<IP_ADDRESS>

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for python310Packages.furo is: "A clean customizable documentation theme for Sphinx"
meta.homepage for python310Packages.furo is: "https://github.com/pradyunsg/furo"
meta.changelog for python310Packages.furo is: ""
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact

Checks done (click to expand)

built on NixOS

Warning: a test defined in passthru.tests did not pass

0 of 0 passed binary check by having a zero exit code.

0 of 0 passed binary check by having the new version present in output.

found 2<IP_ADDRESS> with grep in /nix/store/m9z29s7nls796fwlh6dinlassyxmmq9j-python3.10-furo-2<IP_ADDRESS>

found 2<IP_ADDRESS> in filename of file in /nix/store/m9z29s7nls796fwlh6dinlassyxmmq9j-python3.10-furo-2<IP_ADDRESS>

Rebuild report (if merged into master) (click to expand)

6 total rebuild path(s)

3 package rebuild(s)

3 x86_64-linux rebuild(s)
3 x86_64-darwin rebuild(s)

First fifty rebuilds by attrpath
kitty
python310Packages.furo
python39Packages.furo

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/m9z29s7nls796fwlh6dinlassyxmmq9j-python3.10-furo-2<IP_ADDRESS> \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A python310Packages.furo https://github.com/r-ryantm/nixpkgs/archive/de0c00190bdb99f702f8d14259d0dc5ffba34f6a.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/m9z29s7nls796fwlh6dinlassyxmmq9j-python3.10-furo-2<IP_ADDRESS>
ls -la /nix/store/m9z29s7nls796fwlh6dinlassyxmmq9j-python3.10-furo-2<IP_ADDRESS>/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

3 packages built:

kitty
python310Packages.furo
python39Packages.furo

Maintainer pings
cc @Luflosi for testing.

@ofborg eval


BUG unable to associate a document with a parent folder

expected behavior
this code should upload a document inside the specified folder
  const uploaded = await client.uploadEPUB(
    'testDoc',
    '9bf66920-a8ed-45ee-98cf-5ba52a860a0f', // file ID
    myFileBuffer,
    '71e70cd9-8d72-5c8b-addc-0d18c74364bc', // folder ID
  );

actual behavior

no parentID is sent to the Remarkable API
the document is uploaded at the root location of cloud storage

this functionality was broken in this commit https://github.com/Ogdentrod/reMarkable-typescript/commit/13c3ad226d58c2b9f8dad061d5af2011cb40ec74
created PR for proposed fix
https://github.com/Ogdentrod/reMarkable-typescript/pull/14/files

@Ogdentrod 👀

Oops! Thanks for the catch!
Merged and published on npm :)


Fix union type partial signature matching

Fixes #5401.

@sandersn This is the best way to fix the issue. We were a bit too permissive in partial signature matches involving rest parameters.

:+1: with the comment that I like Daniel's name change idea.


Thirdweb useContract return undefined

Hi, I am using useContract function for a tutorial project. The return result was not what i expected. Could anyone help me with this

Thank you

i found the solution:


Renaming symbols is broken in v6.0.0

When I try to rename a symbol sometimes it works and sometimes the new name goes into the wrong place. This only started to happen after the update to v6.0.0
(a simple code example):
//
// Perform the actual command
//
let goGetUrl (url: string) =
let query = goGetUrlAsync url
query.Wait()
query.Result
let fromCSV csv =
CsvFile.Parse(csv, hasHeaders = false, ignoreErrors = true)
If I try to renamed the fromCSV function to something like fromCSVBad I get this:
//
// Perform the actual command
//
let goGetUrl (url: string) =
let query = goGetUrlAsync url
query.Wait()
fromCSVBadesult
let fromCSV csv =
CsvFile.Parse(csv, hasHeaders = false, ignoreErrors = true)
Notice that the line was inserted on the last line of the previous functions (it looks like query was replaced as well) and that the symbol that I wanted to rename is unchanged.

Yeah, I've seen this off and on for a few versions now - we may have an off by one hiding around.

I'm having this issue with 6.0.6 (vscode 1.68.1, win x64, tried reinstalling ionide, tried restarting vscode&PC)


could not load library "libsqlcipher.so"

com.microsoft.office.sfb.sfbdemo E/dalvikvm: dlopen("/data/app-lib/com.microsoft.office.sfb.sfbdemo-2/libacomo.so") failed: Cannot load library: soinfo_link_image(linker.cpp:1636): could not load library "libsqlcipher.so" needed by "libacomo.so"; caused by load_library(linker.cpp:746): library "libsqlcipher.so" not found

Hello everyone final i have resolve this issue.
Here is the problem in Application class -
private static boolean LoadNativeLibrary() {
try {
if(!sLibComoIsLoaded.get()) {
System.loadLibrary("gnustl_shared");
System.loadLibrary("roottools");
System.loadLibrary("RtmMediaManagerDyn");
System.loadLibrary("acomo");
sLibComoIsLoaded.set(true);
}
} catch (UnsatisfiedLinkError var2) {
String strException = "Failed to load native library";
throw new RuntimeException(strException);
}
    return sLibComoIsLoaded.get();
}

There is missing
System.loadLibrary("sqlcipher");
please add this line before
System.loadLibrary("sqlcipher");
        this.application = Application.getInstance(this.getApplication().getApplicationContext());

@techlectdinesh I don't think you need to do that. Which version of the sdk are you using?

Hi @rahulyaa
i am using SkypeForBusinessAppSDKAndroid-<IP_ADDRESS>

@techlectdinesh Is this resolved? If yes, what was the issue?


Pop up window option to preview slide show animation

I'm not sure if we really need this feature, since right now we just have one animated transition.

Closing as this will not be supported in the mid-term


Fix cors issue and disable proxy when under https

Resolving issue with browser identification using window instead of process
Disabling proxy when browser is under https

Something strange happened when I published the newest version<EMAIL_ADDRESS>with this change. Before the module is published, it goes through the build process and it's generating different dist files for me... even if I'm currently on the HEAD of master.
Any idea? The impression I get is it's building the previous version:

Quick question @filipedeschamps , at this diff that you sent can you please check the one that actually changes the typeof proccess === 'undefined' to isBrowser()? 🤔
Because if it's updated maybe there's a cached dependency at @ivmarcos computer (at a higher or lower version) and maybe that's affecting how rollup is bundling and precompiling our resources. 🤔
(Still pretty confusing though)

Sorry guys, I checked it here and seems that what @lucianopf  mentioned is true, I was using yarn and after reinstalling the node_modules using npm the build files are with the same green lines as posted above by @filipedeschamps. Anyway, I tested here 3.3.6 and it's nice, no cors errors! Thanks for the support :smiley:!


Update Student tests to include the session data related changes

We've reached 70%+ coverage on our code. Will set this as low priority.

Covered by #346.


Is it possible for a protobuf plugin to add a field to a message?

I'm working on a protobuf plugin called Gorums, and I am looking for a way to add an ID field to any message type. Currently, the option that Gorums exposes requires the user to manually add a field to the messages in order to use the feature. Here's an example:
service GorumsTest {
	rpc ReadOrdered(stream ReadRequest) returns (stream ReadResponse) {
		option (gorums.qc_strict_ordering) = "MsgID";
	}
}

message State {
	string Data = 1;
	uint64 Timestamp = 2;
}

message ReadRequest {
	string Key = 1;
	uint64 MsgID = 2;
}"

message ReadResponse {
	bool OK = 1;
	State State = 2;
	uint64 MsgID = 3;
}

Gorums will use the MsgID field to keep track of which requests and responses belong together.
What I want is to have the MsgID field injected into the messages by Gorums so that the user does not have to specify it in their messages. I have looked into protoreflect, but can't find any info on whether it can be used to accomplish this. I tried calling msg.ProtoReflect().NewField(), but that only caused a panic.

There is no way for a plugin to inject a field into a message.
Protobufs are a language-neutral data interchange format. The shape of a protobuf message (that is, the fields that it is composed of) is defined in a .proto file so that it can be shared between every client of that file. Modifying the definition of a message outside the .proto file would run counter to the goal of protobufs being a neutral interchange format.
You might want to look into protobuf extension fields (proto2 only), which essentially allow one .proto file to add fields to a message defined in a different .proto file. However, for your use case, it might be simpler to wrap the user-provided messages in a message type of your own:
message Request {
  uint64 ID = 1;
  google.protobuf.Any message = 2; // user-provided message
}

I see. Thanks for the suggestion, I will definitely look into that!


[Sprint: 41] XD-2594 Update spring-data-hadoop version to 2.1.0.RC1

adding hadoop26 as the default distro

removing hadoop24

changing hdp21 to hdp22

updating cdh5 to 5.3.0

Did ran some tests and started single node and doodled around. Looks good to me.

Rebased

added spring-yarn-boot-cli back to dependencies.properties. somehow it got removed due to rebasing. anyway tested, ran singlenode and xd on yarn.
merged.


⚠ ClamAV has degraded performance

In c776765, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 178 ms

Resolved: ClamAV performance has improved in 115ce5b.


[LIB] Migrate emma-lib under the emma project.

Maintaining emma-lib under a different project was a bad idea.
This will increase build times a little, but will allow to make simultaneous releases of emma-language and emma-lib and catch problems early.
In addition, reworked the kMeans example and fixed some bugs in the dscfInv transformation.

The 0.2.1 release can be tested with the following Maven profile:

    <profile>
      <id>emma-staging</id>
      <activation><activeByDefault>true</activeByDefault></activation>
      <repositories>
        <repository>
          <id>emma-staging</id>
          <url>https://oss.sonatype.org/content/repositories/orgemmalanguage-1023</url>
          <releases><enabled>true</enabled></releases>
          <snapshots><enabled>false</enabled></snapshots>
        </repository>
      </repositories>
    </profile>


[8.x] Allow including a closure in a queued batch

This PR allows including a closure in a queued batch:
Bus::batch([
    new ProcessPodcast,
    function(){
        // ...
    },
    new ReleasePodcast
])->dispatch();

Is this inspired by that Twitter thread? If so, would this solve that use case since if the function starts a chain that function itself would complete even while the chain is still running? So the batch might look completed before it actually is?

@taylorotwell yes it’s inspired by the Twitter thread. I think that’s what the OP meant. To start the chain as part of the batch.

I've played around with this for a bit as well and stumbled across the issue that Taylor mentioned. I'm pretty sure that doing it like this would incorrectly calculate the number of jobs in the batch.
Bus::batch([
    new Job1(),
    function () {
        Bus::chain([
            new Job2(),
            new Job3(),
        ]);
    }
]);

This would incorrectly be calculated as having 2 jobs, Job1 and the closure, when in fact it should be 3 jobs.

The bigger issue is that the jobs inside the chain wouldn't be aware of the batch (which might not be an issue but it feels off that they are treated differently). I'll try and polish up my proof of concept later.

I opened up a discussion about it where I also try and explain my use case for this: https://github.com/laravel/framework/discussions/34332

@ksassnowski The batch won't know about the jobs in the chain yes. This PR just allows you to dispatch a closure based job inside a batch. Inside that job, you may dispatch a chain if you want, that's one of the use cases.

Yeah, I think this PR is fine but doesn't really solve @ksassnowski's problem which I don't think is possible given the current code in the framework. The only way to make it happen is to create a batched job which dispatches the chain onto the sync queue so that all the chain jobs run sequentially and synchronously within that batch job's process. Does that solution work for you @ksassnowski?

Can you have a look my draft PR to see if this is something worth including in the framework? https://github.com/laravel/framework/pull/34337


Nuki Opener not pairing: CRC Error

Hi,
i have problems with pairing my Nuki Oper with several esp32 boards now, so I can say it´s not a hardware thing.
I tried the debug version from issue https://github.com/technyon/nuki_hub/issues/39
Could You look into thins?
Thats what the log states:
[163800][D][NukiBle.cpp:229] onResult(): Found nuki in pairing state: Nuki_Opener_XXXXXX addr: 54:d2:72:XX:XX
Nuki opener start pairing
lld_pdu_get_tx_flush_nb HCI packet count mismatch (1, 2)
[164242][W][NukiBle.cpp:138] connectBle(): BLE Connect failed, retrying
[164424][D][NukiBle.cpp:1105] onConnect(): BLE connected
[165336][D][NukiBle.cpp:604] pairStateMachine(): ##################### REQUEST REMOTE PUBLIC KEY #########################
[165463][E][NukiUtils.cpp:91] crcValid(): CRC CHECK FAILED!
[165493][E][NukiUtils.cpp:91] crcValid(): CRC CHECK FAILED!
lld_pdu_get_tx_flush_nb HCI packet count mismatch (0, 1)
[194244][D][NukiBle.cpp:1111] onDisconnect(): BLE disconnected
[195408][W][NukiBle.cpp:749] pairStateMachine(): Pairing timeout
[195458][D][NukiBle.cpp:107] pairNuki(): pairing result 2
Nuki opener start pairing
[204126][D][NukiBle.cpp:1105] onConnect(): BLE connected
[205098][D][NukiBle.cpp:604] pairStateMachine(): ##################### REQUEST REMOTE PUBLIC KEY #########################
[205121][E][NukiBle.cpp:1033] handleReturnMessage(): Error: 10 for command: 00:01
Guru Meditation Error: Core  0 panic'ed (Double exception).

Please flash version 8.22 and post the content of your the system information page.

The debug build from the other thread is quite old, and we've made fixes to the pairing code in the meantime.
Can you flash this binary and get new logs?
nuki_hub-8.23-pre-2-dbg.zip

Ok flashed it and that are the logs now: Still refering to CRC Errors
[ 39386][D][NukiBle.cpp:254] onResult(): Found nuki in pairing state: Nuki_Opener_1B15B3A2 addr: NUKI_BLE
Nuki opener start pairing
[ 39514][E][NukiBle.cpp:635] retrieveCredentials(): Getting data from NVS issue
[ 39514][D][NukiBle.cpp:85] pairNuki(): Nuki in pairing mode found
[ 39518][D][NukiBle.cpp:132] connectBle(): connecting within: nuki
[ 39523][D][NukiBle.cpp:139] connectBle(): connection attemnpt 0
[ 39608][D][NukiBle.cpp:1173] onConnect(): BLE connected
[ 40062][D][NukiBle.cpp:927] registerOnGdioChar(): GDIO characteristic registered
[ 40407][D][NukiBle.cpp:962] registerOnUsdioChar(): USDIO characteristic registered
[ 40514][D][NukiBle.cpp:674] pairStateMachine(): ##################### REQUEST REMOTE PUBLIC KEY #########################
Sending plain message: 01 00 03 00 27 a7
[ 40524][D][NukiBle.cpp:901] sendPlainMessage(): Command identifier: 01, CRC: a727
[ 40662][D][NukiBle.cpp:987] notifyCallback():  Notify callback for characteristic: a92ae101-5501-11e4-916c-0800200c9a66 of length: 20
Received data: 03 00 17 ca 81 ff bc d1 6e d8 6b 6b f1 e6 e1 96 e2 d7 f6 b6
[ 40675][E][NukiUtils.cpp:99] crcValid(): CRC CHECK FAILED!
[ 40707][D][NukiBle.cpp:987] notifyCallback():  Notify callback for characteristic: a92ae101-5501-11e4-916c-0800200c9a66 of length: 16
Received data: 51 22 a6 75 72 97 3c b7 af 3f b6 8c 5f 6b 33 e0
[ 40721][E][NukiUtils.cpp:99] crcValid(): CRC CHECK FAILED!
[ 70556][W][NukiBle.cpp:814] pairStateMachine(): Pairing timeout
[ 70606][D][NukiBle.cpp:112] pairNuki(): pairing result 2
[ 72179][D][NukiBle.cpp:1179] onDisconnect(): BLE disconnected
[ 73086][D][NukiBle.cpp:254] onResult(): Found nuki in pairing state: Nuki_Opener_1B15B3A2 addr: NUKI BLE

ok, thanks we'll have a look. Is your opener firmware up-to-date?

That might be the catch, I´m on an old firmware, since the support of my intercom got worse on nuki side and even dropped.
I'm on 1.3.1

Why do they remove support for intercoms?
The latest version is 1.10.1, but you can't try since downgrading isn't possible. I'm not aware of changes to the BLE API though. There was a security update, but that wasn't a breaking change as far as I know.

Ok, I got a second (white) Nuki Opener for testing now...
What I can tell is that the delivered firmware was 1.5.3 and the pairing mode was not noticed at all,
after updating to the current version 1.10.1 the paring mode was recognized and the pairing process went through.
I prosume something is different with the paring process after some releases.

That's good to know, I'll add it the readme. Maybe you could use another ESP to talk to your intercom, and wire the ESP to your opener. That way you could update and still use your unsupported intercom.

So I can confirm now: The error takes only place with the older firmwares. With 1.3.1 on my old Opener I had the CRC errors, right after the firmware update to 1.10.1 the error disappeared and the pairing went through with now error at all.
I guess this issue can be closed, I just checked the readme where you added the hint to the newest firmware already.


Ratchet + PThreads

I am trying to get Ratchet to create a new thread when the onOpen event fires but the behavior is not what I expected.
namespace PhpRcon;
use Ratchet\MessageComponentInterface;
use Ratchet\ConnectionInterface;

class Chat implements MessageComponentInterface
{
    public function onOpen(ConnectionInterface $conn)
    {
        echo 'Creating Thread' . PHP_EOL;
        $thread = new MyThread($conn);
        echo 'Thread Created' . PHP_EOL;
    }
    public function onMessage(ConnectionInterface $from, $msg) {}
    public function onClose(ConnectionInterface $conn) {}
    public function onError(ConnectionInterface $conn, \Exception $e) {}
}

class MyThread extends \Thread
{
    public function __construct()
    {
        $this->start();
    }
    public function run()
    {
        echo 'Starting long process...' . PHP_EOL;
        sleep(3);
        echo 'Finished long process...' . PHP_EOL;
    }
}

The output is as follows:
Creating Thread
Starting long process...
Thread Created
Finished long process...

Which is correct but it takes until the thread has finished for the browsers web socket to be fully established. I would expect this to happen as soon as the thread is created and not once the thread has finished.

You can already do this with React: https://github.com/reactphp/react/blob/master/examples/child-process.php

https://github.com/andreybolonin/RatchetMultiInstance

Here are some non-blocking code with Ratchet:
<?php
/*
 * To change this license header, choose License Headers in Project Properties.
 * To change this template file, choose Tools | Templates
 * and open the template in the editor.
 */
namespace App;
use Ratchet\MessageComponentInterface;
use Ratchet\ConnectionInterface;
/**
 * Description of Socket
 *
 * @author<EMAIL_ADDRESS> */
class Socket implements MessageComponentInterface{

    /**
     *
     * @var $clients array of each client
     */
    protected $clients;
    protected $loop;

    /**
     * constructor
     */
    public function __construct(\React\EventLoop\LoopInterface $loop) {
        $this->loop = $loop;
        $this->clients = new \SplObjectStorage;
    }

    /**
     *
     * @param ConnectionInterface $conn attaches clients
     */
    public function onOpen(ConnectionInterface $conn) {
        echo "\nNew Client Connected...\n";
        $this->clients->attach($conn);
    }

    /**
     *
     * @param ConnectionInterface $from
     * @param $msg
     */
    public function onMessage(ConnectionInterface $from, $msg) {

        echo "\nNew Message: " . $msg . "\n";
        foreach($this->clients as $clientSocket){
	    $this->loop->futureTick(function() use ($clientSocket){
		$clientSocket->send("\ndata data start\n");
	    });
	    $this->loop->futureTick(function() use ($clientSocket){
		$clientSocket->send("\ndata data two\n");
	    });
	    $stream = new \React\Stream\ReadableResourceStream(fopen(storage_path('logs/laravel.log'), 'r'), $this->loop);
  	    $loop = $this->loop;
            $stream->on('data', function($data) use ($clientSocket, $loop){
		$loop->futureTick(function() use ($clientSocket, $data){
			$returnValue = file_get_contents(storage_path('logs/laravel.log')) .
					file_get_contents(storage_path('logs/laravel.log')) .
					file_get_contents(storage_path('logs/laravel.log')) .
					file_get_contents(storage_path('logs/laravel.log')) .
					file_get_contents(storage_path('logs/laravel.log')) .
					file_get_contents(storage_path('logs/laravel.log'));
			$clientSocket->send($returnValue);
			$clientSocket->send($data);
		});
            });
            $stream->on('end', function() use ($clientSocket, $loop){
                $loop->futureTick(function() use ($clientSocket){
			$clientSocket->send('finished');
		});
            });
            $this->loop->run();
        }
    }

    /**
     *
     * @param ConnectionInterface $conn
     */
    public function onClose(ConnectionInterface $conn) {
        echo "\nConnection Closed \n";
        $this->clients->detach($conn);
    }

    /**
     *
     * @param ConnectionInterface $conn
     * @param \Exception $e
     */
    public function onError(ConnectionInterface $conn, \Exception $e) {
        echo "\nError on Connection: {$e->getMessage()} on line {$e->getLine()} in file {$e->getFile()} \n";
        $conn->close();
    }
}

@mostafaahamidmanon - This is not really a non-blocking solution as file_get_contents is a blocking call.
This question has more to do with either having threads or multiple processes running with Ratchet.
I am closing this question due to inactivity.

I was trying to tell that this long running task:
$returnValue = file_get_contents(storage_path('logs/laravel.log')) .
					file_get_contents(storage_path('logs/laravel.log')) .
					file_get_contents(storage_path('logs/laravel.log')) .
					file_get_contents(storage_path('logs/laravel.log')) .
					file_get_contents(storage_path('logs/laravel.log')) .
					file_get_contents(storage_path('logs/laravel.log'));

Will not block other messages to reach to the user and notice that these messages will be delivered first event while the previous task is still running in the background using futureTick()
		$clientSocket->send("\ndata data start\n");
	    });
	    $this->loop->futureTick(function() use ($clientSocket){
		$clientSocket->send("\ndata data two\n");
	    }); ```

But if I made all in one futureTick(), it will block all the messages until it is finished fetching the data from the log files.


Add tests

Swift PM supports test targets, I skipped looking at that so I could get something working.

Sorry for the delay on this -- I'm actually having a lot of trouble getting things to run from the test target 😅

As of #22 tests are now running in linux / mac

There just aren't any useful ones :D

Hey guys, I would like to contribute, but just to confirm, are the tests you're looking for right now mostly about whether your Codable model objects handle JSON encoding/decoding properly? Please tell me if you think I should start somewhere else first.

...are the tests you're looking for right now mostly about whether your Codable model objects handle JSON encoding/decoding properly?

Hey @yhkaplan, anything to increase test coverage is fine but I think that would be a great place to start! I remember when I started doing this I got caught up trying to find the test resources because I was trying to run the tests via Xcode (which results in a different cwd) and not using swift test. Let me know if you have any questions 👍

You might be able to use Swift’s #file macro (or whatever it’s called) to get a more consistent directory

I'm happy with the tests we've got so far 👍
Going to close this


add a dirty flag

it would be nice to know if the code wasn't changed since last successful build

Can you elaborate a little bit about what you have in mind?

at any point in time i would like to know if the generated javascript in sync with the current typescript of a project, or put it differently: was the source code changed since the last build
i need it to avoid running a build on source code that  wasn't changed

check the time stamps of the output and compare it to the input.

will do


Integrate Yolo to initiate the SAMURAI

I would love to use Yolo to initiate the SAMURAI.  I tried, but got errors.
SAMURAI mode: False
or
I would get
File "C:\Users\Documents\samurai\sam2\sam2\modeling\sam2_base.py", line 665, in _prepare_memory_conditioned_features
iou_score = output_dict["non_cond_frame_outputs"][i]["best_iou_score"]  # Get mask affinity score
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
KeyError: 17

can you provide more details on your modifications?

thanks for the suggestion, i will give this method a try.


Update Tigray data

Generate new CSV without skipping missing values.

First crop-mask PR woot woot! 🎉
I deleted the branch here, I usually do this when a PR is merged.


⚠ ClamAV has degraded performance

In 1c259a5, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 240 ms

Resolved: ClamAV performance has improved in dbf9915 after 20 minutes.


No Documentation on Pushing a signed image using Classic Pipeline

I am following this documentation on pushing a signed image to an ACR repository from Azure Pipelines - https://github.com/MicrosoftDocs/azure-devops-docs/issues?utf8=✓&q="c10c998c-e2cb-6946-3513-4efcad549fed"&in=body.
However, this only describes changes in YAML. We are not pushing images as part of our build. We are generating a tar file in build, and are pushing the images as part of the release pipeline. We are using a classic editor for that and are pushing the image using a script -
docker load --input "$(System.DefaultWorkingDirectory)/$RELEASE_PRIMARYARTIFACTSOURCEALIAS/drop/outputs/package/images/$BUILD_REPOSITORY_NAME.tar.gz"

docker tag $IMAGE_ID $REPOSITORY_NAME:$BUILD_TAG

docker images

az acr login --name <acr-name> --subscription <subscription-name>

docker tag $IMAGE_ID $(registry)/$REPOSITORY_NAME:$BUILD_TAG

docker push $(registry)/$REPOSITORY_NAME:$BUILD_TAG

Can this be modified to push signed images to ACR or can it only be done from a build pipeline?

@Kruti-Joshi -- Kruti, thank you for your thorough feedback and question.  Please consider these resources:

Azure DevOps Services
Azure DevOps on Stack Overflow

@steved0x -- Steve, please look into this issue.

I made some modifications to sign the image as part of the release pipeline.
I used the secure files in pipelines to download the private key file which was generated in my local and used the below script -
echo '---------Create Private Delegate Key for signing--------'
mkdir -p ./docker/trust/private
echo 'Created Trust Directory'
echo 'Copying $(privateKey.secureFilePath) to ./docker/trust/private'
cp $(privateKey.secureFilePath) ./docker/trust/private

But just copying the file, as per documentation does not work.
I'm getting the below error on running
echo $(SigningPassphrase) | docker push --disable-content-trust=false $(registry)/$REPOSITORY_NAME:$BUILD_TAG
Error:
no valid signing keys for delegation roles
I added the following lines in the script to load the private key -
chmod 600 ./docker/trust/private/$(KeyFileName)
echo '-----Loading Key-----'
docker trust key load ./docker/trust/private/$(KeyFileName)

But this gives me the error
docker: 'trust' is not a docker command.
How do I run the docker trust command to load the private key for signing?
Is there a special package that needs to be installed?

UPDATE
Once I updated docker cli to 20.10.13, docker trust command started working. However, signing of the image is still failing, even after loading the key -

Can you help with this signature failure - no valid signing keys for delegation roles?

@Kruti-Joshi -- Kruti, thank you for your followup.  This repo is solely for documentation-related issues, and this seems implementation-related.  Please see the links in my initial comment.

Thank you @WilliamAntonRohm. I've posted the question on stackoverflow. I hope the steps can also be documented for classic pipelines.

@WilliamAntonRohm , @steved0x , can we mention what the value of $(DOCKER_CONFIG) is in the script -
mkdir -p $(DOCKER_CONFIG)/trust/private
cp $(privateKey.secureFilePath) $(DOCKER_CONFIG)/trust/private
It says the value is retrieved from Docker login. But for people trying to do this via a script, they'll need to know where to place the key file. By default it gets created in home/vsts/.docker.
Should it be inside the workspace? Or in home? Where should .docker folder be created?


Simple ENS Integration

What is this feature about? (1 sentence)
As a user I want to enter ENS names so that I don't have to copy-paste cryptic addresses
Why is it needed? What is the value? For whom do we build it?
ENS is picking up steam and we want to push the standard.
High-level overview of the feature
For this version we keep the integration simple as we have experienced significant UX complications with a deep ENS integration on the mobile apps.

No reverse lookup
Don't display ENS name in transaction list
All address input fields can detect valid ENS names and immediately replace them with their corresponding Ethereum address.

Delay checks by some time (e.g. 500ms) and only check if user pauses


Docker entrypoint broken in v1.0.0-beta.20

The instructions here don't work for 1.0.0-beta.20:
Expected behavior
$ docker run gcr.io/kpt-dev/kpt:v1.0.0-beta.20 version
1.0.0-beta.20

Actual behavior
$ docker run gcr.io/kpt-dev/kpt:v1.0.0-beta.20 version
exec /usr/local/bin/kpt: no such file or directory

It seems to work as expected on 1.0.0-beta.19:
$ docker run gcr.io/kpt-dev/kpt:v1.0.0-beta.19 version
1.0.0-beta.19

@droot As discussed in standup, this is the same error message I got here https://github.com/GoogleContainerTools/kpt/issues/3528#issuecomment-1234631750

@natasha41575 @droot This seems to be fixed in 1.0.0-beta.21
$ docker run gcr.io/kpt-dev/kpt:v1.0.0-beta.21 version
1.0.0-beta.21

Thanks for verifying it @tomasaschan


Issue with switch from strftime() to date()

https://github.com/middlewares/access-log/blob/205cfd7d47c3523c60624815244851fa0a97ed1f/src/AccessLog.php#L182
It seems like the default value for getRequestTime() isn't compatible with the changes. Default format should use date() format rather than strftime()
example:
Before -> [17/May/2022:10:51:10 -0600]
After change -> [%17/%b/%2022:%10:%May:%th %136]

Fixed in v2.1.2. thanks!


Edits for clarity and formatting

Some small edits to use more direct language. The original wording "pod with the lower value will come first" is unclear, so I changed this to "is deleted first" but I'm not 100% sure that's correct.
I added backticks to "using the matching operator <operator>.*" because it didn't look like it was rendering correctly on the page.

Hello @mjgutermuth , gentle reminder on the above PR. Would you be able to incorporate the above suggestions?

Hello @mjgutermuth . This pull request has been open for some time. Are you actively working on the changes?

Hi @mjgutermuth .
I am closing this pull request. When you have time, you are welcome to reopen the PR and continue to work on the changes.
/close


Grammatical updates

Simple grammatical updates to documentation and comments.
Also adds necessary require statements to successfully seed newsletters and snippets.

Thank you 💜


Verifiable random function

Added the VRF for NodeLabel computation, as well as verification.
This PR includes:

A new dependency: the Rust vrf crate.
Updates to the function signatures for various proof structs.
New functions for VRF label computations, proofs and verifications.

NOTE: At the moment, this code includes the VRF secret key from the VRF library and corresponding public key generation. We would want to change this later.
Addresses #34 and #119.

@Jasleen1 do you think we should land this as-is or should we add the VRF crypto trait first? Personally I think we probably shouldn't embed the VRF key material in code directly in the /main branch. I'd like to have this changed prior to merging

As a note, this PR adds full support for VRFs across the various components of the AKD structure (client, lean client, etc). However there is another issue open (#144) which is still pending, hence the commenting out of 2 portions of tests for the moment.


Decompress context properties

Hi @PeterLindgren-se ,
I've been using this repo to import suspended messages into a custom SQL database for reporting purposes.
Very helpful :).
Now, I'm also looking to import context properties of those messages. So I have changed the SQL script slightly to include the imgPropbag column from the Parts table.
I also changed the biztalk-get-suspended-messages.ps1 script to write the context properties to disk.
When decompressing the context properties from disk, biztalk-decompress-message.exe throws an exception:
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Runtime.InteropServices.COMException: Catastrophic failure (Exception from H
RESULT: 0x8000FFFF (E_UNEXPECTED))
   at Microsoft.BizTalk.Message.Interop.CompressionStreams.DecompressSequentialStream(Stream rawStream, Stream& compressedStream, UInt32& bytesRead, UInt32& bytesWritten)
   at Microsoft.BizTalk.Message.Interop.CompressionStreams.Decompress(Stream stm)
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.RuntimeType.InvokeMember(String name, BindingFlags bindingFlags, Binder binder, Object target, Object[] providedArgs, ParameterModifier[] modifiers, CultureInfo culture, String[]
 namedParams)
   at MyClass.DecompressFile(String inputFilename, String outputFilename, Type compressionStreamsType)
   at MyClass.Main(String[] args)

So apparently, the decompression code can't be used to decompress the context properties. Maybe the serialization format is different.
Do you happen to know how I can get the contents of the context properties?
Thank you!

Hi @ngruson, thanks for your feedback!
The Parts.imgPropBag contains some other proprty bag, I'm not sure exactly which. The message context properties are stored in the spool.imgContext column. It is not compressed, it is the serialized IBaseMessageContext inmsg.Context. I haven't looked deeper into it than that, I found some old code that might point you in the right direction.
To extract the data, add these lines to the SQL and PS files, respectively:
	   ,DATALENGTH(s.imgContext) as [DatalengthContext]
	   ,s.imgContext

  # Write imgContext
  $fileName = "$baseFileName$fragmentNamePart.context"
  $bytesToWrite = $_.DatalengthContext
  Write-Output "Writing imgContext $bytesToWrite bytes to $fileName"
  [System.IO.File]::WriteAllBytes("$folderName\$fileName", $_.imgContext[0..($bytesToWrite-1)])

Links for inspiration (looks like you might need .Net code and not Powershell):
https://biztalkmessages.wordpress.com/2009/05/13/unleashing-the-spool-table-well-at-least-partly/
https://maximelabelle.wordpress.com/2009/12/10/serializing-message-context-and-part-properties/
Please let me know if and how you managed to deserialize the context properties. Once you've managed to deserialize, it's up to you how you want to display and use the properties.

@ngruson I've updated the code to download and save the message context. I've also added sample code that read a message context file and dump the content in human-readable form on the console, you should be good to go from there to adapt the code to suit your needs.

Thanks for including this 👍 .


Log frontiers in progress tracking

This PR adds logging of frontiers to the reachability tracker. This is in support of hydration progress tracking in Materialize, which would like to know the per-operator frontiers to know which ones have finished processing their snapshot inputs.
Look at the commit messages for more details. The actual logging change is in the second commit, commit 1 and 2 are supporting changes that try to limit the performance impact of the additional logging.
The performance impact of this change should be negligible (or even negative!) when reachability logging is disabled and small when reachability logging is enabled. I'm including microbenchmark results in a comment below.

Benchmark Results
To test the performance impact of this change, I ran event_driven.rs with 1000 dataflows of a depth of 1000 each for 100k rounds, and measured the time to completion.
The test was performed in two configurations:

with reachability logging disabled
with reachability logging enabled and a logging closure that simply increments a counter on each event

Below are the results of 10 runs each, for both of the configurations against the master branch and each of the three commits in this PR:

From the results we can see that in this benchmark the additional logging has no negative effect on runtime if logging is disabled, and has a small negative effect when logging is enabled. The latter is expected.
The former is maybe expected, though there is some weirdness here in that the results show that on master disabling logging actually slows Timely down! I'm not sure why this is. This result occurs consistently when the benchmark is repeated at different times, so it doesn't seem to be caused by noise in the system. Maybe some edge case related to CPU caching?

I wired operator frontier logging up in Mz to verify that this is useful for tracking hydration progress, and it seems to be!
With a simple cross join dataflow:
CREATE TABLE t (a int);
INSERT INTO t SELECT generate_series(1, 10000);

CREATE VIEW v AS SELECT t1.a + t2.a FROM t t1, t t2 WHERE t1.a < 1000000 AND t2.a > 0;
CREATE DEFAULT INDEX ON v;

We get this frontier output:
materialize=> SELECT id, min(time) as time
FROM mz_internal.mz_dataflow_operator_frontiers
JOIN mz_internal.mz_dataflow_addresses USING (address)
JOIN mz_internal.mz_dataflow_operators USING (id)
WHERE
  time IS NOT NULL AND
  list_length(address) = 3
GROUP BY id
ORDER BY time DESC, id ASC;
  id  |     time
------+---------------
 8587 |<PHONE_NUMBER>660
 8592 |<PHONE_NUMBER>660
 8594 |<PHONE_NUMBER>660
 -- ...
 8636 |<PHONE_NUMBER>660
 8658 |<PHONE_NUMBER>161
 8659 |<PHONE_NUMBER>161
 8661 |<PHONE_NUMBER>161
 -- ...
 8674 |<PHONE_NUMBER>161
(31 rows)

... which implicates operator 8658 as the one that's stuck. And sure enough, looking at the dataflow graph, this operator is the cross join.


Ethereum transaction signing issues

In the current state of the Trezor.Net library, there are some very strange results when attempting to sign an Ethereum transaction.
This is the code I am using to try and sign a transaction:
[TestMethod]
public async Task SignEthereumTransaction()
{
    await GetAndInitialize();

    //Note: these are not reasonable values. They should not be used for a transaction. Looking for a better example here...
    var txMessage = new EthereumSignTx
    {
        Nonce = 0.ToHexBytes(),
        GasPrice =<PHONE_NUMBER>.ToHexBytes(),
        GasLimit = 21000.ToHexBytes(),
        To = "689c56aef474df92d44a1b70850f808488f9769c".ToHexBytes(),
        Value =<PHONE_NUMBER>00000.ToHexBytes(),
        AddressNs = ManagerHelpers.GetAddressPath(false, 0, false, 0, 60),
        ChainId = 4
    };

    var transaction = await TrezorManager.SendMessageAsync<EthereumTxRequest, EthereumSignTx>(txMessage);

    Assert.AreEqual(transaction.SignatureR.Length, 32);
    Assert.AreEqual(transaction.SignatureS.Length, 32);
}

The transaction signing goes wrong in a few different areas.
Firstly, when attempting to sign a transaction on the Rinkeby network through https://mycrypto.com, we can see the output on our device as something related to the following:

Send 1.0 tEth to 0x689c56aef474df92d44a1b70850f808488f9769c?

After confirming this, we get the next message:

Really send 1.0 tEth paying up to 0.000021 tEth for gas?

When attempting to sign the same transaction through Trezor.Net, we get the something similar to the following:

Send message to 0x689c56aef474df92d44a1b70850f808488f97600

After confirming this, we'll get this next message:

Really send message paying up to 0.000021 UNKN for gas?

The first bit of strange behavior is with the fact that even though we are sending an EthereumSignTx, the Trezor is interpreting it as signing a message through EthereumSignMessage. The text 'Send Message' should only really appear if we are signing a message through EthereumSignMessage.
The second bit of strange behavior is seen in the actual address. The address where the message is being sent to the device in its entirety it seems. Sometimes only the last few characters of the address will be '0', sometimes it will be many more.
The last bit of strange behavior is that the Trezor doesn't seem to recognize the ChainCode that we have entered, and therefore states we are trying pay the transaction fee with currency UNKN.
I'm not really sure where the issue lies in all of this, but clearly something is up.

@ThatSlyGuy have you looked at #9 ?
I believe that 9# has been fixed.

This stuff does seem strange. It seems to be doing the same thing with the latest code. It is is sending the EthereumSignTx message which is 58:

It's possible that there are still bugs in the reading/writing to the device.

@ThatSlyGuy @MelbourneDeveloper
Hi I have not managed to sign a transaction successfully yet, banging my head on combinations. Well I do sign stuff, but  it is incorrect the response in comparison with Nethereum, so some of the parameters are not passed correctly.
Signing messages work, not transactions :) I moved to messages to have a simple test case to start with.
This is message signing using Nethereum
 var message = new EthereumSignMessage
                        {
                            AddressNs = KeyPath.Parse("m/44'/60'/0'/0/0").Indexes,
                            Message = Encoding.UTF8.GetBytes("Hello").ToHex().ToHexBytes()
                        };

                        var messageSignature = await trezorManager.SendMessageAsync<EthereumMessageSignature, EthereumSignMessage>(message);

                        //Same as first address
                        Console.WriteLine(ToHex(messageSignature.Address));

                        var signer = new Nethereum.Signer.EthereumMessageSigner();
                        var messageSignature2 = signer.EncodeUTF8AndSign("Hello", new EthECKey(
                            "0x2e14c29aaecd1b7c681154d41f50c4bb8b6e4299a431960ed9e860e39cae6d29"));

And this is where I am with transaction signing, GasLimit is recognised and address correctly, from the response, but it does not recognise the value.
  var txMessage = new EthereumSignTx
                        {
                            Nonce = 10.ToBytesForRLPEncoding().ToHex().ToHexBytes(),
                            GasPrice = 10.ToBytesForRLPEncoding().ToHex().ToHexBytes(),
                            GasLimit = 21000.ToBytesForRLPEncoding().ToHex().ToHexBytes(),
                            To = "689c56aef474df92d44a1b70850f808488f9769c".ToHexBytes(),
                            Value = BigInteger.Parse("10000000000000000000").ToBytesForRLPEncoding().ToHex().ToHexBytes(),
                            AddressNs = KeyPath.Parse("m/44'/60'/0'/0/0").Indexes,
                            //ChainId = 1
                        };
                        var transaction = await trezorManager.SendMessageAsync<EthereumTxRequest, EthereumSignTx>(txMessage);

@juanfranblanco have you debugged the helper methods for ToHex and ToHexBytes etc? I'm not 100% sure that they're correct. If so...
Are you up for doing a test with the Trezor python library ? That would determine if the problem is in the library or in the parameters being passed. If the input is the same and the output is different, it probably means there is a bug in my library and I can take it from there.

@MelbourneDeveloper yes that was the next step.. the ToHex() is Nethereum the ToHexBytes() is yours but only for strings, on a documentation I read the need to remove the 0x and use hex strings.. hence GasLimit is recognised. I have to stop this for a while as it was lots of trying and error. But yeah ToHexBytes for a string seem to work correctly, based on the message, GasLimit and the Address.

Although it might be something with the Transaction Type, that was my other thought.

@juanfranblanco @ThatSlyGuy the problem is pretty clear. On a basic level, the To address is completely wrong. I'm getting the same result with KeepKey It could be that the address is being sent to the device incorrectly, or it could be that the code is reading it incorrectly from the device.
There's enough information here for me to investigate.

I don't have a problem with the To address (check my code) but it appears to sign message, so it might the transaction type. If you try really hard you will see the address here "689c56aef474df92d44a1b70850f808488f9769c"

also check the Gwei here

Here is the full program, it is using the "old" TrezorManager https://github.com/juanfranblanco/Trezor.Net/blob/master/src/Trezor.Net.Sample/Program.cs

Ok further analysis https://github.com/trezor/trezor-mcu/blob/master/firmware/ethereum.c#L267-L268 the Value is send as 0 hence we have "message". The txn type is for WanChain so ignore that.

@juanfranblanco thanks! Actually, I might have found the problem. I will get back to you soon.

@juanfranblanco
Can you try a pull from branch Issue10? I believe this has fixed @ThatSlyGuy 's problem. It may fix yours.

Sure

@juanfranblanco I have added the unit test SignEthereumTransaction2 to master/Issues10 which is from your sample app. You can get it by doing a pull. Am I right in understanding that the problem is the Gas limit is coming up as 210000, and the currency is saying UNK?

What do you expect this test to display on the Trezor?

Correct that is where we are now, although your fix has allowed to see the Value

It should be 21000 and 10 Ether, let me recreate in MEW and post some shots.

Unfortunately, I have to call it a night. But, I can see that there is an issue. I think I have some ideas about what is happening. The Issue10 branch is one step closer. I think I'll be able to fix it tomorrow.

@ThatSlyGuy  can you see if your issues are any closer with the Issue10 branch?

Have a great sleep, yes that branch fixed reading the Value so that is a great start, I don't get the extra zero of the GasLimit.

Ok re-reading the description it is the total Wei Price to it is 21k * 10 hence the 210000, so that is correct. I have checked the signatures and everything is fine.
Also the simplified test case is here, just use Nethereum extensions for this.

                        var txMessage = new EthereumSignTx
                        {
                            Nonce = 10.ToBytesForRLPEncoding(),
                            GasPrice = 10.ToBytesForRLPEncoding(),
                            GasLimit = 21000.ToBytesForRLPEncoding(),
                            To = "689c56aef474df92d44a1b70850f808488f9769c".HexToByteArray(),
                            Value = BigInteger.Parse("10000000000000000000").ToBytesForRLPEncoding(),
                            AddressNs = KeyPath.Parse("m/44'/60'/0'/0/0").Indexes,
                        };

                        var trezorTransactionSignature = await trezorManager.SendMessageAsync<EthereumTxRequest, EthereumSignTx>(txMessage);

                        var transactionSigner = new TransactionSigner();
                        var nethereumSignature = transactionSigner.SignTransaction("0x2e14c29aaecd1b7c681154d41f50c4bb8b6e4299a431960ed9e860e39cae6d29",
                         "0x689c56aef474df92d44a1b70850f808488f9769c", BigInteger.Parse("10000000000000000000"), 10, 10, 21000);
                        var transactionRecovered = new Transaction(nethereumSignature.HexToByteArray());

                        Console.WriteLine("Trezor R: " + trezorTransactionSignature.SignatureR.ToHex());
                        Console.WriteLine("Nethereum R: " + transactionRecovered.Signature.R.ToHex());
                        Console.WriteLine("Trezor S:" + trezorTransactionSignature.SignatureS.ToHex());
                        Console.WriteLine("Nethereum S: " + transactionRecovered.Signature.S.ToHex());
                        Console.WriteLine("Trezor V:" + trezorTransactionSignature.SignatureV);
                        Console.WriteLine("Nethereum V: " + new BigInteger(transactionRecovered.Signature.V));

@MelbourneDeveloper  So the only issue is the currency UNK, the rest is correct and signing correctly.  Next thing is signing token transactions or big data transactions.

More tidying on the message signing side:
 var message = new EthereumSignMessage
                        {
                            AddressNs = KeyPath.Parse("m/44'/60'/0'/0/0").Indexes,
                            Message = Encoding.UTF8.GetBytes("Hello")
                        };

Ill do some pull requests later

@MelbourneDeveloper This is great! Your changes made in the issue10 branch seemed to have completely fixed everything, thanks a lot! Everything seems to be correctly and consistently displayed to the Trezor. I haven't checked if the v/r/s return values are correct yet, but I will get around to that next.
@juanfranblanco The UNKN thing is just a result of not including the ChainId in the EthereumSignTx initialization. If the ChainId is set to 1, then the Trezor displays ETH as the currency. If I set the ChainId to 4 (rinkeby), then the Trezor displays tETH. When no ChainId is included, it is not serialized, which makes the currency type appear as UNKN.
        [TestMethod]
        public async Task SignEthereumTransaction2()
        {
            await GetAndInitialize();

            var txMessage = new EthereumSignTx
            {
                Nonce = 10.ToBytesForRLPEncoding().ToHex().ToHexBytes(),
                GasPrice =<PHONE_NUMBER>.ToBytesForRLPEncoding().ToHex().ToHexBytes(),
                GasLimit = 21000.ToBytesForRLPEncoding().ToHex().ToHexBytes(),
                To = "689c56aef474df92d44a1b70850f808488f9769c".ToHexBytes(),
                Value = BigInteger.Parse("10000000000000000000").ToBytesForRLPEncoding().ToHex().ToHexBytes(),
                AddressNs = KeyPath.Parse("m/44'/60'/0'/0/0").Indexes,
                ChainId = 1
            };
            var transaction = await TrezorManager.SendMessageAsync<EthereumTxRequest, EthereumSignTx>(txMessage);

            Assert.AreEqual(transaction.SignatureR.Length, 32);
            Assert.AreEqual(transaction.SignatureS.Length, 32);
        }

Cool the ChainId makes different types of Ether.
On a similar fashion there is a registry of token addresses, if the address match the amount of tokens will be transferred.

Hopefully there are no more Java arrays ;)

Great!
The whole area of code needs to be rewritten. I only wrote it in one weekend. It can also be performance optimized. It was a literal translation from Java and I took shortcuts because I didn't understand Java too well. When I have time, I will clean this up and make it more readable.

I have merged the fix in to master and will do a NuGet release.
@ThatSlyGuy @juanfranblanco so transactions are all good now? I can close these issues?

@MelbourneDeveloper I can definitely do that. I'm going to close this issue now that everything seems good. I'll reopen it or create a new one if any other problems arise.

All ok on my side, see also the final tests for Message Signing, Ether Transaction and Token Transaction. I need to do further checks on 155 signatures and big data elements. But I don't think it will be an issue. Next step to integrate in Nethereum like ledger :) :)

@juanfranblanco @ThatSlyGuy incidentally, transaction signing now works on KeepKey (KeepKey.Net). It's basically all the same code.


Multiples tags

Hi,
I have multiple data-editable attributes and every time I click on the edit button (pencil) there is a tons of auto generated p tag before and after each data-editable elements. It's normal ?
Thanks.

Hi @jonathandion - your going to have to provide a bit more information I'm afraid, can you provide a copy of the HTML before the edit button is clicked and after?

@anthonyjb sure

asas

@jonathandion is this before or after?

@anthonyjb after the click

OK can you send the HTML before and your code to initialize the editor?

@anthonyjb

test

@anthonyjb  thanks for the help ! I have another question how to know if the editor is ready ? There is no callback like this :
// do some stuff here
})```

and what is the best way to show the the toolbox directly on load ? Right now I'm using the JQuery trigger() function.

Thanks !

Thanks - looks like it may not be coping with the empty  tags I'll take a look and see if I can reproduce.

@jonathandion ok so re. the issue you're reporting I can't reproduce this I'm afraid - I've copied the before HTML you sent me into the sandbox HTML example provided in the library it works as expected and shows the correct number of paragraph tags so my best guess is that somehow something is getting duplicated as part of your integration code. I'm happy to keep looking and to concede it maybe a bug with the library but at this point I need more also, ideally I need you to reproduce the error using the sandbox example and send me the code so I can reproduce the error for myself and track down the problem code.
Re. your question regarding callbacks the editor is set to ready once the init method is called and when stopped (via the stop method), I'm happy to consider adding a init event for the editor so that you can bind to it using the addEventListener (there is already a stop event) but would like to understand what you're are trying to achieve a little better before I recommend a solution.

@anthonyjb thanks allot for your time.
The reason I need a callback it's because i'm trying to target some elements from the plugin. I'm actually targeting my elements after the init function but the length is 0. I'm using a setTimeout or setInterval to check if 'editor.isReady()' and then I can manipulate the DOM. Like I said, I trigger a 'click' on the edit button to show the toolbox without clicking on it. I would like to know if there is a better way to do this.
Thanks.

@jonathandion instead of triggering a click against the ignition switch you could call start against the editor and manually change the state of the switch, for example:
editor.start();
editor.ignition().state('editing');

Would that help?

Thanks @anthonyjb it's working like the trigger but I still need to wrap it in a setTimeout because sometimes it triggered sometime not. No a big issue.

Hi @jonathandion thanks for your input, can you describe when it triggers and when it doesn't? I'd like to identify if there's something that can be done to improve things here.


minor fix for dart porting

I've ported your nice element to Dart (see here) but I had to apply this patch to make it work.
Could you merge it ?

Hi there, and thanks for PR!
I think I need to modify it a bit, would it be OK for you if an empty array would be set to iron-list when items is null

I imagine this would be even better. My solution was not working well with Firefox. But I'll check it as soon as you had patched it.

Should I close the PR or do you still need it?

I can use this PR as a base. Could you try if branch fix/null-items works for you?
bower install iron-data-table#fix/null-items
Thanks!

lgtm
It works like a charm, tnx.


ARROW-5019: [C#] ArrowStreamWriter doesn't work on a non-seekable stream

Allow ArrowStreamWriter to write to a non-seekable stream, like a network stream.
@chutchinson @stephentoub @pgovind

@chutchinson @stephentoub - any feedback here?

@wesm @kou - I believe all feedback has been addressed and this is ready to be merged.


feat: Added GH CLI support for 'Trigger Workflows for Branch' script

Description

Added GH CLI support for 'Trigger Workflows for Branch' script
Done in preparation for a workflow that could centrally trigger all module pipelines on a schedule (and use the GH CLI)

Using this script will be very simple for this use case, effectively only requiring the schedule and invoking the script with it's default parameters.

Also added a feature that by default excludes disabled workflows from the trigger (as those would fail the trigger event)
Validated locally..This is no platform script (yet)
Small fixes like the regex filter

Type of Change

[x] Update to CI Environment or utilities (Non-module affecting changes)
[ ] Azure Verified Module updates:

[ ] Bugfix containing backwards-compatible bug fixes, and I have NOT bumped the MAJOR or MINOR version in version.json:

[ ] Someone has opened a bug report issue, and I have included "Closes #{bug_report_issue_number}" in the PR description.
[ ] The bug was found by the module author, and no one has opened an issue to report it yet.

[ ] Feature update backwards compatible feature updates, and I have bumped the MINOR version in version.json.
[ ] Breaking changes and I have bumped the MAJOR version in version.json.
[ ] Update to documentation

Anybody?

@ChrisSidebotham approved this originally but seems your push of a commit wiped this?


What is the difference between MMBench test, MMBench dev, MMBench test (CN), MMBench dev (CN), CCBench dev?

Thanks for your benchmark and leaderboard, I have a few questions to ask that could not be located in the documentation.

Is dev the training dataset? Then why does CCBench not have test?
The evaluated models in dev and test are not the same.
What is CCBench, and how does differ from MMBench (CN)?

@tonysy @gaotongxiao @YuanLiuuuuuu @kennymckormick

dev means for development usage, with the groundtruth answers released. For the test set, the groundtruth answers are confidential.
That's due to some legacy issues. For some models, the model author just provide the dev or test inference result to use. Thus
the evaluated models in dev and test are not the same (but that can be special cases).
MMBench (CN) is the Chinese translation of MMBench, while CCBench is another set of testing questions focusing on Chinese culture.

Thanks for your replies!
If that is the case, why CCBench does not have test?

CCBench is just a small-scale benchmark for developing purpose, it's not designed to be an official benchmark with test split.


router ux updates

The router target is ambiguous when adding. Maybe add (app) or similar to the view to help distinguish.
target version should use @ not colon

Looks much nicer! Fixed in master


Use another GPIO (not GPIO3)

Hello,
I'm testing this project. I have a problem.
Your example works ok for me.
In this moment I need to use the GPIO3, for the touchscreen.
So, Is it possible to use another gpio, not the gpio3 in this case?
Thanks!

You can use any GPIO available to shutdown the raspberry. Simply update listen-for-shutdown.py to change GPIO number.
Anyway, only GPIO#3 on pin 5 will allow you to wake on your raspberry PI as described in Pi power button guide.

@Nizhile is correct -- pin 5 (GPIO3) is necessary for waking the Pi. If you don't care about this functionality and only want a shutdown button, you can use a different pin. But if you want it to power on too, that's the only pin you can use unfortunately.


Random 404 Error

Decided to work on my app today and got this...
I can still fetch the data fine which confuses me even more

Can you please provide information about that issue you had? Did you solve that?


Include Rakefiles in the project root directory in the indexing target

Related issue: #1
It is Probably not possible to include it with ruby-lsp v0.22.1.

Related implementation: https://github.com/Shopify/ruby-lsp/blob/v0.22.1/lib/ruby_indexer/lib/ruby_indexer/configuration.rb#L119


kFirebaseInstallationsErrorDomain missing NS_SWIFT_NAME

https://github.com/firebase/firebase-ios-sdk/blob/bdac2b87f6d6e604deae914a970a75136b3a62b5/FirebaseInstallations/Source/Library/Public/FirebaseInstallations/FIRInstallationsErrors.h#L19
This is missing the NS_SWIFT_NAME, unfortunately fixing it is a breaking change. It should be named InstallationsErrorDomain.

Merged #9395 into v9


chore(deps): update mantine and typescript

Proposed Changes

Potential Breaking Changes

Acceptance Criteria

[ ] The proposed changes are covered by unit tests
[ ] The potential breaking changes are clearly identified
[ ] README.md is adjusted to reflect the proposed changes (if relevant)

The modalWizard test don't fail anymore? 👀 :hype:

The modalWizard test don't fail anymore? 👀 :hype:

Nope, the ModalWizard component has been fixed


”Putineiu” is listed in the drop-down menu

Description:
"Putineiu" is listed in the drop-down menu alongside its corresponding county and commune, according to the law.
Precondition:
The website is up an running.

Step 1
Type "Putineiu" in the search bar.

Expected results
The location is listed in the drop-down menu as "Comuna Putineiu, GIURGIU".


_auto_example.php wrong

Unless I'm mistaken, it seems that the _auto_example does not do what it is supposed to do. Here is what it does as far as I understand:

for each domain in $domains, it checks if it finds a separate certificate (in $certlocation/$d/cert.pem)

if yes, it checks the expiry date of that certificate, if less than 30 days, $needsgen is set to true
if no, $needsgen is set to true

when $needsgen is true, $le->signDomains($domains) is called

The problem is that $le->signDomains($domains) creates only one certificate (using the first domain in $domains) with all remaining domains as SAN. So the above check will always set $needsgen = true since there are no individual certificates for all the domains.
Am I missing something?

Looks like a bug to me too. I believe it should be checking if there is a certificate for the first domain in the array, and then checking if the certificate contains every other domain in the array as a SAN. However, the example does work for single domain registrations.


[feature request] Macaron support?

Would you consider adding Macaron support?

@aryzing
Thanks for taking your time to create this issue. Right now I don't have the capacity to maintain another CSS framework plus the adoption rate (~150 dl/week) does not look very promising.


BlueFors: make driver compatible with old and new log files

This PR adds the functionality to read old or new log file versions. The code comes from here: https://github.com/pyplotter/pyplotter/commit/eae409e98156bec1a7255acf19050d0af6d0486d Thanks @edumur
Please have a look @astafan8 and @ptar543

Codecov Report

Merging #239 (b24f0d8) into main (51db4fd) will decrease coverage by 0.01%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##             main     #239      +/-   ##
==========================================
- Coverage   19.95%   19.94%   -0.01%
==========================================
  Files         139      139
  Lines       17406    17409       +3
==========================================
  Hits         3473     3473
- Misses      13933    13936       +3

Files Changed
Coverage Δ

...codes_contrib_drivers/drivers/BlueFors/BlueFors.py
0.00% <0.00%> (ø)

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Schould COUNT(C1, C2) work?

Situation
Even with FN_AGG_COUNT + FN_AGG_COUNT_TUPLE activated, we get the following error in the integration test:

Function COUNT should not be called with 2 arguments. Possible number of arguments: 0, 1

Acceptance Criteria

Clarify if COUNT(C1, C2) is supposed to work
If no: document under "known issues"
If yes: fix test and implementation or write ticket to fix VS core

Talked to compiler team. The point here is that tuples need to be set in an extra-parenthesis!
Users must issue something like this SELECT COUNT((C1, C2)) FROM VS.T and the push-down needs to be SELECT COUNT((T.C1, T.C2)) FROM VS.T.


Add JAX integration

This commit adds a ZenML integration for JAX, a Python ML research framework.
Describe changes
See above.
Pre-requisites
Please ensure you have done the following:

[x] I have read the CONTRIBUTING.md document.
[ ] If my change requires a change to docs, I have updated the documentation accordingly.
[ ] If I have added an integration, I have updated the integrations table.
[ ] I have added tests to cover my changes.

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] Other (add details above)

TODOs:

Add an example for using JAX together with one of its associated DL frameworks, possibly https://github.com/nicholasjng/jax-styletransfer.
Add a model materializer for loading weights from a file, e.g. HDF5 (also implemented in the style transfer example I linked above).
Figure out how to enable the installation of CUDA-ready jaxlib packages if requested. jaxlib is the associated computational backend of JAX, and comes in different wheel flavors depending on what type of installation you want (CPU-only, or with CUDA/ROCm support on Linux/Windows). Supporting all different options will require some work.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Thanks for the PR @nicholasjng!
I think this a good draft but needs some work. Your TODO list is actually quite intriguing and is more in line what a JAX integration could look like for us. Do you need any support in drafting the example and materializers?

Thanks for the PR @nicholasjng!
I think this a good draft but needs some work. Your TODO list is actually quite intriguing and is more in line what a JAX integration could look like for us. Do you need any support in drafting the example and materializers?

I think the materializer API is pretty well documented. I would probably go for something that saves and loads a model to/from disk, like I mentioned earlier.
I can reuse the code from my repo in the example (after some reorganization), for everything else, I will check the "Add your own example" guide.
What is your opinion on the installation point I mentioned? The process is described in the JAX readme, by the way, if you want to make up your mind independently of what I wrote there. Happy to hear your thoughts, going back to work :)

What is your opinion on the installation point I mentioned? The process is described in the JAX readme, by the way, if you want to make up your mind independently of what I wrote there. Happy to hear your thoughts, going back to work :)

Sounds to me like the JAX installation can exist outside of the ZenML integration. We can simply do jax[gpu] in our requirements and indicate in the documentation that we simply automate pip install jax[gpu] for you. There is no magic to what we do there anyway, simply do a pip install

Okay. I noticed that my style transfer example is most likely >400 LOC, which would make it more complex than other existing examples. Should I settle for something easier instead?
If I do that, it might still require some more boilerplate than usual due to the functional style of Jax. Do let me know what your preference is.

Would be happy to see a draft and go with what you like!

Apologies for the delay. I worked some more, decided to implement a custom model materializer for Haiku (that way, plugging in different jax NN libraries can also be supported by implementing a materializer each).
Still left TODO:

Reorganize functions in the example (model load -> then augmentation)
Finish the materializer implementation (especially the return)

Really sorry for the turnaround in this review. Somehow the last message escaped my inbox.
I think its hard to review the current state as the zenml repo has sharply diverged from the point of development. Also, please note that we use develop as our trunk branch, meaning your work should branch off of develop and the PR should therefore also be against develop, not main.
Would it be possible to perhaps rebase and resolve conflicts from this point? ill make sure to review faster once thats done. Apologies for the delay!!

Hey, apologies for the delay. I had some difficulties at work, so this fell off the edge.
In principle, I hope to continue with it next week latest (this week is almost fully occupied by my work in the bakery). I still need to implement the HDF5 writing part of the materializer. It should, obviously, reproduce the names of the layer groups and the layers themselves, so I will first need to check how the guy who made the repo saved the pretrained weights.
Otherwise, I don't have an environment ready for testing the pipeline (M1 user...), any chance I can scrape by and test this locally without the infamous ml-metadata dependency for now? (Otherwise I really need to install all the x86-emulated stuff like Python, brew, etc. side-by-side, and that's something that I would really like to avoid).

@nicholasjng No worries, thanks for contributing! Unfortunately there is no way of running ZenML without ml-metadata (for now). I guess you could test the read/write logic of the model outside of ZenML to make sure that works. Once that is ready, feel free to ping me here and I'll make sure either myself or someone else runs the pipeline

@nicholasjng No worries, thanks for contributing! Unfortunately there is no way of running ZenML without ml-metadata (for now). I guess you could test the read/write logic of the model outside of ZenML to make sure that works. Once that is ready, feel free to ping me here and I'll make sure either myself or someone else runs the pipeline

Thanks! (Also thank you for cleaning up the branch).
As a quick point of discussion (perhaps also relevant for @htahir1):
In principle, it might be that the materializer contract of reading and writing the same type is violated in this example. Let me explain:
The arrays from the HDF5 file read into memory are directly used to instantiate the style transfer neural network (so the return type of handle_input would be something like Dict[str, Dict[str, jnp.ndarray]]. However, due to JAX's functional design, this stateful initialization needs to be expressed in a functional way, resulting in the hacky pattern of using an init method inside the training script.
This method returns a params object, which holds the weights in a similar structure as the object resulting from the HDF5 file (an hk.FlatMapping), but which is strictly speaking not the same type (for JAX's intents and purposes, it is, however, as both objects are valid PyTrees of the same structure).
So the input to handle_return would be a hk.Params object (a FlatMapping, if you will). Due to the above, it behaves the same for what our intents are. Is it fine to just silently return the params in this type then, or should I explicitly cast to a bare Python dict? (I think there might be a Haiku API for that - I'll check that before implementing.)

I added the return materializer implementation, at least the way I believe it is supposed to work. Will revisit the whole thing in the coming evenings, and test it on my local Linux machine in a pipeline towards the weekend.

Wanna give this a go @schustmi ? I think the implementation stands as it is right now, the failures are related to linting and spelling

Hi @nicholasjng, sorry for the late reply. Pretty busy at the moment but I hope to give this a try till Friday!
Regarding the config.yaml: Yes that would be awesome, as we use these to run integration tests for all examples

Closing due to inactivity. Please feel free to open a new PR if this is still relevant! Thank you in any case for the contribution!


Allow reading bundles

In order to make it easier to run plans generated by opa build, we should allow running Jarl with a pointer to a bundle rather than a plan file, and have the plan(s) extracted from the bundle.

Resolved in #117


does not generate JSON schema, but simply JSON

Minor issue, but I believe the name of this repo is misleading, as it does not generate a JSON schema (http://json-schema.org/) but simply proprietary JSON string

100% agree with this. "JSON Schema" is a specific format which isn't handled by this module.
Ideally this repo would be renamed to something less likely to cause confusion. At the very least, the documentation should note:
"This module generates a JSON representation of your MySQL schema. It does not generate JSON Schema."


Brainstorming concept : DataTonic, help and find the most optimized LLM model for an user usecase

Problem :
Today there is a large choice of custom and sometime not accessible or complex, time consuming, would need to pay a subscription to *some bullshit AI solution over priced", would need to pay fair subscription price, or to pay expensive a freelance or agency to do it.
Concept :
Datatonic uses autogen, truegen, its own prompt engenering, its agent...
is able to evaluate and do testing in order to find the proper LLM model for given scenario
take into account multimodel (sound, image, text) using Gemini and Truegen.
Benefits for end users/companies :

Find cost efficient "bundle" for anyone with or without prior knowledge
Transparent with personalization and flexibility
always up to date, so always useful (not a temporary concept that will loss relevancy)
Give a clear documentation for part that Datatonic can't do for security reason
No lockup user is free to own its bundles prebuilt or just documented

Benefit for us :

Will probably always have usefulness, gave us additional data/insight through time
Business model with multiple income (pay as you go, agency service) (?)
Probably more

Limitation:
Seem complex, too irrealistic ?
Scenario example
User  :
↳  Add a credential

Pay as you go
Bring your own key

↳ Explain what he want to accomplish

eg: I love monkeys, and i want a model able to ingest  books in PDF about monkeys that can contain image about monkey, or youtube video that can also able to get sounds from it , or photo.

System (still need more information so will follow ask follow up until system has all he needs :
↳ Budget per token max

Dollar giving tokens number _(maybe system could make an a priori guess estimate of the number of tokens that would be consume  for a month/week ?)

↳ Currently using solutions to add as integration :

Integration CTA
Postgres credential

↳ Specific demands for it like for eg :

"Must use Cohere"
"OSS LLM"...
"Intensive and often ingestion or From time to time ?"
"Chat history ? Conversation ? Persistant memory ? "

↳  When user has done and system has all infos he needs open draggrable list with each component from users

Budget
Must use Cohere
OSS LLM
...

↳ Enterprise extra security needed ?
↳ ...
↳ ...
↳  User can re-order the list based on the importance
↳ Or other kind of measure like  a typical five-level Likert item (trivial , not important, important, very important)
↳ Datatonic through its datasets (can be some benchmark of Transformers, embedding/chunking...), Existing evaluations based on the proper metric, on its own evaluation already made and often updated between each langage models, multimodal models (would be more complex (?))
↳ Provide few bundles possibilities (without to have to create each (possible?)
↳ User choose one of these, or through chat input ask for ajustment
Final step 🕺🏿↳ When user has chosen its bundle, Datatonic starts the work and user wil be notified when it's done
### Final step seem too much and seem to add too much complexity and seems a bit a non sense ? when we could provide detailed documentations instead... and give choice to make us built it for him/company  as an agency with a support and on boarding

just one point first : the user intention seems like it's not an objective and more of a technique, which is not exactly what i would expect in the user section. this is too complex to put in work right now , let's do an MVP with a single configuration , that sometimes works at least, then it will be easier to provide alternatives. in fact we can start evaluating alternatives in the context of the TruEra evaluation .

Data Driven Advisory (Use Case)
Phase 1: Engagement Setup (1-2 weeks)
Client Background Information: Company history, mission, vision, and strategic objectives.
Industry Data: Market size, trends, competitors, and regulatory environment.
Stakeholder Information: Key stakeholders, organizational structure, and decision-makers.

Phase 2: Data Gathering and Analysis (3-6 weeks)
Operational Data: Sales figures, production data, supply chain details, employee information.
Financial Data: Profit and loss statements, balance sheets, cash flow statements, budgets.
Customer Data: Customer demographics, satisfaction surveys, purchase history.
Internal Documents: Previous strategy documents, reports, internal analyses.

Phase 3: In-Depth Analysis and Hypothesis Testing (4-8 weeks)
Segmented Data: More detailed operational and financial data broken down by business unit, geography, product line, etc.
Competitive Intelligence: Detailed competitor analysis, market share, business models.
Benchmarking Data: Industry benchmarks, best practices, case studies.
Qualitative Data: Interviews, focus groups, expert opinions.

Phase 4: Solution Development and Validation (2-4 weeks)
Scenario Analysis Data: For testing different strategic options and their potential outcomes.
Risk Assessment Data: Data related to potential risks and mitigation strategies.
Feedback Data: Initial feedback on proposed solutions from a small group of stakeholders or pilot tests.

Phase 5: Final Recommendations and Implementation Planning (2-3 weeks)
Consolidated Analysis: Summarized data and analysis that support the final recommendations.
Stakeholder Feedback: Comprehensive feedback on proposed recommendations.
Implementation Data: Resources required for implementation, timelines, and milestones.

Phase 6: Implementation Support and Closure (Variable)
Performance Data: Metrics and KPIs to track the implementation progress.
Adjustment Data: Ongoing data collection for adjusting strategies as needed.
Final Outcome Data: Data reflecting the impact of the implemented solutions.

Post-Engagement (Optional)
Long-term Impact Data: Data collected over time to assess the long-term impact of the engagement.
Follow-up Feedback: Stakeholder feedback on the effectiveness and outcomes of the project.

Statement Of Work:
(example of a single fixed output useable by autogen)
Overview: Brief description of the client's organization and the context of the engagement.
Purpose of the SoW: Clarification of the document's intent and its role as a guiding agreement.

Project Objectives and Scope
Objectives: Clear and specific goals the project aims to achieve.
Scope of Work: Detailed description of the services and tasks to be performed. This section delineates what is included and, just as importantly, what is not included in the engagement.

Project Approach and Methodology
Methodology: Explanation of the methodologies, frameworks, or strategies the consulting team will employ.
Phases of Work: Breakdown of the project into phases or milestones, each with specific tasks and objectives.

Deliverables
List of Deliverables: Detailed list of expected outputs, reports, presentations, tools, or models to be provided.
Quality Standards: Description of the standards or criteria against which the deliverables will be assessed.

Timeline
Project Timeline: Detailed timeline of the project, including start and end dates, phase durations, and key milestones.
Review Points: Scheduled points for reviewing progress and adjusting plans as necessary.

*6. Roles and Responsibilities
Consulting Team Composition: Names and roles of the consultants involved.
Client Responsibilities: Specific tasks or inputs required from the client, such as data provision, key personnel involvement, etc.**

Pricing and Payment Terms
Fee Structure: Details on how the consulting fees are structured - fixed fee, time and materials, etc.
Payment Schedule: Timeline and conditions for payments.

Confidentiality, Legal, and Ethical Considerations
Confidentiality Clauses: Terms ensuring the confidentiality of shared information.
Legal and Compliance Aspects: Adherence to relevant laws and industry regulations.
Ethical Standards: Commitment to maintaining high ethical standards during the engagement.

Terms and Conditions
Contractual Terms: General terms including contract duration, termination conditions, dispute resolution mechanisms, etc.
Amendment Process: Process for making changes to the SoW.

Signatures
Sign-off: Signatures from authorized representatives of both the consulting firm and the client.

ok since everyone is aligned, i made  a new issue, so we can try it like that now.


Move Applications workflow and related details page

Tried to run this locally and the host couldn't find the following modules:

shared/components/pipeline-run-logs/logs/log-snippet-types' or its corresponding type declarations.
shared/components/pipeline-run-logs/logs/pipelineRunLogSnippet' or its corresponding type declarations.
shared/components/pipeline-run-logs/logs/LogsWrapperComponent' or its corresponding type declarations.
logs/log-snippet-types' or its corresponding type declarations.
logs/logs-utils' or its corresponding type declarations.
logs/LogsWrapperComponent' or its corresponding type declarations.
logs/pipelineRunLogSnippet' or its corresponding type declarations.

Maybe you just missed to commit

Tried to run this locally and the host couldn't find the following modules:

shared/components/pipeline-run-logs/logs/log-snippet-types' or its corresponding type declarations.
shared/components/pipeline-run-logs/logs/pipelineRunLogSnippet' or its corresponding type declarations.
shared/components/pipeline-run-logs/logs/LogsWrapperComponent' or its corresponding type declarations.
logs/log-snippet-types' or its corresponding type declarations.
logs/logs-utils' or its corresponding type declarations.
logs/LogsWrapperComponent' or its corresponding type declarations.
logs/pipelineRunLogSnippet' or its corresponding type declarations.

Maybe you just missed to commit

Same for me.