WIP: Resilience

Type of change
Select the type of your PR

Bugfix
~Enhancement / new feature~
Refactoring

Description
This PR is a large-scale refactoring of the cluster controller with the aim of making it more resilient (fixing #264) and also simplifying it.
The basic idea is to get rid of any distinction between create and update operations.
AbstractOperations layer
In the AbstractOperations layer this takes the form of adding a reconcile() method, which takes a name and a desired resource state: If the resource does not exist it is created, if it does exist it is patched. If the desired resource state is null the resource will be deleted. reconcile() returns a resource-specific description of what it did, but currently this is only used for statefulsets.
The StatefulSetOperations is specialised for two cases (KafkaSetOperations and ZookeeperSetOperations). Partly this is because we want their reconcile() result to
include an indication of whether a rolling update is needed, but it is also anticipated to be necessary in the future when their rollingUpdate(), scaleUp() and scaleDown()operations become more sophisticated.
The class hierarchy is changed to make things a little more typesafe (e.g. only resources which support readiness are equipped with a readiness() method). This hierarchy is reflected in the tests too. For similar reasons, Pod and Endpoint operations are made package-private, because they're not actually needed outside their owning operators.
AbstractClusterOperations layer
In the AbstractClusterOperations layer we can now simplify things considerably. CompositeOperation no longer needs to be generic, or carry around a diff. (CompositeOperation and execute() could probably be simplified still further). We don't need a distinct create() operation.
Taking KafkaClusterOperations.updateKafka as an example:

We no longer use a diff to decide whether to reconcile() the services, configmaps etc (formerly we used the diff to decide whether to patch) because we make use of Fabric8's patch() semantics where it will generate an (empty) diff for a noop patch.
Things a more complicated with the statefulset, because we need to know whether a rolling update is needed, but this is the output of the reconcile() method, so again, there is not need for an up-front diff
We have to compose() the operations because there is now no distinction between create and update. Previously for create we created all the resources in parallel and join()ed on their futures.

Testing
I've put a lot of effort into writing the new KafkaClusterOperationsMockIT, which should make it easy to test all sorts of interesting scenarios.
To Do

[x] Fix the ugly hack I put in around waiting for pods to be ready. Very weird bug there that I'm currently working around.
[ ] Probably some trivial refactoring wrt class and method names.
[ ] Javadoc
[ ] System tests passing
[ ] More tests in KafkaClusterOperationsMockIT.
[x] There still a lot of code which can be removed (CompositeOperation and execute()).
[ ] Other code can probably be moved (e.g. the diffing we do now is different from the previous diffing, that still needs to be tidied up.

Checklist
Please go through this checklist and make sure all applicable tasks have been done

[x] Write tests
[ ] Make sure all tests pass
[ ] ~Update documentation~
[ ] ~Check RBAC rights for Kubernetes / OpenShift roles~
[ ] ~Try your changes from Pod inside your Kubernetes and OpenShift cluster, not just locally~
[ ] Reference relevant issue(s) and close them after merging

I didn't reviewed all the 2500 lines you changed ;-).

The interesting thing about that is this part:
$ git diff --shortstat origin/master -- cluster-controller/src/main/
 28 files changed, 972 insertions(+), 1360 deletions(-)

And I think there's a lot of potential to simplify further.

Now might also be a good time to rename classes if we can agree better names. Right now the naming is confusing, imho.
How about:

io.strimzi.controller.cluster.{operations→operator}.resource.{*Operations→*Operator}
io.strimzi.controller.cluster.{operations→operator}.{cluster→assembly}.{*ClusterOperations→*AssemblyOperator}: The "assembly" name is better than "cluster", I think, as it retains the "collection of things" meaning (and really these are about managing a bunch of resources) without having the connotation of being an actual cluster.
io.strimzi.controller.cluster.{resources→model}, since these are conceptual model.

@tombentley +1 to the renamings

@tombentley I know that @scholzj already approved it and said "merge it" but please wait for my review, this is really a big one and I'm just starting ... it will take "few" time to do that. Thanks !

@ppatierno don't worry I fully intend to wait for your review: You always have useful points to make and in any case the system tests are failing currently.

I guess we need to wait for Kafka Connect endpoints when we create that? Though it's not something we were doing before.

I know that @scholzj already approved it and said "merge it" but please wait for my review

@ppatierno Don't worry ... that was not meant completely seriously given the list of ToDo and the WIP in the title :-)

I guess we need to wait for Kafka Connect endpoints when we create that? Though it's not something we were doing before.

It is not stateful set. So maybe we do not have to wait for all Pods, but can just check the service endpoints?

@ppatierno are you happy with the renaming I proposed?

@tombentley I'm happy with new naming ! +1

This is now a lot closer to where we want it to be:

I have done the renaming mentioned previously, though some uses of the term "cluster" are still present (these can be fixed in later PRs).
We now have integration tests (which we can debug!) for the KafkaAssemblyOperator, and it should be relatively easy to add more tests for the Connect and Connect S2I operators.
I've changed the way we diff/patch (replacing Diffs, with StatefulSetDiff):

We use Json diffing to determine what's changed.
We explicitly ignore changes to SS scale (spec.replicas) when patching (i.e. internalPatch()), and do this instead via scaleTo(). We have integration tests for kafka scale and and scale down. We can add test for ZK scaling.
We likewise ignore changes the volume claims (logging a warning if they changed), but we allow changes to the deleteClaim. We have integration tests that delete claim is honoured for both kafka and zk.
I think there is still scope to simplify this. We don't need all of StatefulSetOperator, KafkaSetOperator, ZookeeperSetOperator and StatefulSetDiff.

With regard to @ppatierno's point about metrics enablement, I don't think this really changes anything. Previously we had logic that we would not do a rolling update if the metrics changed but were disabled. But when creating the SS the enablement was take from the presence/absence of metrics in the CM. So the only proper way to change whether metrics were enabled was to change the metrics in the CM triggering a rolling update anyway. You could change it "improperly", but messing with KAFKA_MTRICS_ENABLED env var in the template for the SS, but I don't think we expected people to actually do that, did we?

I think the main thing still to do here is fixing the CI.
@ppatierno @scholzj if you want to re-review please do.

@tombentley I had another review pass but I miss something (as usual :P) ... where is the way for checking difference in Kafka Connect deployment and then patching ? I see the internalPatch for KafkaSetOperator and ZookeeperSetOperator which use the StatefulSetDiff but not something for Kafka Connect which is based on a Deployment.

where is the way for checking difference in Kafka Connect deployment and then patching ?

Good question @ppatierno, looking at the old code I can see that we would:

scale down
patch service
patch deployment
scale up

When patching the deployment, although the diff() method was computing a value for whether a rolling update was needed, it seems this was never actually consumed.
Right now, in this branch, we do the same sequence as listed above. Since don't do anything special wrt patching vs scale up/down the patching of the deployment will actually perform the scale up (if scaling up is required), making #4 a noop (from my knowledge of how fabric8 and k8s work). We should address that at some point, but it doesn't have to be right now.

@tombentley @ppatierno For Deployments and DeploymentConfigs we rely on the build in rolling update. That is why we do not really need to use the Diff and do rolling update.

Which is fine @scholzj, but I don't really understand why it's necessary to do the scale down, patch and scale up separately, given that they all end up being PATCH verbs in the end anyway. Is the operator for Deployments really so dumb that it will end up updating pods that it will then delete? If if it is, we could combine the patch and scale up, it seems to me.

You added everywhere in fromConfigMap the null check and return null if the config map doesn't exist. But in the update* methods you do not check for null. Makes me wonder if that is something still missing.

I think what's missing is we should be passed the cluster CM from the AbstractAssemblyOperator.reconcileAssembly() down to the subclass's createOUpdate(): The only way createOUpdate() can be called is if there is a (non-null) cluster CM, so the null check should be unnecessary. But passing the non-null one we got avoids a possible race where it get deleted.

For the delete* and fromStatefuSet (fromDeployment etc.) methods, you seem to handle the null check completely in the delete* methods. But in the update* and fromConfigMap methods you do the null check inside fromConfigMap. Wouldn't it be more consistent to do the check in the same way for both operations? Strangely enough, for KafkaConnect there is a null check again inside the fromDeployment method. So maybe these are just some forgotten peieces which need to be completed?

I wrote tests for the Kafka case (but not Connect or ConnectS2I cases) for deletion when some of the resources don't exist. The Connect case that you highlight will NPE if the deployment doesn't exist, and consequently the Service won't get deleted. For Connect we don't even need to get the deployment in the AssemblyOperator: We just need to know the names of the resources to delete.
In reality we're doing delete wrong anyway, because getResources() returns the SS. So if the SS doesn't exist AbstractAssemblyOperator.reconcileAssembly() will never call delete.

In the model classes we have different methods for creating the model from the assembly, but they are always called differently (fromStatefuSet, fromDeployment etc.). Since we are now renaming lot of stuff ... what about renaming them to fromAssembly?

I guess.

The delete methods for KAfka Connect and Kafka Connect S2I seem to be missing the null tests which are in the delete* methods for KafkaAssemblyOperator

I suspect because I've not written tests for these yet.


Separate google-style-rspec gem

I would like to see this project follow the pattern established by the rubocop and rubocop-rspec gems and have a separate google-style-rspec gem. Projects that use the google-style gem but don't use Rspec shouldn't be required to install those dependencies, even as development dependencies.

@blowmage Specifically, for this project, you want this line removed, correct?

I agree with this issue.

Yes, I would remove that line. And I would move all files into a google-style directory and add an empty google-style-rspec directory. Basically, convert this into a monorepo.

Closing due to monorepo prejudice.


Adding license file and headers

@eito Please review, do you want the CONTRIBUTING file as well?

@alpascual looks good, but please add Contributing page and I'm going to need to get access to merge this
@BrentPierce ^^^

Contributing file has been added to the PR

@eito you should have write access as you are in the app group ... do you need elevated privs?


New non-RGI emoji supported by iOS 14.5

iOS 14.5 apparently supports some new non-RGI emoji (although they’re not included in the emoji picker): https://blog.emojipedia.org/ios-14-5-emoji-changelog/#evenmorecouples

Beyond Unicode's RGI (Recommended for General Interchange) list, Apple has also implemented the following:

A couple emoji can include 🧑 Person + 👩 Woman, or 🧑 Person + 👨 Man. In Unicode's list, a gender inclusive person emoji can only be paired with a second gender-inclusive person.
A couple emoji can include one Person, Woman or Man with a neutral yellow skin tone, shown in combination with a Person, Woman or Man that has a realistic skin tone. In Unicode's list, realistic skin tones are not shown alongside the default yellow option.
Any of the above combinations are permitted in any order. For example, this is a standard Unicode emoji: 👩❤👨 Couple with Heart: Woman, Man, but the opposite pairing is 👨❤👩 Couple With Heart: Man, Woman is also supported in this release, despite being non-standard.[3]
[…]
The same applies to the very-similar 💏 Kiss emoji which also has the same set of new options, both standard and non-standard.

Attempt at listing the new emoji: https://gist.github.com/mathiasbynens/5b1d07784b3299648bfc068210574815
I’m not sure how accurate this is though, as it’s based on processing the emoji grid on https://emojipedia.org/apple/ios-14.5/new/ which contains more emoji than Emojipedia claims have been added 🤔
https://twitter.com/mathias/status/1387010272263553026


Added functionality of choosing between Re/Im and Re in 2D complex plots

This PR suggests the following:

Two new buttons (Real/Imag and Real) added to have more control for complex 2D plots
It dynamically switches between Real/Imag, Real and Mag/Phase (selecting one button, disables the other buttons in this category)
It works consistent with other controls like PlotType, e.g., it is possible to change PlotType together with this new added buttons.

It is tested and working as it should.
The following can be worked later to have more improvements and functionalities related to this topic:

Imag button can be added with the functionality for showing only imaginary.
For now, only 2D plots can be used with these new buttons. 1D plots are still showing real and imaginary parts in one figure. This could be improved later.
The default behavior of 2D plot is Real/Imag, but the Real/Imag button is not activated by default. It is a minor thing, but it's nice to have.

I like the idea of adding the additional options.
This conflicts with some pending changes on the plotting system in the current form, though :)
I'm proposing to hold off on this and merge this functionality into an updated system (once we determine what exactly that updated system is). @jenshnielsen - thoughts?

@wpfff
That's right.
As I have heard, the refactoring is going to take some time and in the meantime we have a release (tomorrow if I am not wrong). So, we can have this functionality in this release.
@astafan8

The above commit "2ce47e2" addressed these:

Now, we can have complex options for 1D plots both single trace and multitrace
The complex options are deactivated when the data is not complex.
Some refactoring in 1D plots.

@jenshnielsen
The last commit fixes the logic for complex options, as discussed above. Now:

The empty state is removed.
If the data is complex: All complex options (Real/Imag, Real, Mag/Phase) are enabled, which Real/Imag is activated by default.
If the data is not complex: Only Real is enabled and activated.

@jenshnielsen
The last commit fixes the logic for complex options, as discussed above. Now:

The empty state is removed.
If the data is complex: All complex options (Real/Imag, Real, Mag/Phase) are enabled, which Real/Imag is activated by default.
If the data is not complex: Only Real is enabled and activated.

Looks good and works well. Left a few minor comments

Looks good and works well. Left a few minor comments


loosen priors on survival model

failing sensitivity test for fixed year effect

change b0 prior to dnorm(3, sd = 10) and bAnnual dnorm(0, sd = 10)

done


Compilation error with "MRB_ARY_EMBED_LEN_MAX" in "array.h"

Hello!
I'm sorry I'm not good at English.
I built it on an ARM CPU (32 bit) device using IAR's EWARM compiler.
Then the following error occurs.
` array.c Error [Pe070]: incomplete type is not allowed C: \ work \ tk-optsys \ appl \ mruby_porting \ mruby \ include \ mruby \ array.h 37 Error while running C / C ++ compiler `
The following #define line seems to be the problem.

#define MRB_ARY_EMBED_LEN_MAX ((mrb_int) (sizeof (void *) * 3 / sizeof (mrb_value)))
mrb_value embed [MRB_ARY_EMBED_LEN_MAX];

As it is 32bit CPU,
sizeof (void *) * 3 ==> 12
sizeof (mrb_value) ==> 16
So 12/16 ==> 0 and it seems to be an error.
For now, if you write this #define in a constant, compilation is OK.
What value do you expect for "MRB_ARY_EMBED_LEN_MAX" here?
How many values should I set?
Thank you.

In this environment, it is correct that MRB_ARY_EMBED_LEN_MAX is
zero. However zero size array is not C standard.
For now, can you compile via the following patch?
diff --git a/include/mruby/array.h b/include/mruby/array.h
index 2e6951c0..563d2da7 100644
--- a/include/mruby/array.h
+++ b/include/mruby/array.h
@@ -33,7 +33,6 @@ struct RArray {
       } aux;
       mrb_value *ptr;
     } heap;
-    mrb_value embed[MRB_ARY_EMBED_LEN_MAX];
   } as;
 };

@@ -46,7 +45,7 @@ struct RArray {
 #define ARY_UNSET_EMBED_FLAG(a) ((a)->flags &= ~(MRB_ARY_EMBED_MASK))
 #define ARY_EMBED_LEN(a) ((mrb_int)(((a)->flags & MRB_ARY_EMBED_MASK) - 1))
 #define ARY_SET_EMBED_LEN(a,len) ((a)->flags = ((a)->flags&~MRB_ARY_EMBED_MASK) | ((uint32_t)(len) + 1))
-#define ARY_EMBED_PTR(a) (&((a)->as.embed[0]))
+#define ARY_EMBED_PTR(a) ((mrb_value*)(&(a)->as))

 #define ARY_LEN(a) (ARY_EMBED_P(a)?ARY_EMBED_LEN(a):(a)->as.heap.len)
 #define ARY_PTR(a) (ARY_EMBED_P(a)?ARY_EMBED_PTR(a):(a)->as.heap.ptr)

Hello!
It is OK with the application of the patch you contacted.
The VM on the actual machine also worked.
Thank you very much.

Probably I made this issue resurrected by 4c00167. Sorry.

By the way, I recommend using MRB_WORD_BOXING instead of MRB_NO_BOXING (default), especially on 32bit architectures.

Thank you for your report. I haven't tried it on a newer version yet, but if you have anything else please let me know.

I addressed this issue in the mruby3 branch.

Array embedding is disabled on 32bit platforms with MRB_NO_BOXING
The default mrb_value representation is changed to MRB_WORD_BOXING

I addressed this issue in the mruby3 branch.

Array embedding is disabled on 32bit platforms with MRB_NO_BOXING

Even with MRB_32BIT and MRB_NO_BOXING, if sizeof(mrb_value) == 12, one element can be embedded, but it seems that it is no longer embedded by this change. Is this change intentional?

It's too hard to detect sizeof(mrb_value) < sizeof(void*)*3 in C preprocessor. So I will allow the case MRB_32BIT and MRB_NO_BOXING and MRB_USE_FLOAT32.


Fixed the default paths for conda on windows where the python.exe found was not the correct one

Changelog (should I commit it in the changelog file ?)

Virtual environments now have a bin_paths attribute instead of a single bin. command.run and which both have their path argument renamed paths and supporting a list of paths, to handle it.

Fixed the default paths for conda on windows where the python.exe found was not the correct one. Fixes #256

thanks!


[KED-1090] Refactor graph rendering from D3 to React

Description
Change the FlowChart rendering from one big method containing a bunch of D3 selections and transitions, to instead use React + React-Transition-Group. The motivation behind this is to make this component more maintainable and more testable.
Notice

[x] I acknowledge and agree that, by checking this box and clicking "Submit Pull Request":

I submit this contribution under the Apache 2.0 license and represent that I am entitled to do so on behalf of myself, my employer, or relevant third parties, as applicable.

I certify that (a) this contribution is my original creation and / or (b) to the extent it is not my original creation, I am authorised to submit this contribution on behalf of the original creator(s) or their licensees.

I certify that the use of this contribution as authorised by the Apache 2.0 license does not violate the intellectual property rights of anyone else.

Checklist

[x] Read the contributing guidelines
[x] Opened this PR as a 'Draft Pull Request' if it is work-in-progress
[ ] Updated the documentation to reflect the code changes
[ ] Added new entries to the RELEASE.md file
[ ] Added tests to cover my changes
[x] Assigned myself to the PR
[x] Added Type label to the PR

I am, with deep regret, closing this PR. I liked how the code turned out, but between React-Transition-Group and the repeated setState calls, it significantly degraded animation performance, to the extent that the benefits of the improved code structure were not worth the tradeoffs. Instead, I've moved the drawChart logic to its own file (#63).


please excuse my ignorance - How can I add a bash script to the image and have it run at first boot?

Trying to figure out how to add a bash script to the image and make it run after all the rest of the processes are done.  Any help or examples greatly appreciated.
Alan Bell

There are several ways to do that.  Can I ask a few questions to better help you?
When you say "add a bash script to the IMG", presumably the script already exists on your host computer or network somewhere, is that correct?
"make it run after all the rest of the processes are done". Do you mean you want it to run at the end of customizing an IMG or at the end of the first system boot? And, by "after all the rest" does it need to run absolutely last, or just "towards the end" of the First Boot?

It does not I am wanting to add the file when I set up the img using sdm.
Alan
Sent from my Verizon, Samsung Galaxy smartphone
Get Outlook for Androidhttps://aka.ms/AAb9ysg

From: Benn @.>
Sent: Friday, August 18, 2023 3:38:49 PM
To: gitbls/sdm @.>
Cc: Alan Bell @.>; Author @.>
Subject: Re: [gitbls/sdm] please excuse my ignorance - How can I add a bash script to the image and have it run at first boot? (Issue #87)
There are several ways to do that. Can I ask a few questions to better help you?
When you say "add a bash script to the IMG", presumably the script already exists on your host computer or network somewhere, is that correct?
"make it run after all the rest of the processes are done". Do you mean you want it to run at the end of customizing an IMG or at the end of the first system boot? And, by "after all the rest" does it need to run absolutely last, or just "towards the end" of the First Boot?
—
Reply to this email directly, view it on GitHubhttps://github.com/gitbls/sdm/issues/87#issuecomment-1684359367, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ADFUXS62EFANVY6FTFH52FTXV7AETANCNFSM6AAAAAA3V52WVU.
You are receiving this because you authored the thread.Message ID: @.***>

"It does not". WHAT does not do what? You're going to have to explain to me what you tried, or what you want to do.
It's easy to add the script to the IMG (personal plugin or runatboot).
The runatboot plugin runs a script as part of the first boot of the system.
Alternatively, you could use your own personal plugin. With that you could copy your script into the IMG and then cause it to run (by some mechanism) during the first system boot, doing effectively what the runatboot plugin does.
If neither of the above help you sufficiently, then please explain in sufficient detail what you're trying to do. My crystal ball is in the repair shop.

References for the two methods mentioned above.

https://github.com/gitbls/sdm/blob/master/Docs/Plugins.md#runatboot — runatboot plugin
https://github.com/gitbls/sdm/blob/master/Docs/Example-Plugin.md — Code example for a plugin

Very sorry that was in response se to the email I received which asked of the file was already on the system. I have batch file that I want to deposit if you will in the image file using sdm and then have it run during the first boot up.
Sent from my Verizon, Samsung Galaxy smartphone
Get Outlook for Androidhttps://aka.ms/AAb9ysg

From: Benn @.>
Sent: Friday, August 18, 2023 4:03:00 PM
To: gitbls/sdm @.>
Cc: Alan Bell @.>; Author @.>
Subject: Re: [gitbls/sdm] please excuse my ignorance - How can I add a bash script to the image and have it run at first boot? (Issue #87)
"It does not". WHAT does not do what? You're going to have to explain to me what you tried, or what you want to do.
It's easy to add the script to the IMG (personal plugin or runatboot).
The runatboot plugin runs a script as part of the first boot of the system.
Alternatively, you could use your own personal plugin. With that you could copy your script into the IMG and then cause it to run (by some mechanism) during the first system boot, doing effectively what the runatboot plugin does.
If neither of the above help you sufficiently, then please explain in sufficient detail what you're trying to do. My crystal ball is in the repair shop.
—
Reply to this email directly, view it on GitHubhttps://github.com/gitbls/sdm/issues/87#issuecomment-1684380806, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ADFUXSZP63W7SENDBJUT7U3XV7C7JANCNFSM6AAAAAA3V52WVU.
You are receiving this because you authored the thread.Message ID: @.***>

The resources I previously mentioned document two different ways for doing this:

https://github.com/gitbls/sdm/blob/master/Docs/Plugins.md#runatboot — runatboot plugin
https://github.com/gitbls/sdm/blob/master/Docs/Example-Plugin.md — Code example for a plugin

With the first method, runatboot, you provide a /path/to/script on your host system. The runatboot plugin copies that into the IMG and runs it during the first boot of the system
With the second method, you would do all the steps that the runatboot plugin does, but with your own code.
It sounds to me like the runatboot plugin would be a better approach for you. Try the following:

sdm --customize /path/to/img.img --plugin runatboot:"script=/path/to/script" [...plus any other switches you want]
When you boot the system, the script will run as part of the first boot of the system

If you're going to use sdm much, you will benefit by becoming familiar with the sdm documentation, perhaps by skimming through it. You'll get a much better understanding of how it works and how you can make it work for you.

Thank you.  I am doing so tests.  If I want to make sure a file which my startup script depends on would I use --exports to add that file to the img and then change the path in my startup script to look for it in the /etc/exports directory?  How about if I need more that one file do I just call the --exports command multiple times?

I think you are confused about the --exports switch. --exports is used with the network file sharing system (NFS) and specifies what directories on the system are to be shared (made available) to other systems via NFS, and with what protections.
In other words, it's not the correct switch for you to be using as you have suggested.
sdm doesn't provide a command-line switch to copy files into the IMG, but it's really easy for you to do it yourself.
If you have files that you want to put into the IMG during a --customize so that they are available when the system boots, the easiest way to do it is to write your own plugin.
It's pretty simple. use sudo and copy the file /usr/local/sdm/plugins/sdm-plugin-template to /usr/local/sdm/local-plugins/myplugin (or whatever name you want to give it).
Then, edit that file using sudo and your editor, and make the changes you need. For examples of copying files into an IMG in a plugin, see https://github.com/gitbls/sdm/blob/master/Docs/Example-Plugin.md, and look for the line
cp -f $csrc/home/$myuser/{.bashrc,.colordiffrc,.dircolors,.emacs,.inputrc,.vimrc,.Xmodmap,.Xdefaults,.tmux.conf} $SDMPT/home/$myuser

That line copies some files from a directory on my local system ($csrc/home/$myuser) into the IMG in the $SDMPT/home/$myuser directory.
A simpler example: Suppose you want to copy the file /home/pi/my-special-file.txt into the IMG at /home/$myuser:
cp /home/pi/my-special-file.txt $SDMPT/home/$myuser

Some details:
$csrc holds the value from the command line --csrc switch. $myuser is the name of the user provided via the --user switch. I recommend using this switch rather than defaulting to the "pi" user, but it's totally up to you. If you don't use the --user switch, $myuser will not be defined.
$SDMPT is a special variable defined by sdm. During sdm Phase 0, the IMG is mounted and is accessed like any other mounted disk. To make sure that you're writing into the IMG rather than your running host system, use $SDMPT.
$SDMPT is not required during the phase 1 or post-install phases, but it doesn't hurt to use it in those phases, since it's defined
to be the null string then.
Note that the example plugin copies many files into the IMG. You can do the same thing, you don't need to have each file copy in a separate plugin, unless you want to, of course.
Lastly, sdm plugins run as root. If the file needs to be owned by a different user, you'll have to chown user:group file to set the file ownership, and you may need to chmod it to set the correct file protection.
Also, you might find this page to be helpful, as it describes the sdm phases, and an overall view of writing a plugin:
https://github.com/gitbls/sdm/blob/master/Docs/Programming-Plugins-and-Custom-Phase-Scripts.mdI
I hope this helps. If anything is unclear, please ask!

If you're having any issues figuring out how to build a plugin to copy the files you want into the IMG, hold off for a bit.
I'm putting together a plugin to do exactly that: Copy specified files into the IMG where you want them, and with the specified owner and protection.
I'll have it done later today or tomorrow at the latest, and will let you know.

Ok, the copyfile plugin is available. Easiest would be to simply reinstall via EZsdmInstaller. Hope you find it useful.

Thank you so much! I am sure I will find it very useful!
Going to give it a try first thing in the morning. It's getting late out here on the east coast!
I am very grateful that you took the time to do this.
Sent from my Verizon, Samsung Galaxy smartphone
Get Outlook for Androidhttps://aka.ms/AAb9ysg

From: Benn @.>
Sent: Sunday, August 20, 2023 8:08:45 PM
To: gitbls/sdm @.>
Cc: Alan Bell @.>; Author @.>
Subject: Re: [gitbls/sdm] please excuse my ignorance - How can I add a bash script to the image and have it run at first boot? (Issue #87)
Ok, the copyfile plugin is available. Easiest would be to simply reinstall via EZsdmInstaller. Hope you find it useful.
—
Reply to this email directly, view it on GitHubhttps://github.com/gitbls/sdm/issues/87#issuecomment-1685434896, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ADFUXS6OUYZEJ2QWMADV7GTXWKRI3ANCNFSM6AAAAAA3V52WVU.
You are receiving this because you authored the thread.Message ID: @.***>

Apologies...forgot to point to the doc: https://github.com/gitbls/sdm/blob/master/Docs/Plugins.md#copyfile

Something to tweak in the doc. The description of the "to" argument says it must be a directory, not a file, but the example shows it as a file...
Arguments

from — /full/path/to/sourcefile on the host system
to — /path/in/IMG to place the file in the IMG. This must be a directory, not a file. The directory must already exist
...

Examples

--plugin copyfile:"from=/usr/local/bin/myconf.conf|to=/usr/local/etc/myconf.conf" The config file will be copied from /usr/local/bin/myconf.conf on the host system to /usr/local/etc/myconf.conf in the IMG during Phase1. The file will be owned by the same user:group as on the host, the file protection will be the same as well.

OOPS.
Since you're using the bootset stuff, can I get your opinion on something? I'm thinking about moving the following command switches into a plugin (and hence removing the command line switches). The names and arguments would stay the same, just changing them into a plugin: bootadd, bootconfig, bootset, dtoverlay, dtparam, hdmigroup, hdmimode, hdmiforcehotplug, hdmiignoreedid.
It reduces complexity in sdm and co-locates all the boot configuration switches (and documentation) in a single plugin.
Your thoughts are appreciated...

BTW, I'm not trying to barge into someone else's thread, but I did try the copy file plugin and it worked great. I had just a single file I wanted to deploy during the burn (having one such file specific to each pi), and copyfile worked great.

BTW, I'm not trying to barge into someone else's thread, but I did try the copy file plugin and it worked great. I had just a single file I wanted to deploy during the burn (having one such file specific to each pi), and copyfile worked great.

No worries, I appreciate the feedback. Good to hear that it worked for you. Still waiting to hear from @AlanEbell on his experience with it, hoping to turn him into a satisfied user as well 👍

It is working fantastic! I am still wrapping my head around the whole process and it's taking me time to familiarize myself with SDM most of the issues I am finding are with my personal knowledge base so I have been slow to respond as I take my time testing.
I appreciate the plug-in and the software a great deal.
On August 22, 2023 9:21:41 PM EDT, Benn @.***> wrote:

BTW, I'm not trying to barge into someone else's thread, but I did try the copy file plugin and it worked great. I had just a single file I wanted to deploy during the burn (having one such file specific to each pi), and copyfile worked great.

No worries, I appreciate the feedback. Good to hear that it worked for you. Still waiting to hear from @AlanEbell on his experience with it, hoping to turn him into a satisfied user as well 👍
--
Reply to this email directly or view it on GitHub:
https://github.com/gitbls/sdm/issues/87#issuecomment-1689123922
You are receiving this because you were mentioned.
Message ID: @.***>

Great! Be aware that there's a bug in the plugin: calling it multiple times on the command line does not work correctly. Will be fixed later this morning USPT.

Multiple invocation bug is corrected in V8.6. Rerun EZsdmInstaller to pick it up.

Thank you!
Sent from my Verizon, Samsung Galaxy smartphone
Get Outlook for Androidhttps://aka.ms/AAb9ysg

From: Benn @.>
Sent: Wednesday, August 23, 2023 1:29:13 PM
To: gitbls/sdm @.>
Cc: Alan Bell @.>; Mention @.>
Subject: Re: [gitbls/sdm] please excuse my ignorance - How can I add a bash script to the image and have it run at first boot? (Issue #87)
Multiple invocation bug is corrected in V8.6. Rerun EZsdmInstaller to pick it up.
—
Reply to this email directly, view it on GitHubhttps://github.com/gitbls/sdm/issues/87#issuecomment-1690358729, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ADFUXS2T77PYHJPOBGWJKSLXWY4WTANCNFSM6AAAAAA3V52WVU.
You are receiving this because you were mentioned.Message ID: @.***>

Closing this issue since no updates in several weeks.


output types not consistent for residual_positions_from_displacements

I have an issue where the outputs of residual_positions_from_displacements are not of consistent type.
As an example:
lon_res, lat_res = residual_positions_from_displacements(ds["lon"],ds["lat"],x,y)
returns lon_res as a numpy.array but lat_res as a xarray dataarray. This makes sense if one looks at the code. I'd like to know how you think this should be handled code-wise.  As an example make sure that everything returned is of type numpy array.

github copilot suggests
def my_function(data):
    if isinstance(data, xr.DataArray):
        data = data.values
    # rest of the function code


jarek/8100: fix kernel restarting

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Terminal stuck after generation

When I run the addon via command in terminal, the terminal freezes after generating all the models. When I comment out the return deffered.promise; line, it works without freezing, but the models arent generated.

I've not come across that. Can I see your openapi.json?

i use this for parsing:
http://petstore.swagger.io/v2/swagger.json

the models are created perfectly :(. But the terminal stops. Seems like there is a process waiting for something.

Hi @queenvictoria ,
Forked my own version, but that one locally is a mess right now :(.
Got back to my local installation and found out was was going on:
At some point the addon is saving the models. When I run it for now i ran into the problem that a file allready exists, but i manually changed some attributes. The terminal is waiting for an answer for overwrite or skip, but I didnt see the question in the terminal and came by after accidentially pushing the Y button on my keyboard.
Next step I ran into a similar problem, when all files are identical, the process isnt stopped properly leaving the terminal in a kind of frozen state.
Do you have a solution?

So the scenario is

The models already exist
There are no changes

And the outcome is

The command never exits.

Is this correct?

Yes!

I've diagnosed this. What is happening is that Blueprint.install is detecting a change in the model and prompting for the user to approve the overwrite. The promises swallow these so nothing appears on screen. If you press return instead of CTRL+C then you 'yes' the prompts and the changes are applied. Neither CLI nor Blueprint seem to provide options for automatically overwriting. So at this time I have no fix.
The alternative is to use the --cli switch which will print out the ember commands which can then be copy pasted into the terminal.


Feature Request: Swipe to toggle unread status ala Mail.app [macOS]

Swipe to mark as read/unread just like in Mail.app (should be same direction too for muscle memory).

I have working code for this if it's decided it should be added. Currently the actions are just text; images à la Mail would be good.

I'm assigning this to 5.1 since we have working code and are going to try to get design assets for it.

This will be part of the 5.1 release.


Force constant chunk size when specified in ForwardDiff

[!WARNING]
This change is technically non-breaking due to ambiguous semantics in ADTypes, but it will make your code error if you specify an AutoForwardDiff chunk size to be larger than the input length. Such code did not error before.

DI extensions

ForwardDiff: for AutoForwardDiff{C}, always build a Chunk{C} without caring about the length of x. This will trigger errors when C > length(x).
PolyesterForwardDiff: pick same chunk size as ForwardDiff (forgotten to add that before)

DIT source

Adapt backend to input length of scenario by thresholding chunk size when necessary. It's either that or filtering the scenarios by input size, which is even worse.

DI tests

Test type stability of preparation for AutoForwardDiff with fixed chunk size

Codecov Report
Attention: Patch coverage is 12.00000% with 22 lines in your changes missing coverage. Please review.

Project coverage is 6.41%. Comparing base (1208b44) to head (f818238).

Files with missing lines
Patch %
Lines

...DifferentiationInterfacePolyesterForwardDiffExt.jl
0.00%
6 Missing :warning:

...entiationInterfaceTest/src/test_differentiation.jl
0.00%
6 Missing :warning:

...dDiffExt/DifferentiationInterfaceForwardDiffExt.jl
0.00%
3 Missing :warning:

DifferentiationInterface/src/utils/batchsize.jl
50.00%
3 Missing :warning:

...erentiationInterfaceTest/src/scenarios/scenario.jl
0.00%
3 Missing :warning:

...xt/DifferentiationInterfaceForwardDiffExt/utils.jl
0.00%
1 Missing :warning:

:exclamation:  There is a different number of reports uploaded between BASE (1208b44) and HEAD (f818238). Click for more details.
HEAD has 49 uploads less than BASE

Flag
BASE (1208b44)
HEAD (f818238)

DIT
10
1

DI
42
2

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #539       +/-   ##
==========================================
- Coverage   98.63%   6.41%   -92.22%
==========================================
  Files         106      93       -13
  Lines        4606    4424      -182
==========================================
- Hits         4543     284     -4259
- Misses         63    4140     +4077

Flag
Coverage Δ

DI
9.45% <18.75%> (-89.31%)
:arrow_down:

DIT
0.13% <0.00%> (-98.22%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

@Vaibhavdixit02 this breaking change appears to make it difficult to specify the chunk size in Optimization.jl, since the same AD backend is used for all funcitons, gradient, constraint jacobian, lagrangian hessian. I just had code broken by updating patch releases due to this.
Ideally, this stance

This change is technically non-breaking due to https://github.com/SciML/ADTypes.jl/issues/90 in ADTypes, but it will make your code error if you specify an AutoForwardDiff chunk size to be larger than the input length. Such code did not error before.

would be avoided, relying on poorly defined semantics to justify making breaking changes in a patch release makes it difficult for users to guard themselves against breakage.

I completely agree, which is why I opened https://github.com/SciML/ADTypes.jl/issues/90 to keep track and clarify the semantics. I decided that such a change wasn't worth the ecosystem-wide update, but I apologize because I know that some code was broken as a result.
Let me explain why this change occurred. Prior to the present PR, I did not understand what fixing the batch size meant for those who use AutoForwardDiff(; chunksize=10), mainly the SciML ecosystem. It is actually used to guarantee type-stability when ForwardDiff.gradient is called without prior creation of a config object. This requires creating a Chunk{C} with a statically-known C=chunksize, which is what this PR ensures. As a result, picking a chunk size smaller than the input length will now cause errors which did not exist before.
In a way, you could think of the previous implementation as incorrect. You asked DI to use ForwardDiff with chunksize=10, and it ended up creating a Chunk{C} with C=min(chunksize,length(x)) (which was type-unstable).
In the case of Optimization.jl, the real issue is that the same backend can be used for all operators. But since Optimization.jl now uses the preparation mechanism of DI, specifying the chunk size has basically no efficiency benefits (I think). So I would recommend using the plain old AutoForwardDiff() for this package. Do you have a specific reason to specify chunk size in your application @baggepinnen?

Do you have a specific reason to specify chunk size in your application @baggepinnen?

Yes, I want to keep the chunk size smaller than the automatically chosen one in order to minimize compile time, which may otherwise reach 10+ minutes for my problem

I understand, and it is a consequence I didn't have in mind. As I said, this PR tries to fix chunk size semantics to be consistent with the rest of the ecosystem. I asked the question on Slack before merging it, and the consensus on the SciML side (to which Optimization.jl belongs) seemed indisputable.

Again, the fact that this happened in a patch release of DI is my fault, and arguably not optimal (unless you consider the previously wrong semantics to be a bug). But the semantics that are in place now seem to be the right ones.
The main issue that remains is sparse AD, because the compressed input dimension is only known after coloring, and usually much smaller than length(x). So there is no good way to set a chunk size at the moment, and I have opened https://github.com/gdalle/DifferentiationInterface.jl/issues/593 to keep track.

I asked the question on Slack before merging it, and the consensus on the SciML side (to which Optimization.jl belongs) seemed indisputable.

This SciML slack is dominated by academics and students to whom stability and maturity of a software ecosystem matters very little. Industrial users, where a team may have few julia experts and tight timelines, breakage like this is seen as a red flag and reasons to not use a technology.
In this particular case, I caught the break before our customers were exposed to it, but us shipping code and customers experiencing breakage when they perform an update that should not break anything looks bad, and is a common complaint I hear about the julia ecosystem in general.
I don't want to give you too much headache here, you're doing a fantastic job with DI.jl! Julia is also making it very hard to nail down interfaces, causing unintended reliance on something poorly defined almost inevitable.

I understand, and I'm grateful for the feedback. To be fair we're also in a corner case of SemVer, where it could be considered either a bug fix or a breaking change. But next time I'll consider the lesson learned and tag a breaking release instead of relying on bad interface definitions.
For dense AD, the solution is to always pick a chunk size smaller than the input. For sparse AD, as explained in #593, it is much trickier, and I could use a hand in figuring things out.


New usage of new architecture gets me confused

I think the official website is very disordered. Maybe as a veteran, you can't feel it. But for newbie, it seems a total mess. I list some issues:

Everyone knows that RN's great new architecture. but when I read the official website, I can't tell which part belongs to new arch, which part doesn't. If the document is the old one, where to find the new document?
If the 0.69 contains the new arch, do we have a dedicated document to explain how to turn on the new arch?
The scaffold and website are out of sync. I use npx react-native init TechApp --template react-native-template-typescript to create a project, then I switch to new arch according to the website's instruction. I have two examples:

It seems the scaffold already has a switch to turn on new arch: "RCT_NEW_ARCH_ENABLED", but it's not enough, because the document says if you want to enable TurboModule, RCTEnableTurboModule(YES) must be called, but the sample code about the switch in scaffold doesn't have this code:

#if RCT_NEW_ARCH_ENABLED
  _contextContainer = std::make_shared<facebook::react::ContextContainer const>();
  _reactNativeConfig = std::make_shared<facebook::react::EmptyReactNativeConfig const>();
  _contextContainer->insert("ReactNativeConfig", _reactNativeConfig);
  _bridgeAdapter = [[RCTSurfacePresenterBridgeAdapter alloc] initWithBridge:bridge contextContainer:_contextContainer];
  bridge.surfacePresenter = _bridgeAdapter.surfacePresenter;
#endif

The document instruction says:" you will create a RCTTurboModuleManager in your bridge delegate’s jsExecutorFactoryForBridge", the code in the document is like this:

#pragma mark - RCTCxxBridgeDelegate

- (std::unique_ptr<facebook::react::JSExecutorFactory>)jsExecutorFactoryForBridge:(RCTBridge *)bridge
{
  // Add these lines to create a TurboModuleManager
  if (RCTTurboModuleEnabled()) {
    _turboModuleManager =
        [[RCTTurboModuleManager alloc] initWithBridge:bridge
                                             delegate:self
                                            jsInvoker:bridge.jsCallInvoker];

    // Necessary to allow NativeModules to lookup TurboModules
    [bridge setRCTTurboModuleRegistry:_turboModuleManager];

    if (!RCTTurboModuleEagerInitEnabled()) {
      /**
       * Instantiating DevMenu has the side-effect of registering
       * shortcuts for CMD + d, CMD + i,  and CMD + n via RCTDevMenu.
       * Therefore, when TurboModules are enabled, we must manually create this
       * NativeModule.
       */
       [_turboModuleManager moduleForName:"DevMenu"];
    }
  }

  // Add this line...
  __weak __typeof(self) weakSelf = self;

  // If you want to use the `JSCExecutorFactory`, remember to add the `#import <React/JSCExecutorFactory.h>`
  // import statement on top.
  return std::make_unique<facebook::react::HermesExecutorFactory>(
    facebook::react::RCTJSIExecutorRuntimeInstaller([weakSelf, bridge](facebook::jsi::Runtime &runtime) {
      if (!bridge) {
        return;
      }

      // And add these lines to install the bindings...
      __typeof(self) strongSelf = weakSelf;
      if (strongSelf) {
        facebook::react::RuntimeExecutor syncRuntimeExecutor =
            [&](std::function<void(facebook::jsi::Runtime & runtime_)> &&callback) { callback(runtime); };
        [strongSelf->_turboModuleManager installJSBindingWithRuntimeExecutor:syncRuntimeExecutor];
      }
    }));
}

but the code in scaffold is like this:
#pragma mark - RCTCxxBridgeDelegate

- (std::unique_ptr<facebook::react::JSExecutorFactory>)jsExecutorFactoryForBridge:(RCTBridge *)bridge
{
  _turboModuleManager = [[RCTTurboModuleManager alloc] initWithBridge:bridge
                                                             delegate:self
                                                            jsInvoker:bridge.jsCallInvoker];
  [bridge setRCTTurboModuleRegistry:_turboModuleManager];

  return RCTAppSetupDefaultJsExecutorFactory(bridge, _turboModuleManager);
}

the biggest difference is the return, they are totally different. which one should I use?
Please help!!!!!

My advice:

Create brand new web pages for new architecture, when users want to try new arch, they click a button on the old web page, then everything is totally switched to the new arch.
We can have an independent tutorial to teach users to upgrade to the new arch, instead of scattering in every corner of the document.

Hi @jasonjiwork, huge thanks for your feedback. We are actually already working on that:

This is a (big) update to The New Architecture section with a lot of extra information and documentation. Our plan is to release it with 0.70. It took some time to write all this stuff and we couldn't release it with 0.69, unfortunately.
This is a review of the whole website, keeping the new architecture into consideration. We are going to merge this one into the first one and then the first one in the main branch and it will become the official documentation on the New Architecture.

The pages you are currently looking at are named New Architecture but they actually contains the Migration steps to move from a version which does not support the new architecture to a version which does. It is clearly stated in the very first page, below the experimental warning:

This migration guide is designed for React Native library authors and application developers. It outlines the steps you need to follow to roll out the new Architecture, composed by the new NativeModule system (TurboModule) and the new Renderer (Fabric) to your Android and iOS libraries and apps.

If you create a new app with the template, you don't need to execute any step of the documentation, the template is already configured to work with both the architectures.
For example, you can see a call to RCTAppSetupPrepareApp(application); that refers to this other function which turn on the TurboModules for you.
If you want to enable the New Architecture for your app, tarting from the template, you just have to:

for iOS:

Navigate to the your_app/ios folder
run bundle install to install the ruby dependencies
run RCT_NEW_ARCH_ENABLED=1 bundle exec pod install to install the iOS dependencies enabling the new architecture.
move back to your_app root folder
run npx react-native run-ios to run the app.

for Android:

open the your_app/android/gradle.properties file
switch the newArchEnabled flag from false to true
run npx react-native run-android to run the app.

I hope this could help you. Bear with us, the documentation will arrive soon.
Thank you again for all your feedback! :pray:

Wow, so quick response!
Thank you @cipolleschi !
my confusion has gone, now I can safely use the template without anxiety.
Really looking forward to 0.70, it's a big step for RN.
By the way, I can't find the documentation of how to create TurboModule, do we have related documentation now?

You are welcome, happy to help.
In the draft I shared with you there is a page called pillars-turbomodule which explain that. Moreover, you can refer to these two other repos:

RNNewArchitectureApp
RNNewArchitectureLibraries

The first contains a step-by-step migration of apps from 0.67.4 (the last version without the New Architecture support) to 0.68, 0.69 and 0.70.0-rc1.
The last steps of these guides are how to create TurboModules and Fabric Components for the New Architecture.
The latter repo contains guides on how to create backward compatible TurboModules and Fabric Components for 0.68 and 0.70.0-rc.0 (nothing really changed from this perspective between 0.68 and 0.69, that's why we don't have a 0.69 version).

my confusion has gone, now I can safely use the template without anxiety.

Thanks for raising your concerns @jasonjiwork and sharing your feedbacks. As @cipolleschi said, we took an iterative approach to the New Architecture and we're updating the older documentation to make clearer the separation of Old vs New Architecture.
Feel free to share more comments/feedbacks if you have any.
Closing as the issue is resolved 👍


Fixes a typo in a logging message

Review by @sjrd

Can one of the admins verify this patch?

LGTM, Jenkins, test this please

Refer to this link for build results (access rights to CI server needed):
https://scala-webapps.epfl.ch/jenkins/job/scalajs-pr/2945/
https://scala-webapps.epfl.ch/jenkins/job/scalajs-matrix-build/3820/
Test PASSed.


Unable to start fullnode/complie

I'm following the instructions on https://docs.sui.io/build/fullnode on how to a sui full node.
I'm on Ubuntu 20.04.4
When starting/compiling using the following command:
cargo run --release --bin sui-node -- --config-path fullnode.yaml
I get the following error message:
error[E0658]: use of unstable library feature 'bool_to_option'
   --> /root/.cargo/git/checkouts/mysten-infra-e26c6e4a001ad289/7e9b756/crates/telemetry-subscribers/src/lib.rs:203:57
    |
203 |         let json_storage_layer = config.json_log_output.then_some(JsonStorageLayer);
    |                                                         ^^^^^^^^^
    |
    = note: see issue #80967 <https://github.com/rust-lang/rust/issues/80967> for more information

For more information about this error, try `rustc --explain E0658`.
error: could not compile `telemetry-subscribers` due to previous error

Am I missing something? Thanks!

Ah, you are probably using a rust version that's older than the one recommended for use in compiling the project. Here's what the project recommends:
huitseeker@Garillots-MBP.localdomain➜~/tmp/sui(main✗)» cat rust-toolchain
1.62.1

Here's how to discover what you have installed:
huitseeker@Garillots-MBP.localdomain➜~/tmp/sui(main✗)» rustc --version
rustc 1.62.1 (e092d0b6b 2022-07-16)

If you have installed rust with rustup, your tooling should pick that up automatically and resolve any issues. If you have installed it with your distro's package manager, I actually recommend uninstalling and installing rustup for an overall nicer experience.
/cc-ing @Clay-Mysten for a possible need to update our docs.

Ah, per @damirka, we recently updated:
https://docs.sui.io/build/install#rust
To recommend first:
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
Then:
rustup update stable
Is this hard-coding the version to 1.2? Should I revert that change?
Also, I'm running into license check errors on merge and wondering if I need to to install cargo-hakari and run: cargo hakari generate
Related? Adding folks who should know.

More reports of fullnode breakage:
https://mysten-labs.slack.com/archives/C03Q0KVGJ9K/p1658360418874029
Namely, a missing Compose file.

Thanks I was on 1.59, upgraded to 1.62 and was able to compile and run.
Had to remove cargo first since rustup wasn't installed and I wasn't able to install it with rust already installed.
Here are the steps I used to upgrade to 1.62
sudo apt remove cargo
sudo apt autoremove

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Thanks, Steven. I still think we can do a better job in the docs. Reopening.


pattern= doesn't seem to work as expected

See jsfiddle at http://jsfiddle.net/d5w4jpxq/71/ :
<input name="hashtags" type="tags" pattern="^#" placeholder="#hashtags">
Still allows data that doesn't start with a hash to be entered. The docs mention HTML5 pattern support - should we use pattern to limit the input or should we do this via change events?

PS - thankyou Jason!

@mikemaccana Looks like there were actually a few copies of tags-input.js on that page and the one being used didn't have a placeholder defined.  I've modified it to just use one copy.  Also the placeholder attribute is a pre-anchored regex, so the pattern was invalid.  Here's an example with a valid placeholder and some CSS to show validation state:
http://jsfiddle.net/developit/nL1h3dwk/
However, you're right - the field gets marked as invalid like we would expect, but tags-input doesn't care and happily adds the tag.  The fix here should be pretty simple - we need to have the method that commits a tag from the input field first check that input.validity.valid is true.


Unable to View Queries in Call Tree for a Spring Boot Application

Summary

핀포인트 웹에서 개별 요청을 선택 후 Call Tree를 보면 쿼리가 조회되지 않습니다. Call Tree에서 쿼리까지 확인하려면 어떻게 해야하나요?
JPA 사용과 관련 있는 것 같은데 어떻게 해결해야 할지 모르겠습니다.
When I select an individual request in Pinpoint and look at the Call Tree, the query that is sent to the actual DB (MariaDB) is not traced, but only up to TransactionAspectSupport.invokeWithinTransaction() in the SPRING_TX API. How can I see up to the query in the call tree?
It seems to be related to using JPA, but I don't know how to solve it.

Versions

HBase: 1.4.9
Pinpoint Web: 2.5.2
Pinpoint Collector: 2.5.2
Pinpoint Agent: 2.5.2
Spring Boot: 2.7.3
MariaDB: 10.5.18

Env.

Current Outcome

I am using Pinpoint to monitor a Spring Boot application which is using JPA(Hibernate) for MariaDB connections.
When I select an individual request in Pinpoint and look at the Call Tree, I don't see the actual query sent to the DB but only up to invokeWithinTransaction() method of which belongs to SPRING_TX API.

Expected Outcome

SQL queries shown in the call tree (demo)

Config file (Agent)
pinpoint-agent-2.5.2/profiles/release/pinpoint.config
# MYSQL
#
# Profile MySQL.
profiler.jdbc.mysql=true
# Allow profiling of setautocommit.
profiler.jdbc.mysql.setautocommit=false
# Allow profiling of commit.
profiler.jdbc.mysql.commit=false
# Allow profiling of rollback.
profiler.jdbc.mysql.rollback=false
# Trace bindvalues for MySQL PreparedStatements (overrides profiler.jdbc.tracesqlbindvalue)
#profiler.jdbc.mysql.tracesqlbindvalue=false

#
# MARIADB
#
# Profile MariaDB
profiler.jdbc.mariadb=true
# Allow profiling of setautocommit.
profiler.jdbc.mariadb.setautocommit=true
# Allow profiling of commit.
profiler.jdbc.mariadb.commit=true
# Allow profiling of rollback.
profiler.jdbc.mariadb.rollback=true
# Trace bindvalues for MariaDB PreparedStatements (overrides profiler.jdbc.tracesqlbindvalue)
profiler.jdbc.mariadb.tracesqlbindvalue=true

#
# MSSQL JDBC
#
# Profile Mssql
profiler.jdbc.mssql=false
profiler.jdbc.mssql.setautocommit=false
profiler.jdbc.mssql.commit=false
profiler.jdbc.mssql.rollback=false
profiler.jdbc.mssql.tracesqlbindvalue=false

Similar Issue
#9703

https://github.com/pinpoint-apm/pinpoint/tree/master/plugins/mariadb-jdbc
Could you check the driver support of Maria db?

Thank you for your guidance in resolving the issue. After adjusting the MariaDB version, I can now see the actual query in the Call Tree.
Just to provide some additional information, I have tested multiple versions of the org.mariadb.jdbc:mariadb-java-client as follows:

org.mariadb.jdbc:mariadb-java-client version
SQL Query Shown

3.1.4
x

3.1.0
x

3.0.7
x

3.0.3
x

2.7.9
o

2.7.0
o

It appears that from version 3.0.3 onwards, the SQL queries are not shown in the Call Tree.
So i think the current compatibility range for org.mariadb.jdbc:mariadb-java-client need to be changed to [1.1, 2.7]
@emeroad

org.mariadb.jdbc:mariadb-java-client version
SQL Query Shown

3.1.4
x

3.1.0
x

3.0.7
x

3.0.3
x

2.7.10
x

2.7.9
o

2.7.0
o

Also 2.7.10 version is not supported.

@Jinwook94
#11577
Thank you.


Cannot format time properly unless using your directives

I have tried repeatedly to use the picker without your directives for formatting the resulting time and it never work correctly... Is it possible for you to add a sample that does not require your standardTimeNoMeridian and standardTimeMeridian directives? Or just embed them as options for the timePicker?

As i mentioned in the description, the current time picker only supports UTC time. The standard way is to pass the epoch time. If you want to convert the time from epoch to other format, then you can use any of my directive.
Could you please show me your code? So that i can get exact idea of how you want to use my component.


Remove cluster group and subscription overrides for stress test release

The Azure Pipelines UI marks parameters that default to an empty string as "Required" which defeats the whole purpose of
having empty defaults. The overrides are only there as an option for people to use the pipeline for custom dev
environments. Since I think this scenario is pretty rare, I'm just going to remove support for it (you can still
deploy to custom dev environments via the same underlying scripts). Instead, this change simplifies the parameters
to have an environment drop-down that shows you all options to pick from.

The following pipelines have been queued for testing:java - templatejava - template - testsjs - templatenet - templatenet - template - testspython - templatepython - template - tests You can sign off on the approval gate to test the release stage of each pipeline. See eng/common workflow

/check-enforcer reset

/check-enforcer evaluate

@weshaggard bump


open-vsx support

Please support https://open-vsx.org/
It's used for vscodium https://vscodium.com/

I'll see. The last time I looked at it, they required far too much personal information simply to create an account, but I'll see if I can get around that.
In the meantime, you can download the vsix from the MS marketplace, and install it manually.

It's published

thank you


Fix hw_scsi_model option

This change sets the hw_scsi_model to "virtio-scsi". The use of
"virtio-blk" is assumed however for performance reasons, as noted in
horizon and by redhat we need to set this to "virtio-scsi" [0] to get
the most out of the images we provide by default.
[0] - https://redhatstackblog.redhat.com/2017/01/18/9-tips-to-properly-configure-your-openstack-instance/
Signed-off-by: Kevin Carter<EMAIL_ADDRESS>Issue: RO-3080

recheck_cit_artifact


Add functionality to create RepositoryServer CR using kanctl command line

Change Overview
Add functionality to create  RepositoryServer CR using kanctl command line tool.
Like Profile, RepositoryServer could also be created using kanctl create repository-server
Pull request type
Please check the type of change your PR introduces:

[ ] :construction: Work in Progress
[ ] :rainbow: Refactoring (no functional changes, no api changes)
[ ] :hamster: Trivial/Minor
[ ] :bug: Bugfix
[x] :sunflower: Feature
[ ] :world_map: Documentation
[ ] :robot: Test

Issues

fixes #issue-number

Test Plan

[x] :muscle: Manual
[ ] :zap: Unit test
[ ] :green_heart: E2E

Testing
1) Build kanctl
go build -o kanctl cmd/kanctl/main.go

2) Create Repository Server using kanctl
./kanctl create repository-server \
--prefix /repo-controller/ \
--repositoryServerUser kanisterUser --repositoryServerUserAccessSecret repository-server-user-access \
--repositoryServerAdminUser kanisterAdmin --repositoryServerAdminUserAccessSecret repository-admin-user \
--tlsSecret kanister/repository-server-tls-cert --repositoryPasswordSecret repo-pass \
--locationSecret s3-location --locationCredsSecret s3-creds

repository-server 'kopia-repo-server-nzc5f' created

@r4rajat can you please update the PR description with right command kanctl create, it seems to use --repositoryServerAdminUser which we have removed

Can we write a couple of tests for creation of repository server?

@r4rajat can you please update the PR description with right command kanctl create, it seems to use --repositoryServerAdminUser which we have removed

Updated !


Add AUP signature deletion description

In particular, update the AUP signature API section with what has been added with this PR: https://github.com/indigo-iam/iam/pull/777.

PR https://github.com/indigo-iam/iam-website/pull/124.


Template to control selected value

I think we need more control over the selected value.  A function to transform the value based on the selected document would solve this problem. I.e match is made on Document.X but selection result is Document.Y or something more complex (like Document.X + '-' + Document.Y)
I hit this problem when writing an emoji look up using a colon : as a token. The result was ::smile: when i wanted :smile

+1 but I guess this could also be done with just an option to exclude the token or?

+1 I love to have this feature as well

+1

I don't understand exactly what the request is. Can you provide an example?

For example, an emoji's code has a name wrapped with colons like :grin:. So when user is typing in text in an input section, the expected behavior is that when user types in :, the autocompletion should be triggered. However, if we set the token to be : and try to match the emoji's code, then after the user selects the item from the suggestion list, the input content will be something like ::grin:.

Actually there is another issue with this example. Right now if the field being matched contains character same as the token, then a new completion is triggered. So if the user types in :grin: trying to match the emoji code :grin:, this package will treat the input as :grin and :. I think sometimes this is desired, but it would be nice to have an option to allow match text "overwriting" the token.

Another example is tagging other users. An user may wants to tag a person by searching for the real name. Let's say we match the field lastname, but when the user selects one item from the suggestion list, we may want to show username instead.
If there is a user with lastname Brown and username br012, and another person might want to type in @Brown and sees the input content as @br012.

Hi all - I am currently working on adding autoform support and would allow the control of selected values as part of that. Any input would be appreciated.

+1


server: Iteratively remove jobs on cancellation and allocate av2-all correctly

Make sure we kill all the jobs in the list, since we are having more than one runs for a given run_id due to set and preset nature.

This PR also addresses small bug-fix for av2-all allocation which did not work all the time.

For c18495a3e540e916ec4d9f3e54e54b3037ad960c, relevant frontend change https://github.com/xiph/awcy/pull/337


Temp

Pull Request Test Coverage Report for Build 304

0 of 0 (NaN%) changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 90.211%

Totals

Change from base Build 303:
0.0%

Covered Lines:
2350

Relevant Lines:
2605

💛  - Coveralls


Is there a plan for I18N?

hi,
This is a very good software, a great tool. If you can have a localized display would be better, what do you think?

I am definitely interested in localization.  I will look in to this.

I'm working on moving all of my text strings to a separate resource file which will help with localization.  Once that's done, would you be able to help with translation?

I am very willing to contribute to the localized translation of Simplified Chinese.

I can do Fench


Pinned oqpy version breaks tensorflow imports

Describe the bug
The oqpy version specified as a dependency (0.1.0) pins numpy==1.21.6. Recent versions of tensorflow-macos compile against newer versions of numpy, causing subsequent tensorflow imports to fail.
To reproduce
From within a fresh conda virtual environment on python 3.9.13, I ran:
python -m pip install amazon-braket-sdk
python -m pip install tensorflow-macos
python
>>> import tensorflow as tf

The import statement results in:
RuntimeError: module compiled against API version 0x10 but this version of numpy is 0xe
RuntimeError: module compiled against API version 0x10 but this version of numpy is 0xe
ImportError: numpy.core._multiarray_umath failed to import
ImportError: numpy.core.umath failed to import
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/danielcanuto/miniconda3/envs/dep-bug/lib/python3.9/site-packages/tensorflow/__init__.py", line 37, in <module>
    from tensorflow.python.tools import module_util as _module_util
  File "/Users/danielcanuto/miniconda3/envs/dep-bug/lib/python3.9/site-packages/tensorflow/python/__init__.py", line 42, in <module>
    from tensorflow.python import data
  File "/Users/danielcanuto/miniconda3/envs/dep-bug/lib/python3.9/site-packages/tensorflow/python/data/__init__.py", line 21, in <module>
    from tensorflow.python.data import experimental
  File "/Users/danielcanuto/miniconda3/envs/dep-bug/lib/python3.9/site-packages/tensorflow/python/data/experimental/__init__.py", line 96, in <module>
    from tensorflow.python.data.experimental import service
  File "/Users/danielcanuto/miniconda3/envs/dep-bug/lib/python3.9/site-packages/tensorflow/python/data/experimental/service/__init__.py", line 419, in <module>
    from tensorflow.python.data.experimental.ops.data_service_ops import distribute
  File "/Users/danielcanuto/miniconda3/envs/dep-bug/lib/python3.9/site-packages/tensorflow/python/data/experimental/ops/data_service_ops.py", line 24, in <module>
    from tensorflow.python.data.experimental.ops import compression_ops
  File "/Users/danielcanuto/miniconda3/envs/dep-bug/lib/python3.9/site-packages/tensorflow/python/data/experimental/ops/compression_ops.py", line 16, in <module>
    from tensorflow.python.data.util import structure
  File "/Users/danielcanuto/miniconda3/envs/dep-bug/lib/python3.9/site-packages/tensorflow/python/data/util/structure.py", line 23, in <module>
    from tensorflow.python.data.util import nest
  File "/Users/danielcanuto/miniconda3/envs/dep-bug/lib/python3.9/site-packages/tensorflow/python/data/util/nest.py", line 36, in <module>
    from tensorflow.python.framework import sparse_tensor as _sparse_tensor
  File "/Users/danielcanuto/miniconda3/envs/dep-bug/lib/python3.9/site-packages/tensorflow/python/framework/sparse_tensor.py", line 24, in <module>
    from tensorflow.python.framework import constant_op
  File "/Users/danielcanuto/miniconda3/envs/dep-bug/lib/python3.9/site-packages/tensorflow/python/framework/constant_op.py", line 25, in <module>
    from tensorflow.python.eager import execute
  File "/Users/danielcanuto/miniconda3/envs/dep-bug/lib/python3.9/site-packages/tensorflow/python/eager/execute.py", line 23, in <module>
    from tensorflow.python.framework import dtypes
  File "/Users/danielcanuto/miniconda3/envs/dep-bug/lib/python3.9/site-packages/tensorflow/python/framework/dtypes.py", line 34, in <module>
    _np_bfloat16 = _pywrap_bfloat16.TF_bfloat16_type()
TypeError: Unable to convert function return value to a Python type! The signature was
	() -> handle

Expected behavior
To be able to import tensorflow.
System information

Amazon Braket Python SDK version: 1.33.0.post0
Amazon Braket Python Schemas version: 1.12.0
Amazon Braket Python Default Simulator version: 1.10.0
Python version: 3.9.13

Additional context
I don't know if it would break things elsewhere, but oqpy 0.1.1 has relaxed its numpy version requirement in a way that's compatible with tensorflow-macos. In other words, in a fresh virtual environment, the following works:
python -m pip install oqpy==0.1.1
python -m pip install tensorflow-macos
python
>>> import tensorflow as tf
>>> print(tf.reduce_sum(tf.random.normal([1000, 1000])))
tf.Tensor(-1386.055, shape=(), dtype=float32)

Hello, we are in the process of making the update to oqpy v0.1.1. This can be tracked with https://github.com/aws/amazon-braket-sdk-python/pull/469.

Oqpy update is complete, marking this issue as resolved. Feel free to reopen if there are still issues!


SDA-8838 | fix: STS default warning does not show for hcp

Related issue: https://issues.redhat.com/browse/SDA-8838

/lgtm


Throwing errors on Ubuntu both nodejs and nodejs-legacy

><EMAIL_ADDRESS>start /root/src/node_modules/bitcore-wallet-service
> node app.js

Bitcore Wallet Service running on port 3001

events.js:72
        throw er; // Unhandled 'error' event
              ^
OpenError: IO error: ./db/bws.db/LOCK: No such file or directory
    at /root/src/node_modules/bitcore-wallet-service/node_modules/levelup/lib/levelup.js:118:34

npm ERR<EMAIL_ADDRESS>start: `node app.js`
npm ERR! Exit status 8
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>start script.
npm ERR! This is most likely a problem with the bitcore-wallet-service package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node app.js
npm ERR! You can get their info via:
npm ERR!     npm owner ls bitcore-wallet-service
npm ERR! There is likely additional logging output above.
npm ERR! System Linux 3.16.0-23-generic
npm ERR! command "/usr/bin/nodejs" "/usr/bin/npm" "start"
npm ERR! cwd /root/src/node_modules/bitcore-wallet-service
npm ERR! node -v v0.10.25
npm ERR! npm -v 1.4.21
npm ERR! code ELIFECYCLE
npm WARN This failure might be due to the use of legacy binary "node"
npm WARN For further explanations, please read
/usr/share/doc/nodejs/README.Debian

npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR!     /root/src/node_modules/bitcore-wallet-service/npm-debug.log
npm ERR! not ok code 0

I followed instructions on http://stackoverflow.com/questions/21168141/can-not-install-packages-using-node-package-manager-in-ubuntu and installed legacy using sudo apt-get install nodejs-legacy
Still does not fix the problem...
Any tips?

0 info it worked if it ends with ok
1 verbose cli [ '/usr/bin/nodejs', '/usr/bin/npm', 'start' ]
2 info using<EMAIL_ADDRESS>3 info using<EMAIL_ADDRESS>4 verbose run-script [ 'prestart', 'start', 'poststart' ]
5 info prestart<EMAIL_ADDRESS>6 info start<EMAIL_ADDRESS>7 verbose unsafe-perm in lifecycle true
8 info<EMAIL_ADDRESS>Failed to exec start script
9 error<EMAIL_ADDRESS>start: `node app.js`
9 error Exit status 8
10 error Failed at the<EMAIL_ADDRESS>start script.
10 error This is most likely a problem with the bitcore-wallet-service package,
10 error not with npm itself.
10 error Tell the author that this fails on your system:
10 error     node app.js
10 error You can get their info via:
10 error     npm owner ls bitcore-wallet-service
10 error There is likely additional logging output above.
11 error System Linux 3.16.0-23-generic
12 error command "/usr/bin/nodejs" "/usr/bin/npm" "start"
13 error cwd /root/src/node_modules/bitcore-wallet-service
14 error node -v v0.10.25
15 error npm -v 1.4.21
16 error code ELIFECYCLE
17 verbose exit [ 1, true ]

Here's the /bitcore-wallet-service/npm-debug.log

Hello @dabura667 ,
This error is because the folder db/bws.db doesn't exist. We will fix that as soon as possible. Anyway you can manually create it with mkdir -p db/bws.db
Thanks for report!


Removed: webpack build notifications

If you want to add notifications back, please directly use
webpack-notifier
Closes #859

I wanted it behind a flag in config, it is really useful, if for some reason it becomes annoying you could just disable it

I might add better notifications more accurate, with Phenomic icon.

(like Jest does)

Thats better!

Failure on appveyor is realted to jest-snapshot update, no big deal (fixed in master)


Migrate elements to usage via CDN

Fixes the elements crash
Migrate to using elements via UMD - read about differences and benefits here
Use elements UMD types package for TS support

@baktun14 I have migrated elements to usage via CDN instead of using as an NPM package. It fixes the crash that's happening due to a Nextjs SWC minification issue.
Some of the other benefits of the UMD build are

Bundle size of your dApp decreases
No build/bundling for elements needed

The leap wallet styling overrides the app styling

@baktun14 thank you for making a note of this. It's a bug with the stylesheet for version 1.4.1. We're releasing version 1.4.2 today/tomorrow which will have the fix for this. Will update then.

Hi @baktun14 I have fixed the issue with the stylesheets. It won't affect the rest of the app.

Hi @baktun14 I have fixed the issue with the stylesheets. It won't affect the rest of the app.

The styling issues seem to have been fixed. 👍

@siddharthborderwala Hey, I rebased and fixed some issues, but for some reason the @leapwallet/elements package is still a dependency of the types packages and it messed up our CI...
https://github.com/akash-network/console/actions/runs/10888482586/job/30213077526?pr=317


Feat/response in use indicator

https://app.asana.com/0/1170145499389908/1169689593066905

feat: indicator in the story editor when a response is used in another story
chore: re-usable HoverablePopup component (semantic Uis is buggy)

I missed a minor thing the "utter_..."  is not bold as in the task description

The app crashes when clicking the links in the popup. See video in Asana


Ongoing: update to MUI v5

The following components would benefit from an upgrade to v5 of MUI - shortcut ticket:

Button (v5 allows for intents)
Tooltip (v5 has the followCursor prop by default, which tooltipTable and tooltipCondensed use)
Tabs (TabsProps)

These components have been modified in accordance with SDS's migration to MUI V5.


use addHandlerForQuerySelector

Rationale
use addHandlerForQuerySelector when rendering helpPopup to reduce number or registered events.
48714
Related Pull Requests

Changes

Would it be worthwhile to cap the number of handlers in PageConfig to something that's ridiculously high but enough to protect the server? 100,000? 1,000,000? We could log an error/warning.

Would it be worthwhile to cap the number of handlers in PageConfig to something that's ridiculously high but enough to protect the server? 100,000? 1,000,000? We could log an error/warning.

Probably a good idea.


Map Ruby's C variable error names to their Ruby names. Fixes #1270

Description
PR to address the mapping of core Ruby error names when using the Ruby C API.
https://github.com/lsegal/yard/issues/1270
I added a Rake task to scrape the master Ruby source for the names of the error classes:
bundle exec rake update_error_map

There' wasn't any existing .rake files in this project, so I modelled it after what RuboCop does.
Completed Tasks

[x] I have read the Contributing Guide.
[x] The pull request is complete (implemented / written).
[x] Git commits have been cleaned up (squash WIP / revert commits).
[x] I wrote tests and ran bundle exec rake locally (if code is attached to PR).

Coverage increased (+0.04%) to 93.806% when pulling 7a0fb20131cce90634712178aef8a40cdc35db07 on thomthom:dev-error-inheritance into 890d4ad5f714856a138a5e2aae3b226255e74137 on lsegal:master.

This took a while but thanks for the work on this!


Add Left Navigation for Emissions Module

As an ECMPS user, I want to be able to navigate to hourly, daily, and MATS pages via the left navigation panel in the ECMPS browser.

[ ] Add left-hand navigation items

[ ] Similar to QA Cert and Test extension Menu items

Note: This ticket does not cover any view generation after clicking on one of the navigation buttons. The page does not exist will be displayed

@aprematta Ticket 3959 is ready for 508 testing and has been added to the Emissions Module 508 Testing Epic #4071


NaN in output - running Tensorflow2 classification with softmax on DPUCZDX8G

Summary
I have trained and successfully quantized and compiled a 3-layer MNIST classification model for DPUCZDX8G. But when I'm trying to run my model on the ZCU102 board I get unexpected NaNs and zeros in the classification output vector where only a few instances of the test images results in random float numbers but with very poor accuracy.

Output examples:
...most of the output:

[[nan  0. nan  0.  0. nan  0.  0.  0.  0.]]

...and in some instances:

[[2.9359882e-30 3.3529205e-04 2.2591801e-06 9.9949145e-01 2.2591801e-06
1.5222234e-08 4.5376844e-05 9.3528639e-14 1.2334704e-04 8.5287086e-17]

I had no issues in the previous deployment steps so what could cause this? Am I somehow interpreting the output wrong?
MNIST model definition
 inputs = tf.keras.layers.Input(shape=(784,))
    dense = tf.keras.layers.Dense(128, activation='relu')(inputs)
    net = tf.keras.layers.Dense(10)(dense)
    prediction = tf.keras.layers.Activation('softmax')(net)
    float_model = tf.keras.Model(inputs=inputs, outputs=prediction, name='mnist_softmax_model')

Model: "mnist_softmax_model"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_1 (InputLayer)         [(None, 784)]             0
_________________________________________________________________
quant_input_1 (VitisQuantize (None, 784)               4
_________________________________________________________________
quant_dense (QuantizeWrapper (None, 128)               100487
_________________________________________________________________
quant_dense_relu (QuantizeWr (None, 128)               4
_________________________________________________________________
quant_dense_1 (QuantizeWrapp (None, 10)                1300
_________________________________________________________________
activation (Activation)      (None, 10)                0
=================================================================
Total params: 101,795
Trainable params: 101,770
Non-trainable params: 25
_________________________________________________________________

Evaluation of quantized model
2022-05-12 13:12:36.399445: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)
313/313 [==============================] - 1s 968us/step - loss: 0.2457 - sparse_top_k_categorical_accuracy: 0.9963
Evaluate on test data
79/79 [==============================] - 0s 2ms/step - loss: 0.2457 - sparse_top_k_categorical_accuracy: 0.9963
test loss, test acc: [0.24574635922908783, 0.9962999820709229]
Generate predictions for 3 samples
Prediction: 9   Truth:  9
Prediction: 2   Truth:  2
Prediction: 7   Truth:  7

Compiling model for DPUCZDX8G
$ ARCH=/opt/vitis_ai/compiler/arch/DPUCZDX8G/ZCU102/arch.json
$ MODEL=quantized_models/mnist_MLP_softmax_q.h5
$ NAME=mnist_MLP_softmax_c
$ vai_c_tensorflow2 -m $MODEL -a $ARCH -o compiled_models/ -n $NAME
[INFO] Namespace(batchsize=1, inputs_shape=['1,784'], layout='NHWC', model_files=['quantized_models/mnist_MLP_softmax_q.h5'], model_type='tensorflow2', named_inputs_shape=None, out_filename='/tmp/mnist_MLP_softmax_c_org.xmodel', proto=None)
in_shapes: [[1, 784]]
[INFO] tensorflow2 model: /workspace/quantized_models/mnist_MLP_softmax_q.h5
[INFO] keras version: 2.6.0
[INFO] Tensorflow Keras model type: functional
[INFO] parse raw model     :100%|██████████| 6/6 [00:00<00:00, 12222.35it/s]
[INFO] infer shape (NHWC)  :100%|██████████| 12/12 [00:00<00:00, 13213.87it/s]
[INFO] perform level-0 opt :100%|██████████| 2/2 [00:00<00:00, 528.38it/s]
[INFO] perform level-1 opt :100%|██████████| 2/2 [00:00<00:00, 1931.08it/s]
[INFO] generate xmodel     :100%|██████████| 12/12 [00:00<00:00, 769.08it/s]
[INFO] dump xmodel ...
[INFO] dump xmodel: /tmp/mnist_MLP_softmax_c_org.xmodel
[UNILOG][INFO] Compile mode: dpu
[UNILOG][INFO] Debug mode: function
[UNILOG][INFO] Target architecture: DPUCZDX8G_ISA0_B4096_MAX_BG2
[UNILOG][INFO] Graph name: mnist_softmax_model, with op num: 16
[UNILOG][INFO] Begin to compile...
[UNILOG][INFO] Total device subgraph number 4, DPU subgraph number 1
[UNILOG][INFO] Compile done.
[UNILOG][INFO] The meta json is saved to "/workspace/compiled_models/meta.json"
[UNILOG][INFO] The compiled xmodel is saved to "/workspace/compiled_models//mnist_MLP_softmax_c.xmodel"
[UNILOG][INFO] The compiled xmodel's md5sum is 5706a3276db2dc636b0448d03f5c121a, and has been saved to "/workspace/compiled_models/md5sum.txt"
**************************************************
* VITIS_AI Compilation - Xilinx Inc.
*************************************************

Python VART code for deployment on ZCU102
import xir
import vart
from vitis_ai_library import GraphRunner
import numpy as np
import sys
import tensorflow as tf
import math

def trans_3D_to_2D(tensor):
    if type(tensor) is not np.ndarray:
        tensor = np.array(tensor)
    shape = tensor.shape
    if len(shape) != 3:
        print("Tensor needs to be 3D")
        return None
    # keep batch size (dim 0) dimension
    return tensor.reshape(shape[0], shape[1]*shape[2])

"""Import MNIST data"""
mnist = tf.keras.datasets.mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()
# reshape data to 2D tensors (batch, pixel vector)
#x_train = trans_3D_to_2D(x_train)
x_test = trans_3D_to_2D(x_test)

"""Import model execution graph"""
model_name = str(sys.argv[1])#'models/MLP_2L.xmodel'
g = xir.Graph.deserialize(model_name)

"""Print graph Info"""
subgraphs = g.get_root_subgraph().toposort_child_subgraph()
for s in subgraphs: print(s.get_name())

runner = GraphRunner.create_graph_runner(g)
print("created runner")

"""Define I/O buffers"""
input_tensor_buffers = runner.get_inputs()
output_tensor_buffers = runner.get_outputs()

input_ndim = tuple(input_tensor_buffers[0].get_tensor().dims)
batch = input_ndim[0]
width = input_ndim[1]
try:
    height = input_ndim[2]
except:
    height = "no dim"
else:
    height = input_ndim[2]
fixpos = input_tensor_buffers[0].get_tensor().get_attr("fix_point")

print("INPUT INFO:\t", input_ndim, "\nBATCH\t", batch, "\nWIDTH\t", width, "\nHEIGHT\t", height, "\nFIXPOS\t", fixpos)

"""init input data """
input_data = np.asarray(input_tensor_buffers[0])

for imgnr in range(10):               #Test the first 10 MNIST images
    input_data[0] = x_test[imgnr]
    #show_img(input_data[0])

    """ run graph runner"""
    job_id = runner.execute_async(input_tensor_buffers, output_tensor_buffers)
    runner.wait(job_id)
    print("started graph runner")

    pre_output_size = int(output_tensor_buffers[0].get_tensor().get_element_num() / batch)
    raw_output = output_tensor_buffers[0]
    output_data = np.asarray(raw_output, dtype=np.float32, order="C")
    pred = np.argmax(output_data)

    # print output arrays without NaN and just a few with
    if not(np.isnan(output_data).any()) or (imgnr%201 == 0):
        print("IMAGE NR ", imgnr)
        print("OUTPUT:\n", output_data, "\nRAW OUTPUT:\n", raw_output, "\nPRE OUTPUT SIZE:\n", pre_output_size, "\nPREDICTION:\t", pred, "\tTRUTH:\t", y_test[imgnr])

Results of running the model
The great majority of all test images in the data set gives an output with zeros and nan as shown below as for example image nr 0.
root@xilinx-zcu102-2021_2:~/loo-vart-exploration# python mnist_runner.py models/mnist_MLP_softmax_c.xmodel
subgraph_input_1
subgraph_quant_input_1_reshaped
subgraph_quant_dense(TransferMatMulToConv2d)
subgraph_activation
created runner
INPUT INFO:	 (1, 784)
BATCH	 1
WIDTH	 784
HEIGHT	 no dim
FIXPOS	 -2
started graph runner
IMAGE NR  0
OUTPUT:
 [[nan  0. nan  0.  0. nan  0.  0.  0.  0.]]
RAW OUTPUT:
 TensorBuffer{@0xaaaac273f930,tensor=xir::Tensor{name = activation, type = FLOAT32, shape = {1, 10}},location=HOST_VIRT,data=[(Virt=0xaaaac431eaa0, 40)]}
PRE OUTPUT SIZE:
 10
PREDICTION:	 0 	TRUTH:	 7
started graph runner
started graph runner
started graph runner
started graph runner
started graph runner
started graph runner
started graph runner
started graph runner
started graph runner

Only a few of all test images generate an output that does not contain zeros and nan, like for example image nr 34:
IMAGE NR  34
OUTPUT:
 [[3.6964852e-41 3.3333334e-01 3.3333334e-01 1.7476286e-22 1.0737801e-27
  3.3333334e-01 9.5417287e-21 2.6616346e-30 1.3799792e-08 4.3319380e-25]]
RAW OUTPUT:
 TensorBuffer{@0xaaaafa467990,tensor=xir::Tensor{name = activation, type = FLOAT32, shape = {1, 10}},location=HOST_VIRT,data=[(Virt=0xaaaafc474630, 40)]}
PRE OUTPUT SIZE:
 10
PREDICTION:	 1 	TRUTH:	 7

The resulting accuracy of these results is very poor.

Hi @Orchidaceae
The input and output of the model are in fixed-point format, and the fixed-point position needs to be considered when writing and reading data.src

Hi @Orchidaceae
Is this issue solved?

Hi @Orchidaceae
Since we haven't received your reply for a long time, we assume you have solved this issue and I'm going to close it. If you still have any questions, please feel free to reopen it. Thank you very much.


Update billing.md

The necessary subscription role is Owner not administrator.

Learn Build status updates of commit 226f33f:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

articles/active-directory-b2c/billing.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the learn.microsoft.com contributor guidesPost your question in the Learn support channel

@kengaderdus,
Can you review the proposed changes?
IMPORTANT: When the changes are ready for publication, add a #sign-off comment to signal that the PR is ready for the review team to merge.
#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

#sign-off

Learn Build status updates of commit 83777dc:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.

File
Status
Preview URL
Details

:x:Error

Details

[Error: GitBranchDeletedOrForcePushed]

Cannot sync git repo to specified commit because ref +refs/pull/102210/merge has been deleted or has been force pushed.
 If this is pull request, please ensure it is mergeable (no merge conflict) and open (not closed).

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Learn Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings or suggestions unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the learn.microsoft.com contributor guidesPost your question in the Learn support channel


WIP: initial access logs implementation

This PR adds "Access logs". There are logs emitted in three cases:

Premature failure (invalid src/dst, etc)
Connection start (debug level)
Connection end

These emit a variety of info about the result, as well as src/destination information.
This PR introduces a ton of changes outside of logging. The primary reason around this is refactoring to allow us to accurately and safely emit a log at the appropriate times (notably, at the end of the request). Things like ? short-circuit our ability to record.  So there is a lot of refactoring around that. Additionally, some RBAC logic is deferred until we have proper src/dst metadata, so we can record that in telemetry if RBAC is denied; I don't see any reason this would be an issue.
Example logs (not some are debug -- these are not visble by default)
2024-04-11T00:26:42.295630Z DEBUG access: connection opened src.addr=shell-6d8bcd654d-t88gp:59690 src.workload="shell" src.namespace="default" src.identity="spiffe://cluster.local/ns/default/sa/default" dst.addr=namespace-istio-waypoint-9ccc6c96-hthtk:15008 dst.hbone_addr="echo:80" direction="outbound"
2024-04-11T00:26:42.309245Z DEBUG access: connection opened src.addr=namespace-istio-waypoint-9ccc6c96-hthtk:36782 src.workload="namespace-istio-waypoint" src.namespace="default" src.identity="spiffe://cluster.local/ns/default/sa/namespace-istio-waypoint" dst.addr=echo-8659868fcc-z2mvm:80 dst.hbone_addr="echo-8659868fcc-z2mvm:80" dst.workload="echo" dst.namespace="echo" dst.identity="spiffe://cluster.local/ns/default/sa/default" direction="inbound"
2024-04-11T00:26:42.311513Z  INFO access: connection complete src.addr=shell-6d8bcd654d-t88gp:59690 src.workload="shell" src.namespace="default" src.identity="spiffe://cluster.local/ns/default/sa/default" dst.addr=namespace-istio-waypoint-9ccc6c96-hthtk:15008 dst.hbone_addr="echo:80" direction="outbound" bytes_sent=67 bytes_recv=490 duration="16ms"
2024-04-11T00:26:47.314293Z  INFO access: connection complete src.addr=namespace-istio-waypoint-9ccc6c96-hthtk:36782 src.workload="namespace-istio-waypoint" src.namespace="default" src.identity="spiffe://cluster.local/ns/default/sa/namespace-istio-waypoint" dst.addr=echo-8659868fcc-z2mvm:80 dst.hbone_addr="echo-8659868fcc-z2mvm:80" dst.workload="echo" dst.namespace="echo" dst.identity="spiffe://cluster.local/ns/default/sa/default" direction="inbound" bytes_sent=399 bytes_recv=144 duration="5005ms"

This depends on https://github.com/istio/ztunnel/pull/915 because doing it without that is a major annoyance. But if we need to keep it, I can update this PR to allow it.

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@howardjohn: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

test_ztunnel
ebde11acff99f292f2bf6fe3c59ade32c94fd527
link
true
/test test

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.


How to generate #current-cloud-backend build folder without successful push

Note: If your question is regarding the AWS Amplify Console service, please log it in the
official AWS Amplify Console forum

** Which Category is your question related to? **
AWS Amplify push process
** What AWS Services are you utilizing? **
AWS Amplify
** Provide additional details e.g. code snippets **
Is it possible to generate the build folder for #current-cloud-backend? It somehow got deleted and the push process is now trying to create all resources again and throwing errors. I also have a use case where I need to modify the backend build folder before pushing because of inconsistencies between backend and #current-cloud-backend however it is auto-generated each push. Is this overridable?

Maybe this will solve your problem:
amplify env pull ENV_NAME --restore && amplify push --yes

Unfortunately it doesn't. pull --restore doesn't seem to pull all of my resources. I am stuck in a state where amplify push is trying to create 2 tables that already exists. I was able to see the stacks json files and stacks in cloud formation-template.json within the #current-cloud-backend folder and so was trying to copy that into backend/build folder as that doesn't pick up on the 2 already created stacks.

Unfortunately it doesn't. pull --restore doesn't seem to pull all of my resources. I am stuck in a state where amplify push is trying to create 2 tables that already exists.

@ChrisFieldsII what resources are missing when you do an amplify env pull --restore. Are these the GraphQL API resources? Could you shed some light as to how you got to this state where those table got created?

Closing this issue due to lack of user response. Please feel free to commnet on this thread if you're still stuck and want to re-open this issue.

pull --restore does not create the build directory in #current-cloud-backend. This breaks ensureMissingStackMappings from working, as it reads from build/stacks.


Updated view width of floating panel in landscape

This is a little more natural than a 1/3 full panel.

Ill submit a new PR!


UI bug, Pie chart does not show long labels

As it can be seen in the above picture, if a spending has a very long label, it does not show all the labels including short labels. But it seems to show the data correctly

Have limited tags to 15 chars in new PR.
Do you mind trying to edit the tag to 15 chars on this set of data (if you still have them) and see whether the tags can be seen? To see if the limit of 15 chars work.
Couldn't run the program using the json file from yesterday.

The Pull request from choongyx, #241 fixed this bug


Word segmentation on strings without spaces

Could aho corasick (this module in particular) be used for such task ?  I am asking as I have not previously used said algorithm  and I didn't find an example showcasing this use.
For Example:
"hellotherehowareyou" --> "hello there how are you"  Now with dynamic programing you could do this in O(N^2) . . .  can we do better ? aho corasick seems like the way to go?

The Aho-Corasick algorithm allows you to find all occurrences of words from a list. So I think it may help in your problem.
However, I'm pretty sure it won't solve problem of ambiguous strings. What about: "thereare..." -> "there are..."/"the rear e..."?
You will get positions of all words from your list, but later you need to select one of many valid options. Although locating words is done in linear time, I guess the whole algorithm will have worse complexity.

Thank you for your answer.
The ambiguity will not be a major problem in my case . Can it also match substrings instead of just prefixes? Say I search for 'hi' in a list of strings. Can I get back the word "this" ? Right now If I search 'hi' I will get something like "hi","hill" ,"hip","hixxx" ect

Automaton searches for all words you add to it, in order to recognize "this" and "matching" you must add these words. Here is a sample program:
import ahocorasick

# build automaton
A = ahocorasick.Automaton()
for index, word in enumerate('hi this matching ching'.split()):
    A.add_word(word, (index, word))

A.make_automaton()

# search words
for item in A.iter('this matching'):
    print(item)

It prints:
(2, (0, 'hi'))
(3, (1, 'this'))
(10, (0, 'hi'))
(12, (2, 'matching'))
(12, (3, 'ching'))

As you see all words were found, even overlapped.

On the same aspect, any built-in option to avoid matching search words appearing as sub-words?

@matanster There's no such option. AFAIK automaton is not able to distinguish a proper word from its sub-words.