Sqflite compat pr

These changes are a result of my work on an implementation of sqlite CRDT implementation for Drift (see: https://github.com/JanezStupar/drift/tree/sqlite_crdt/drift_crdt).
My work relies on the work of @cachapa (sql_crdt and sqlite_crdt) libraries and Simon Binder's drift_sqflite library.
The changes are mostly related to expansion of the support for the sqflite API within sql_crdt of the scope that was necessary to get the drift_crdt working and some refactoring to avoid duplicate implementation.
Only new features are the migration within sql_crdt.dart and a "table of special queries" within base_crdt.dart.
I felt like those were appropriate places to dump the new features.
Let me know if you are open to accepting this contribution and what changes you feel need to be made to make them acceptable.

On the surface this seems like a lot of code to support one specific ORM.
The changes to database_api.dart alone are enough to give me pause, as they cascade into extra work for every downstream implementation - granted there are only two right now (sqlite_crdt and postgres_crdt), but the goal was to keep them as minimal as possible.
Don't get me wrong, Drift is a great project, and sql_crdt probably wouldn't have been possible without it but that doesn't mean we have to provide a complete implementation for their whole stack.
Maybe I'm too unfamiliar Drift as I never used it, but I wonder if a drift_crdt wouldn't be better implemented in the other way around: forking Drift and replacing sqflite with an extension (or fork) of sqlite_crdt implementing the rest of the sqflite API.
In any case I am thankful for your contribution but I don't think that it fits the goals for this project.
I'm closing this PR for now but I'm happy to continue the discussion here.

Thank you for your attention.
It is not about supporting one specific ORM, at least not exclusively. The changes I proposed have a lot more to do with compatibility with sqlite in broad and sqflite in the narrow sense.
Based on the work I did I am pretty sure you will run into these same issues down the road when your work gets more adoption.
With that out of the way. I agree with you upon reflection. The sqlite specific changes I forced into sql_crdt really don't make sense when looked at in the context of say postgres.
I am looking at this as I write this. I will come back with a different approach soon - one that is better fitted to the situation at hand and the scope you have in mind.

Thank you, but I do encourage you to discuss your proposed technical implementation before putting in the work as it saves us both a lot of time.
Can you expand on what you meant by "you will run into these same issues down the road"?

First off I think you misunderstood the nature of the rawUpdate, rawInsert... methods. They are not meant to bypass the CRDT at all. They are merely a reflection of the sqflite APIs (see: https://pub.dev/documentation/sqflite/latest/sqlite_api/Database-class.html - e.g. query vs rawQuery methods)
In fact the API implemented in the sqlite_crdt is misnamed. The query method has the signature of the rawQuery method from Sqflite. The reason why I added them is because the client I am using is using the raw SQL statement API instead of parameterized query like API.
However I agree that they should not be within sql_crdt library. But they cannot be implemented in sqlite_crdt - where they fit for sure, due to the private nature of the code within base_crdt.
Next issue is the isolation of the database connection to the base_crdt. While I can appreciate the sentiment I ran into an issue while implementing the transaction. Sqflite rolls back the transaction on error. sql_crdt implementation however does not and when trying to make sqlite_crdt compliant to the sqflite API this became an issue.
And since sql_crdt guards its implementation so closely I can not implement the required changes in superclass of TransactionCrdt. To do what is needed I would have to basically fork everything - to the extent of my knowledge.
And it seems to me these issues will crop up whenever somebody tries to include your libs within their existing project.
On the topic of migrations, I personally feel like that is a nice optional thing to have. Especially when it can be implemented in a database agnostic way. But that is a matter of personal preference. So by putting them inside the sql_crdt I though I was killing two birds with one stone.
But the real issue is again, that since the database connection cannot be accessed by the downstream code it is not possible for my own database library (whatever that is) to extend the behaviour of sqlite_crdt in a pragmatic fashion. And I have noted your suggestion to open another database connection. However it is my understanding that in sqflite it is assumed there is only one open connection at any given time to the database at this point in time. I could be wrong on this one.
So in a nutshell it would seem to me that the decision to keep so much of the base stuff private is really detrimental to brownfield scenarios and I am pretty sure that other developers will be affected in the future. I can understand you find this approach elegant and it might make sense for your use case, but it has already costed me a lot of time to this point (not enough for me to warrant implementing CRDT from scratch, mind you) and while it has facilitated me learning about CRDTs by proxy (which is good too) it doesn't have to be like that.
Have you considered using the '@protected' decorator instead?
p.s.: I did some prototyping to see whether I missed something when originally implementing these changes. And I realized that without making CRDT code more accessible I really don't have an alternative.

I see what you mean by the rawQuery methods. You're right that I misunderstood their intent.

In fact the API implemented in the sqlite_crdt is misnamed.

It's not misnamed, it's only incidental that it has the same name as a method from the sqflite package. As I said before this library is intended to be a generic implementation of CRDTs backed by an SQL engine.
Moreover sqlite_crdt is an implementation of CRDTs backed by a Sqlite engine - not a sqflite engine - otherwise I would've named it sqflite_crdt.
The fact that sqlite_crdt uses sqflite is an implementation detail. It may change in the future (however unlikely).

Next issue is the isolation of the database connection to the base_crdt. While I can appreciate the sentiment I ran into an issue while implementing the transaction. Sqflite rolls back the transaction on error. sql_crdt implementation however does not and when trying to make sqlite_crdt compliant to the sqflite API this became an issue.

Can you provide a bit more detail here? I'm under the impression that sqlite_crdt does roll back transactions. It uses the sqflite transaction() method which I believe handles cleanup.

On the topic of migrations, I personally feel like that is a nice optional thing to have. Especially when it can be implemented in a database agnostic way. But that is a matter of personal preference. So by putting them inside the sql_crdt I though I was killing two birds with one stone.

That is a good point, and one I might come back to later even if at this moment I'm still trending to no.
The only suggestion I would make is to separate such pieces of work into separate PRs since it makes PRs smaller, easier to review and to focus the discussion.
I actually mention this in the contribution guidelines.

But the real issue is again, that since the database connection cannot be accessed by the downstream code it is not possible for my own database library (whatever that is) to extend the behaviour of sqlite_crdt in a pragmatic fashion. And I have noted your suggestion to open another database connection. However it is my understanding that in sqflite it is assumed there is only one open connection at any given time to the database at this point in time. I could be wrong on this one.

Actually that was not a suggestion, just a statement of fact. I really don't think you should be accessing your CRDT databases using other tools unless it's read-only.
The correct way to access the underlying database is to update the version and use the onUpgrade callback to get a database object that's closer to bare metal.
Naturally this also opens the door to making invalid changes to the CRDT, but it does it in a limited space.
Maybe it would be helpful if you could provide examples of things you'd like to do with direct access to the underlying database that you can't do now.
To be clear I have migrated hundreds of thousands of users from brownfield applications to sqlite_crdt and postgres_crdt both in mobile apps and backend, and managed to do it with the exitsting API.
So I'm wondering if I'm missing some use case, or if perhaps the documentation isn't clear enough.


feat: added postgres docker container with postgis

Pull Request Test Coverage Report for Build 120

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 90.732%

Totals

Change from base Build 108:
0.0%

Covered Lines:
186

Relevant Lines:
205

💛  - Coveralls


Prefix B+-tree

Is your feature request related to a problem? Please describe.
Not necessarily, this is more of an optimization.
Describe the solution you'd like
I'd like to experiment with converting the B+-tree into a prefix B+-tree as described in this paper.
This has the potential to save quite a bit of space in internal nodes, depending on how similar the keys are.
Also, if pivot keys are very long, we may be able to avoid having them spill onto overflow pages.
Describe alternatives you've considered
This seems to be the paper that introduced this concept, making it a good starting point.
Additional context

NOTE: The paper talks about B*-trees, rather than B+-trees, but there doesn't seem to be too much of a difference, at least for our purposes.

This feature can be implemented by modifying PayloadManager::promote(), which is used to convert a cell read from an external node into a pivot cell suitable for posting to an internal node.
It needs to accept the cell immediately to the left of the cell being promoted.
Additionally, the cell being promoted must be detached before being passed to PayloadManager::promote(), since it will be overwritten.
PayloadManager::promote() should be modified to compare the keys, s and t, of the 2 cells.
The shortest key, k, should be determined, such that s < k <= t.
This is the shortest key that is necessary to direct searches toward the correct external node.
The cell should then be modified to hold k (the key size and overflow ID should be corrected).
Note that k will never be longer than either s or t, so we may be able to avoid copying an overflow chain.
At some later point, Tree::redistribute_cells() should be further changed to consider multiple candidates around the chosen split point, selecting the one that produces the shortest pivot key.
This technique is described in the paper, but should be easier to implement here due to the presence of the indirection vector.
I'm pretty sure that's it for the "simple prefix B-tree" part of the paper. At this point, we can consider implementing the "prefix B-tree".
It would likely produce further fan-out improvement at the cost of some more complexity (pivots may need to be recomputed after SMOs).

099ba10 added basic suffix truncation using Tree::make_pivot() and Tree::post_pivot(). Suffix truncation turned out to be less intrusive on the current design and was easy to implement.
I'm still considering different types of prefix compression.


"Invalid tag for list size" and "Invalid node"

Does anyone know what might be causing this exception all the time?

Is that critical? Are there tags that are not mapped, or is it something else, like a real bug?
On Baileys project this does not seem to happen, but I could not find the difference comparing both implementations.
Any help?

I think I solved the problem, soon if any one want I can send a pull request

Right now I can't reproduce the error, but that doesn't have to mean anything.
If you could make a pull request once you have fixed the error that would be great.
I've just released a few more minor changes to the code, updating the WhatsApp Web version among other things. Before that, I only got a message like this:
"1623777717.--0,["Cmd",{"type": "update"}]"
Maybe something has changed in the encoding of the messages?

HI JicuNull.. First of all, now that I have the opportunity, thank you for your project, it's great to have such implementation in Java, personally I think Java implementations are likely to be much more comprehensible and organized.
About the issue, sooner today I got the same message
"1623777717.--0,["Cmd",{"type": "update"}]"
but the bug I related in this issue is not related to that.
As you said, updating whatsapp web version fix the "message problem", and I haven't identified any change in the encoding of the messages yet.. So, I think the message response from whatsapp websocket is really just a matter with the whatsapp web version sent in login request.
About the issue, as soon as possible I sent the pull request.
But, briefly, among another minor changes, the most important is that I reimplemented the function readNode() in BinaryDecoder class:
`	private String readNode() {
int readByte = readByte() & 0xff;
int listSize = readListSize(readByte); // Needs to cast to unsigned byte or int
int descrTag = readByte() & 0xff;
if(descrTag == BinaryConstants.Tags.STREAM_END) {
System.err.println("Unexpected stream end");
}
	String descr = readString(descrTag);
	if(listSize == 0 || descr == null) {
		System.err.println("Invalid node");
	}

	String attrs = readAttributes((listSize - 1) >> 1);
	String[] content = null;

	if(listSize % 2 == 0) {
		int tag = readByte() & 0xff; // Needs cast to unsigned byte / int
		if(isListTag(tag)) {
			content = readList(tag);
		} else {
			/*
			 * Handles messages with the message description only
			 * The byte array gets further processed using protobuf
			 *
			 */

			String decodedString = "";
			byte [] decodedArray = null;

			switch(tag) {
				// Conversation messages
				case BinaryConstants.Tags.BINARY_8:
					byte[] bin8 = readBytes(readByte() & 0xff);
					decodedArray = bin8;
					break;
				// Image, video, extended and rarely conversation messages
				case BinaryConstants.Tags.BINARY_20:
					byte[] bin20 = readBytes(readInt20());
					decodedArray = bin20;
					break;
				// ?
				case BinaryConstants.Tags.BINARY_32:
					byte[] bin32 = readBytes(readInt(4));
					decodedArray = bin32;
					break;
				default:
					decodedString = readString(tag);
					break;
			}

			if (descr.equals("message") && decodedArray != null) {
				try {
					String base64Decoded = Base64.getEncoder().encodeToString(
							WebMessageInfo.parseFrom(decodedArray).toByteArray());

					content = new String[] {"\"" + base64Decoded + "\""};

				} catch (InvalidProtocolBufferException e) {
					e.printStackTrace();
				}

			} else {
				content = new String[] {"\"" + decodedString + "\""};

			}

		}
	}

	return "[\"" + descr + "\", " + attrs + ", " + Arrays.toString(content) + "]"; // Node
}

`

["Cmd",{"type":"update"}]
I am also getting this error even if I update WhatsApp version. 🤔

["Cmd",{"type":"update"}]
I am also getting this error even if I update WhatsApp version. 🤔

Are you sure you are specifying a commit version on Jitpack to use the latest updates? The last commit contains the updated WhatsApp web version:
implementation 'com.github.JicuNull:WhatsJava:f9b17488fc'
It's not about the app version on your phone.

Now, It works. Thanks.


Update AsyncWorkService Documentation

This PR adds the necessary code to autogenerate HTML documentation for AsyncWorkService.

Codecov Report

Merging #611 (e129f93) into development (39e6f11) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##           development     #611   +/-   ##
============================================
  Coverage        89.98%   89.98%
============================================
  Files              247      247
  Lines            10632    10629    -3
============================================
- Hits              9567     9565    -2
+ Misses            1065     1064    -1

Impacted Files
Coverage Δ

...icroservices/asyncworkservice/AsyncWorkService.hpp
100.00% <ø> (ø)

...mpendium/AsyncWorkService/src/AsyncWorkService.cpp
100.00% <ø> (ø)

...ium/DeclarativeServices/src/SCRBundleExtension.cpp
100.00% <ø> (ø)

...ativeServices/src/manager/ComponentManagerImpl.cpp
93.00% <ø> (ø)

...ativeServices/src/manager/ComponentManagerImpl.hpp
83.33% <ø> (ø)

...tiveServices/src/manager/ConfigurationNotifier.hpp
100.00% <ø> (ø)

...um/DeclarativeServices/src/SCRAsyncWorkService.cpp
78.46% <100.00%> (-0.65%)
:arrow_down:

...tiveServices/src/manager/ConfigurationNotifier.cpp
81.37% <100.00%> (ø)

...ramework/include/cppmicroservices/detail/Threads.h
100.00% <0.00%> (ø)

...include/cppmicroservices/detail/TrackedService.tpp
85.48% <0.00%> (+1.61%)
:arrow_up:


D3D12: Be explicit about all-resources texture barriers

This helps D3D12 implementations that would think any subresource specification other than the special "all resources" value comprises only a subset of those, regardless the detailed specification does cover all of them.

Comment was added, approved.

Thanks!


Make retries configurable; align constants

Using the same default values for the library and the
binary now. Tweaked the values a bit.

Hum, the tests failed because the test_quirks is getting flaky. Specifially the Twitter one:
> lychee fixtures/TEST_QUIRKS.txt
Issues found in 1 input(s). Find details below.

[/path/to/fixtures/TEST_QUIRKS.txt]:
✗ https://twitter.com/zarfeblong/status/1339742840142872577: Network error while trying to connect to an endpoint via reqwest

🔍 3 Total ✅ 2 OK 🚫 1 Error (HTTP:1)


Allow toggle support for sprint and sneak buttons on the gamepad but not for the keyboard

Is your feature request related to a problem? Please describe.
It's quite inconvenient that you can't set it so that sprint and sneak is a toggle button instead of requiring to hold them when using a gamepad. I understand that you can change that with the game accessibility options but that change will apply to both the keyboard and the gamepad when I'd only want the toggle option for the gamepad.
Describe the solution you'd like
Add a setting in the gamepad options menu that allows you to make the sprint and sneak buttons togglable for the gamepad but not for the keyboard.
Describe alternatives you've considered
N/A
Additional context
N/A

Yes please, it’s pretty annoying having to switch it in accessibility settings even though it switches seamlessly between controller and kb+m


Nested, or Stacked Icons

Feature Request
I'd like the ability to nest or stack icons on top of each other.
I think the nested icon should be smaller and appear in the same area that badges appear now if you were to add the badge class to an icon.
I'm picturing putting a plus icon over a user icon, or putting an upload icon on top of a file or folder icon.

@jacobbutton
The following plunkr should give you some direction on how to do this with clarity icons. As you can see in the example, clarity icons aren't exactly geared towards doing this sort of thing and the effect is only workable at bigger sizes.
Additionally, this opens a lot of issues in terms of supporting icons on top of one another. One of the reasons we went with an SVG-based icon framework is because we didn't want our product teams cobbling together pieces of icon fonts to create non-standard metaphors.
Between alerted and badged icons, 80% of our need for stacked or nested icons is taken care of. And the other portion can be accomplished with CSS.


ML 5.14 : Cross Validation (V)

Description
📌  Issues for Week 5
Welcome to 'ML' Team, good to see you here
:arrow_forward: This issue will helps readers in acquiring all the knowledge that one needs to know about ----
Cross Validation.
:red_circle: To get assigned to this issue, add your Batch Numbers mentioned in the spreadsheet of "Machine Learning", the approach one would follow and choice you prefer (Documentation, Audio, Video). You can go with all three or any number of options you're interested to work on.
✔ Domain : Machine Learning
:red_circle::yellow_circle: Points to Note :

The issues will be assigned on a first come first serve basis, 1 Issue == 1 PR.

"Issue Title" and "PR Title should be the same. Include issue number along with it.

Changes should be made inside the Machine Learning/Machine_Learning/Feature_Engineering_and_Performance_Metrices Branch.

Follow Contributing Guidelines & Code of Conduct before start Contributing.

This issue is only for 'GWOC' contributors of 'Machine Learning' domain.

:white_check_mark: To be Mentioned while taking the issue :

Full name
Batch Number
GitHub Profile Link
Which type of Contribution you want to make :

[ ] Video

Happy Contributing 🚀
All the best. Enjoy your open source journey ahead. 😎
Domain
Machine Learning
Type of Contribution
Video
Code of Conduct

[X] I follow Contributing Guidelines & Code of conduct of this project.

@abhisheks008
Hello, I am M. Ashish Reddy, I would like to work on this issue.
Batch Number: 2
Serial Number: 260
GitHub Profile Link: https://github.com/ashish-reddy-20-08
Approach: A detailed explanation with examples.

issue assigned @ashish-reddy-20-08

@abhisheks008 @Isha307 , can you please evaluate the PR


feat: Add Spark-compatible implementation of SchemaAdapterFactory

Which issue does this PR close?

Part of https://github.com/apache/datafusion-comet/issues/1040
Rationale for this change

We would like to use DataFusion's ParquetExec but need to provide a SchemaAdapterFactory that uses Spark-compatible casts.
What changes are included in this PR?

Implement cast_is_supported in native code. I filed https://github.com/apache/datafusion-comet/issues/1171 for removing the Scala copy and have it delegate to native.
Implement schema adapter and a minimal test
Add support for reading unsigned ints as signed ints

How are these changes tested?

@parthchandra @mbutrovich PTAL when you can

@kazuyukitanimura @viirya could I get a committer review pls


Backport 2.x: Run all.sh components in a subshell

Fixes #4909
This is the 2.x backport of the already merged #4765.
Once this is merged, backport https://github.com/ARMmbed/mbedtls/pull/5011.

The last pr-head run failed due to then-ongoing CI issues. https://jenkins-mbedtls.oss.arm.com/blue/organizations/jenkins/mbed-tls-pr-merge/detail/PR-4932-merge/12/pipeline/ passed and that's good enough to merge unless something else subsequently causes pr-merge to fail.


Splitting a centered view does not cause editor to take full width

Have a centered view
Split the editor
The editor is moved to the left side, but does not expand its width

Video

@bpasero this is happening within the editor part not at the workbench level. the editor part is properly full width.

Centered editor layout -> Isi

I can reproduce this. This works in stable so something broke in the meantime.
Easier repro: F1 > toggle centered editor mode. F1 > toggle centered editor mode -> layout broken
And yes we try to leave centered editor mode when there is more than 1 editor.
I will bisect now....

Seems like this commit https://github.com/microsoft/vscode/commit/0449a18b7bb1d8b3cf7138df63e03126ba47c21d broke centered layout. Thus also assigning to @joaomoreno

Good bisecting! 👌


Vickel: Watch wmcasino.cc

Vickel requests the watch of the watch_keyword wmcasino\.cc. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

wmcasino\.cc has been seen in 0 true positives, 0 false positives, and 0 NAAs.

Approved by Jeff Schaller in Charcoal HQ


Can I request a feature? Thank you!

Is your feature request related to a problem? Please describe.

Describe the solution you'd like:

Describe alternatives you've considered:

Additional context:

### Tasks
- [ ] https://github.com/microsoft/vscode-edge-devtools/issues/1647
- [ ] https://github.com/microsoft/vscode-edge-devtools/issues/1646

Closing as incomplete. Please feel free to add more information if you have any and we can re-open the issue.


What is the file format for individual datasets?

I'm looking to upload a system output (e.g. for the samsum corpus), but it was not clear to me what format the file should be in.

Maybe after someone selects a task or corpus, the expected file format could be displayed on the screen to give guidance about the format?

Yeah, that should be pretty straightforward. @pfliu-nlp if you have time could you add a description for each task in the SDK? I can then add an icon next to the form that displays a message on mouse hover or something like that. You can also provide a list of file formats for the users to choose from (tsv, csv, json, conll, etc).
If we want to add dataset level descriptions for the system outputs, then I think that needs to go in the datasets database? But maybe that's not necessary because currently, all the processors are at the task level (and if a dataset needs to be treated differently, we will add a specific processor for that dataset).
I guess this is also related to the discussion of having a standardized test set for explainaboard with standard sample ids #57 . We can also display a URL of that test set or a description of how users should generate the system outputs on this UI.

Hi, @lyuyangh Added following fields into our task schemas and feel free to make further edits based on new requirements:

description
supported_formats
https://github.com/neulab/ExplainaBoard/blob/8fac9976dabffe45e6086eecd21d72f97c602eb6/explainaboard/tasks.py#L56

One other thing that would be nice to have to the "new system" menu is when people search for a dataset, a link that says "Don't see what you want?" and links to information about how to add support of a new dataset to ExplainaBoard. For example, it could be at the bottom of the menu here:

The doc could be just a TBD page for now, but I'm happy to fill it in later after I work through the process myself.

This https://github.com/neulab/explainaboard_web/commit/d9a84f71a4d3b8f3bebabc1682b6ad83075c97c9 just went into production. A link to the system output documentation has been added to the tooltip. We can work on improving the UI to make this more obvious for the users in the future.


sst/examples/create-sst-rds: Add dynamic JSDoc database type to migration template

The problem
The migration.ejs.t file uses JSDoc to provide some autocompletion to the db type, but it doesn't make use of the auto-generated types in sql.generated.
This has led to some folks in the SST Discord (and myself) trying to set up TypeScript compilation for these files. But there is a simpler solution.
The solution
JSDoc supports imported types from other files. This makes for some really nice autocompletion without requiring a separate build step or having to worry about anything happening with runtime of the migration itself.

Oh that's interesting. Thanks for tracking down.

We are transition this repo to https://github.com/sst/v2 and are closing this issue for now. If it is still relevant please reopen it there - sorry for the trouble


Only calls imgSrcset()

Hi!
Thanks so much for your work on this plugin - been using it since the release of Kirby 3 and it's been great. With the new update it doesn't seem to be calling the picture tag snippet, I've tried all the file methods and all of them seem to only make an img srcset tag - no picture tag or sources.
I fiddled around a bit in the index.php, swapping out places you call the snippets and this seems to be the case - I got it to call the picture tag by snippet referenced on this line:
'plugin-srcset-img' => __DIR__ . '/snippets/srcset-img.php',
Hopefully you can figure out why.

there where .instead of , at the file method definitions. thanks for reporting this bug.


BayeSearchCV AttributeError: 'str' object has no attribute 'decode' when n_iter is set to a a high number

AttributeError: 'str' object has no attribute 'decode' on the line bsearch.fit() (fitting BayesSearchCV)
I am running a project on kaggle's notebooks with scipy version 1.6.3., sklearn version 0.23.2, and skopt version 0.8.1, and Python version 3.7.10 | packaged by conda-forge | (default, Feb 19 2021, 16:07:37)
[GCC 9.3.0]

The error can be replicated by increasing n_iter to a high number. I specifically got the error from setting n_iter = 500. I found a similar issue #981, where the person was also using skopt version 0.8.1, but that issue was closed. My issue differs in that the code works when n_iter is set low (such as 5), but raised an error when n_iter was set to 500.
If you'd like I can try to replicate the error because it takes about 5 hours to run. Unfortunately, I deleted the old output error message while trying to run fixes found on google.
program works with n_iter = 200
Reproduced error just now with n_iter = 500, it happens 4-5 hours after running fitting.

AttributeError                            Traceback (most recent call last)
 in
1 hyperParameterTuning = HyperParameterTuning(train_data, test_data, y_train_targets, modelChoices = ['XGBReg']) # setting n_iter too high gives attribute error:
----> 2 hyperParameterTuning.main()
 in main(self)
200                                                 }, n_iter = 500,
201                                                          cv = 3, verbose = 0, random_state = 0) #scoring = 'neg_root_mean_squared_error'
--> 202             bsearch.fit(self.train_data, self.y_train_targets) # fit_params = fit_params
203
204             self.tuneParams['XGBReg'] = (bsearch.best_params_)
/opt/conda/lib/python3.7/site-packages/skopt/searchcv.py in fit(self, X, y, groups, callback)
692                 optim_result = self._step(
693                     X, y, search_space, optimizer,
--> 694                     groups=groups, n_points=n_points_adjusted
695                 )
696                 n_iter -= n_points
/opt/conda/lib/python3.7/site-packages/skopt/searchcv.py in _step(self, X, y, search_space, optimizer, groups, n_points)
563
564         # get parameter values to evaluate
--> 565         params = optimizer.ask(n_points=n_points)
566
567         # convert parameters to python native types
/opt/conda/lib/python3.7/site-packages/skopt/optimizer/optimizer.py in ask(self, n_points, strategy)
415                 opt.tell(x, (y_lie, t_lie))
416             else:
--> 417                 opt.tell(x, y_lie)
418
419         self.cache = {(n_points, strategy): X}  # cache the result
/opt/conda/lib/python3.7/site-packages/skopt/optimizer/optimizer.py in tell(self, x, y, fit)
534             with warnings.catch_warnings():
535                 warnings.simplefilter("ignore")
--> 536                 est.fit(self.space.transform(self.Xi), self.yi)
537
538             if hasattr(self, "next_xs") and self.acq_func == "gp_hedge":
/opt/conda/lib/python3.7/site-packages/skopt/learning/gaussian_process/gpr.py in fit(self, X, y)
193                 noise_level=self.noise, noise_level_bounds="fixed"
194             )
--> 195         super(GaussianProcessRegressor, self).fit(X, y)
196
197         self.noise_ = None
/opt/conda/lib/python3.7/site-packages/sklearn/gaussian_process/gpr.py in fit(self, X, y)
232             optima = [(self.constrained_optimization(obj_func,
233                                                       self.kernel.theta,
--> 234                                                       self.kernel.bounds))]
235
236             # Additional runs are performed from log-uniform chosen initial
/opt/conda/lib/python3.7/site-packages/sklearn/gaussian_process/_gpr.py in _constrained_optimization(self, obj_func, initial_theta, bounds)
502                 obj_func, initial_theta, method="L-BFGS-B", jac=True,
503                 bounds=bounds)
--> 504             _check_optimize_result("lbfgs", opt_res)
505             theta_opt, func_min = opt_res.x, opt_res.fun
506         elif callable(self.optimizer):
/opt/conda/lib/python3.7/site-packages/sklearn/utils/optimize.py in _check_optimize_result(solver, result, max_iter, extra_warning_msg)
241                 "    https://scikit-learn.org/stable/modules/"
242                 "preprocessing.html"
--> 243             ).format(solver, result.status, result.message.decode("latin1"))
244             if extra_warning_msg is not None:
245                 warning_msg += "\n" + extra_warning_msg
AttributeError: 'str' object has no attribute 'decode'

According to #981 , scikit-learn<=0.23.2 is not compatible with scipy>=1.6.0. Since you do have Scikit-learn 0.23.2 and scipy 1.6.3, this is likely the problem.
You can downgrade scipy, or upgrade both scikit-learn and scikit-optimize to fix that. Since scikit-optimize 0.9 is not released yet, this involve installing from source. You can find instruction there: https://github.com/scikit-optimize/scikit-optimize/pull/988#issuecomment-889274620.
In case you don't already know that, it is tricky to use pip in a conda environment. The conda doc advise not to use conda install after using pip install, so I would recommend making a dedicated conda environment if you what to upgrade scikit-learn/optimize (see https://www.anaconda.com/blog/using-pip-in-a-conda-environment)

According to #981 , scikit-learn<=0.23.2 is not compatible with scipy>=1.6.0. Since you do have Scikit-learn 0.23.2 and scipy 1.6.3, this is likely the problem.
You can downgrade scipy, or upgrade both scikit-learn and scikit-optimize to fix that. Since scikit-optimize 0.9 is not released yet, this involve installing from source. You can find instruction there: #988 (comment).
In case you don't already know that, it is tricky to use pip in a conda environment. The conda doc advise not to use conda install after using pip install, so I would recommend making a dedicated conda environment if you what to upgrade scikit-learn/optimize (see https://www.anaconda.com/blog/using-pip-in-a-conda-environment)

I see, thank you very much for explaining this issue to me.

According to #981 , scikit-learn<=0.23.2 is not compatible with scipy>=1.6.0. Since you do have Scikit-learn 0.23.2 and scipy 1.6.3, this is likely the problem.
You can downgrade scipy, or upgrade both scikit-learn and scikit-optimize to fix that. Since scikit-optimize 0.9 is not released yet, this involve installing from source. You can find instruction there: #988 (comment).
In case you don't already know that, it is tricky to use pip in a conda environment. The conda doc advise not to use conda install after using pip install, so I would recommend making a dedicated conda environment if you what to upgrade scikit-learn/optimize (see https://www.anaconda.com/blog/using-pip-in-a-conda-environment)

Upgrading sklearn will cause another issue (#1006).

@voloddia Using scikit-learn==0.24.2 and scikit-optimize==0.9.dev0 works fine.
I couldn't reproduce #1006 with combinations of scikit-learn and scikit-optimize version. I think it might have been a problem while 0.9 was being updated to scikit-learn 0.24 but this was not complete.
Anyway, right now scikit-optimize<0.9 requires scikit-learn<0.24. You need to install scikit-optimize>=0.9 from source to use scikit-learn>=0.24, but using the updated scikit-optimize works correclty.


Add a contributing guide

Something simple to encourage community members to contribute.

I can take this with #5 on k3ai-sites and make a proposal.

Added Contributing guide to docs in #34


kore/Att.hashCode fix

Converted from Att(att: Map[(String, Class[_]), Any]) to Att(att: Map[(String, String), Any])
Related to #377
Further performance optimization possible.
Benchmark results
I did this simple benchmark:

Run clean kompile kevm 10 times
Compute average time for the line "Total" in kompile logs
Do both for old Att and new Att

Results:

Old att: 30.185 s average
New att: 31.268 s average

Further optimization ideas
It is possible to improve performance by refactoring call sites. Operations over (String,Class) can be replaces with operations over respective (String,String), with former class argument being constant.
For example:
https://github.com/kframework/k/blob/504ecde7cc489d5ffe5a25291dc915d14151308b/kore/src/main/scala/org/kframework/Strategy.scala#L26
Can be replaced with
var someField = classOf[Sort].getName
Att.empty.add(someFielde, Sorts.KItem)

This should compensate for any potential performance decrease.
Do you want me to do this?

@dwightguth ready for review

@dwightguth Sorry, benchmark was clean kompile kevm, not kprove.

Alright. Can I see a krun of one of the blockchain tests as well then?

@dwightguth Comments addressed, to the extent possible. I didn't install the infrastructure to run kevm tests. If I implement the other performance optimization I proposed in the description, I don't see any way performance could decrease. The number of execution steps will strictly decrease in all cases, because we get rid of unnecessary class->string/string->class conversions.

Benchmark results for running 10 times this command:
./kevm test tests/ethereum-tests/BlockchainTests/bcBlockGasLimitTest/BlockGasLimit2p63m1.json

K-master:
real	0m4.327s
user	0m1.406s
sys	0m2.891s

this PR:
real	0m4.312s
user	0m1.344s
sys	0m2.859s

Can I see kevm run, not kevm test? I'm somewhat confident that kevm test isn't going to run krun at all.

And as I said I want to see it with kserver running.

Here are timing numbers I was able to collect for KEVM vm tests:
* fe5d30d5 - !!! .build/k: K PR-att                                                           - Everett Hildenbrandt (HEAD -> tested-time-vm-java-ocaml, testing)
| 0 0.11s 0.05s 0.06s 16008KB bash -c make clean
| 0 65.37s 164.13s 9.85s 639196KB bash -c make deps        -B
| 0 101.26s 451.17s 11.55s 2776584KB bash -c make build       -B -j6
| 0 3.63s 2.94s 0.64s 130108KB bash -c make split-tests -B
| 0 0.00s 0.00s 0.00s 3400KB bash -c rm -rf .build/logs/tests.log
| 0 0.07s 0.06s 0.02s 26416KB bash -c export KSERVER_SOCKET=kserver.ocaml.socket && .build/k/k-distribution/target/release/k/bin/spawn-kserver kserver.ocaml.log
| 0 101.29s 142.86s 34.81s 319800KB bash -c export KSERVER_SOCKET=kserver.ocaml.socket && make TEST_CONCRETE_BACKEND=ocaml test-vm -j12 -k && .build/k/k-distribution/target/release/k/bin/stop-kserver || true
|
* e4fdb6eb - !!! .build/k: K master                                                           - Everett Hildenbrandt
| 0 0.12s 0.06s 0.05s 15968KB bash -c make clean
| 0 66.15s 163.98s 10.16s 628760KB bash -c make deps        -B
| 0 102.27s 462.88s 11.26s 2926428KB bash -c make build       -B -j6
| 0 3.64s 3.03s 0.56s 129712KB bash -c make split-tests -B
| 0 0.00s 0.00s 0.00s 3468KB bash -c rm -rf .build/logs/tests.log
| 0 0.09s 0.06s 0.03s 26452KB bash -c export KSERVER_SOCKET=kserver.ocaml.socket && .build/k/k-distribution/target/release/k/bin/spawn-kserver kserver.ocaml.log
| 0 100.33s 142.53s 33.89s 323296KB bash -c export KSERVER_SOCKET=kserver.ocaml.socket && make TEST_CONCRETE_BACKEND=ocaml test-vm -j12 -k && .build/k/k-distribution/target/release/k/bin/stop-kserver || true
|
* 75bfd4e8 - !!! kevm: run ocaml backend tests through krun frontend                          - Everett Hildenbrandt
| 0 0.07s 0.06s 0.00s 16040KB bash -c make clean
| 0 52.79s 77.25s 8.33s 470440KB bash -c make deps        -B
| 0 98.22s 458.54s 11.93s 3062732KB bash -c make build       -B -j6
| 0 3.62s 2.98s 0.59s 129612KB bash -c make split-tests -B
| 0 0.00s 0.00s 0.00s 3448KB bash -c rm -rf .build/logs/tests.log
| 0 0.08s 0.06s 0.02s 26404KB bash -c export KSERVER_SOCKET=kserver.ocaml.socket && .build/k/k-distribution/target/release/k/bin/spawn-kserver kserver.ocaml.log
| 0 92.30s 142.04s 33.94s 306060KB bash -c export KSERVER_SOCKET=kserver.ocaml.socket && make TEST_CONCRETE_BACKEND=ocaml test-vm -j12 -k && .build/k/k-distribution/target/release/k/bin/stop-kserver || true
|


Added NULL check in hipEventSynchronize

Added null check in hipEventSynchronize
Minor correction in descriptions

I have messed up my branch hence closing this PR will raise new PR shortly.


Add a deno pet

Added a lil dinosaur pet to represent Deno.

Erm. This is amazing. Well done on the PR. I'll take some time to try this out with a few different scenarios then give a +1

Codecov Report
Attention: Patch coverage is 45.45455% with 6 lines in your changes are missing coverage. Please review.

Project coverage is 17.07%. Comparing base (3c3a51e) to head (0fa85cb).

Files
Patch %
Lines

src/panel/pets/deno.ts
50.00%
4 Missing :warning:

src/panel/pets.ts
33.33%
2 Missing :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #567      +/-   ##
==========================================
+ Coverage   16.80%   17.07%   +0.26%
==========================================
  Files          22       23       +1
  Lines        1166     1177      +11
  Branches      158      158
==========================================
+ Hits          196      201       +5
- Misses        964      970       +6
  Partials        6        6

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


upgrade poison dependency

I would like to upgrade poison dependency, I also made cleanup in tests.
next step will be updating plug for it.

The problem is that some libraries I'm using already migrated to recent version and I have conflicts. Time for forward, I verified that code is working well on it. I would keep deps updated to recent versions.

The ~> 1.4 or ~> 2.0 requirement would handle your issue. I'm in a similar position, so the above allows for 2.0 where it is possible, but doesn't cut off those who still have hard requirements for Poison 1.x off.

I've updated the poison dependency in f1fd200. It's a more liberal requirement which is still backwards compatible.


Only use /etc/crowbar/network.json for adjusting the network proposal

Don't touch the template in /opt/dell at all. This should be especially helpful
for the QA clouds as their customized configs would no longer need to contain
the whole template (including the "deployment" section)  but just the values
that are different from the template.

Here is the (sofar) successful build on qa2 hardware: https://ci.suse.de/view/Cloud/view/Builds/job/cloud-mkphyscloud-qa/176/console

rebased the patch on top of the latest changes.

Another quick test was successful: https://ci.suse.de/view/Cloud/view/Builds/job/cloud-mkphyscloud-qa/183/consoleFull

Tested multiple times on qa2 hw with modified config without sections deployment, id and description, worked without issues.
+1

+1

The gating has failed again and it was not due to this PR, can we merge this already? I have tested it multiple times on our hw.


Upgrade to tokio 0.3

If https://github.com/hyperium/hyper/issues/2352 is finished (and h2 upgraded to tokio 0.3), this library can also be upgraded to tokio 0.3 if I am not mistaking

It's not yet in crates.io! Yeah, will upgrade, and sure hoping we'll be in stable 1.0 tokio soon, so no more need to keep up :)

Superseded by https://github.com/pimeys/a2/issues/51 @pimeys @Jasperav

Closed because it was superseded and this has been stagnant for over 1 year


[BUG] latest.json is not being added

Since these days, latest.json is not being added to the release of my project, but two days ago it was working fine, some changes done on release v.0.5.17 on tauri-action project could break this.
with latest.json file: https://github.com/eythaann/Seelen-UI/releases/tag/v2.0.9
no latest.json present: https://github.com/eythaann/Seelen-UI/releases/tag/v2.0.10

Closing in favor of #975


Remove custom change detection code from incremental Java compile

With https://github.com/gradle/gradle/pull/8499, the custom alignment code for incremental Java compilation should not be necessary any more and the normal incremental information can be used.
Maybe we should dogfood after we also support better incremental task inputs.

The Android Gradle Plugin currently extends JavaCompile with quite some logic which means that moving to InputChanges would be a breaking change for them.

FTR, it also has its own way to specify additional sources, so our workarounds for getSource() could be problematic, too.

@wolfs created an issue on the Android issue tracker: https://issuetracker.google.com/issues/133354240

\o/ https://issuetracker.google.com/issues/133354240#comment9:

This should be released in 3.6.0-alpha05.
Fixed with ag/I1a1a636546a07e38d5a220fdcb2b09b0c376bd2d.

I think this means we can break JavaCompile for extension in 6.0.

I we break JavaCompile in 6.0, then AGP 3.5 will not work anymore with Gradle 6.0. So no stable version of AGP would work with Gradle 6.0 when it comes out, right? I don't think we can do this...

6.0 GA is planned for end of August now. When is AGP 3.6 planned?

Kotlin extends AbstractCompile only:
https://github.com/JetBrains/kotlin/blob/master/libraries/tools/kotlin-gradle-plugin/src/main/kotlin/org/jetbrains/kotlin/gradle/tasks/Tasks.kt#L267-L272
And it doesn't mess with getSource():
https://github.com/JetBrains/kotlin/blob/master/libraries/tools/kotlin-gradle-plugin/src/main/kotlin/org/jetbrains/kotlin/gradle/tasks/Tasks.kt#L83-L85

Using InputChanges in JavaCompile is now tracked as part of #9817 and is a requirement for removing the custom change detection code.


Unresolved conflict in ModelMakeCommand

While working with the master branch I noticed unresolved conflicts that cause artisan to fail during the installation.

Duplicate of #19939


Excessive CPU usage

When I use WebRTC for audio and video communication, the audiodg.exe program takes up too much CPU, causing the program to crash. What could be the reason? The picture above shows my system version, CPU, memory and other information. Looking forward to hearing from you!

It will bee the sipsorcery video encoding/decoding that's killing your CPU. It's meant for demos and proof of concepts and not production. Getting an efficient video pipeline with dotnet is a BIG job all by itself.
Some pragmatic measures you can take are to reduce teh frame rate and/or the resolution. I've found 640 x 480 @ 30fps is normally ok, albeit an i7, or greater, with 16GB of RAM. With your i5 and 8GB your mileage may vary.


Docs rearrangement + enhancement

This PR does two main things:

Fixes a couple of links in daisy_workflows/README.md, and
Addresses most of the feedback I wrote down while originally exploring the Daisy documentation.

There are also a few small typo/wording corrections here and there.
I also moved the glossary from the daisy/README.md to the workflow config spec, which probably should have been done in the first place, when I split the spec into its own document.

@zmarano There actually is a link -- it just tiny and easy to miss. The first paragraph ends in "Documentation can be found here."
I'll make a new "Docs" heading to make it more obvious.


how long do the token stay available?

I had some issues with tokens disappearing and am not sure what is the default behavior.

Have a look at the config.php file to see the default configuration options - you can always override them in your own app depending on what you need.
By default a token stays valid for 1440 minutes - one day from the time they are created. They are deleted when they expire, but you can change that if you like.
If you've got more detailed information about what is going wrong in your project along with some sample code to demonstrate the issue I'll be more than happy to take a look.

I figured out one has to publish the config file first (whops)


[firebase_dynamic_links] allow custom paths for short URLs

Is your feature request related to a problem? Please describe.
I can create short dynamic links with my own custom domain no problem. They look like "https://app.myapp.com/2dBY" for example and link a user within my app. Problem: "2dBY" doesnt look cool. Within firebase console, custom paths are possible, so my link could look like "https://app.myapp.com/Chris123" for example. In flutter, I did not find a way to do this.
Describe the solution you'd like
It would be cool if "DynamicLinkParameters class" could be extended with a new parameter called "urlSuffix" for example. This suffix would be used for short dynamic links, so that even within flutter I could generate userfriendly short links like "https://app.myapp.com/Chris123"
Additional context
Guess (and hope) that no more info is required :)

I think it's a really great feature to add. However, I realized that there are a lot of things to consider. When building a new dynamic link using the api, it's trying creating a new one. What if the custom path already exists? And how do we make sure that the new and existing paths have the same link?

If the program is using usernames, which are unique(the app has to deal with this), and the same user generates a new dynamic link again, why not just overwrite the old link . The custon path feature would  be very useful it'll allow user to generate better looking links to get other users to signup eg. instagram/someusername


exercise memcached in test

What this PR does:
exercise memcached features in chart test

https://github.com/cortexproject/cortex-helm-chart/blob/5061ce897f279b9c55ead01b6a9cb163d8f74d6c/ci/test-values.yaml#L54-L56
Test is failing because this is invalid. ct uses a random release name, e.g., cortex-wou93c6h5x-memberlist that doesn't match this config.

https://github.com/cortexproject/cortex-helm-chart/blob/5061ce897f279b9c55ead01b6a9cb163d8f74d6c/ci/test-values.yaml#L54-L56
Test is failing because this is invalid. ct uses a random release name, e.g., cortex-wou93c6h5x-memberlist that doesn't match this config.

Is there no way to disable the random suffix for releases? According to the code: https://github.com/helm/chart-testing/blob/main/pkg/chart/chart.go#L202 there is no way to disable this behaviour

The best solution is probably to have the chart build a template for the service name with {{.Release.Name}}, like it does for the memcached config. This way chart users aren't required to construct the value themselves.

The best solution is probably to have the chart build a template for the service name with {{.Release.Name}}, like it does for the memcached config. This way chart users aren't required to construct the value themselves.

I am afraid I can't quite follow. .Release.Name is already used across the cortex-helm-chart. The problem is that ct randomizes the release name at runtime and there is no real way of knowing that beforehand, unless we hardcode the headless memberlist service name so that it doesn't change anymore.

I'm saying the fix is to define
 54 to 56 in 5061ce8

 memberlist:
   join_members:
   - "{{.Release.Name}}-memberlist.cortex"

in a context that evaluates templates as opposed to in the ci values as it is now https://github.com/cortexproject/cortex-helm-chart/blob/5061ce897f279b9c55ead01b6a9cb163d8f74d6c/ci/test-values.yaml#L54-L56

I'm saying the fix is to define
 54 to 56 in 5061ce8

 memberlist:
   join_members:
   - "{{.Release.Name}}-memberlist.cortex"

in a context that evaluates templates as opposed to in the ci values as it is now https://github.com/cortexproject/cortex-helm-chart/blob/5061ce897f279b9c55ead01b6a9cb163d8f74d6c/ci/test-values.yaml#L54-L56

Oh yeah, of cause. I totally forget about that possibility. I like it

I ran
kubectl create namespace cortex
ct install --config ct.yaml --namespace cortex

against my kubernetes (1.20.9) cluster and there a quite a lot of issues. However it passes and finishes "successfull".
I ran the same again using
helm install cortex . --namespace=cortex --create-namespace -f ci/test-values.yaml

got the same errors but everything went "online" eventually, which brings me to my point. It just takes a while to get everything up and running, especially if we publish endpoints which are not ready yet. @kd7lxl Could it be that we just have to increase the timeout?
By the way the HPA doesn't really work for the ci since the querier doesn't have any resource/requests applied

invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: missing request for cpu
Warning  FailedGetResourceMetric       3m26s (x40 over 13m)  horizontal-pod-autoscaler  failed to get cpu utilization: missing request for cpu

I can't find anything else

I ran
kubectl create namespace cortex
ct install --config ct.yaml --namespace cortex

against my kubernetes (1.20.9) cluster and there a quite a lot of issues. However it passes and finishes "successfully".
I ran the same again using
helm install cortex . --namespace=cortex --create-namespace -f ci/test-values.yaml

got the same errors but everything went "online" eventually, which brings me to my point. It just takes a while to get everything up and running, especially if we don't publish endpoints (publishNotReadyAddresse only for the distributor, query-frontend, alertmanager) which are not ready yet. @kd7lxl Could it be that we just have to increase the timeout?
I also checked the cortex status page and everything was healthy ¯\_(ツ)_/¯

The problem I assumed we needed to solve was this, from the CI logs:
ts=2021-08-17T14:40:34.70607837Z caller=memberlist_logger.go:74 level=warn msg="Failed to resolve cortex-memberlist.cortex: lookup cortex-memberlist.cortex on <IP_ADDRESS>:53: no such host"

But now with your change it looks like this, so I guess it's a problem with the memberlist endpoints not being ready immediately as opposed to the wrong hostname:
ts=2021-08-17T14:40:34.70607837Z caller=memberlist_logger.go:74 level=warn msg="Failed to resolve cortex-3p6brmjpl5-memberlist.cortex: lookup cortex-3p6brmjpl5-memberlist.cortex on <IP_ADDRESS>:53: no such host"

This seems to clear up eventually. 👍

By the way the HPA doesn't really work for the ci since the querier doesn't have any resource/requests applied

invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: missing request for cpu
Warning  FailedGetResourceMetric       3m26s (x40 over 13m)  horizontal-pod-autoscaler  failed to get cpu utilization: missing request for cpu

Also, the metrics-server is not deployed in Kind. That's okay -- we don't need the autoscaler to scale in the test, but at least submitting the resource to k8s allows it to be checked for compliance.

I can't find anything else
Noticeable Job output:

W0817 14:40:02.390348    7312 warnings.go:67] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W0817 14:40:02.392074    7312 warnings.go:67] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W0817 14:40:02.561890    7312 warnings.go:67] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W0817 14:40:02.563516    7312 warnings.go:67] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
Error: timed out waiting for the condition

We'll want to do something like this to handle the changing apiVersion: https://github.com/helm/helm/blob/d4cc130fa90027e8c1c69ad2a06c6dee4c31d29d/pkg/chartutil/create.go#L223-L229

I can't find anything else
Noticeable Job output:

W0817 14:40:02.390348    7312 warnings.go:67] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W0817 14:40:02.392074    7312 warnings.go:67] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W0817 14:40:02.561890    7312 warnings.go:67] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W0817 14:40:02.563516    7312 warnings.go:67] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
Error: timed out waiting for the condition

We'll want to do something like this to handle the changing apiVersion: https://github.com/helm/helm/blob/d4cc130fa90027e8c1c69ad2a06c6dee4c31d29d/pkg/chartutil/create.go#L223-L229

Yeah we can control that with capabilities but I actually meant the bold marked line. I think the current problem is that it simply doesn't get ready in time. Which does make sense considering the extra pods, and that it aborts after roughly 10minutes. We can adjust the timeout in ct.yaml and increase it to 15min.
helm-extra-args: --timeout 900s

@kd7lxl  Alright found the culprit. Some memcached Pods can't be scheduler, not enough resources. Memcached comes with default requests and limits.

Warning  FailedScheduling  29s (x17 over 15m)  default-scheduler  0/1 nodes are available: 1 Insufficient cpu.


Construct binary tree from preorder and inorder traversal

Coverage remained the same at 100.0% when pulling b069bb356bb79b2f9313d64427d37bb7dcbe5ec5 on construct-binary-tree-from-preorder-and-inorder-traversal into 0728dbfcbd75b9f6e23befae4852795f1c947959 on master.


Columns header is printed twice Fixed

Fixes #3264
describe the change in one sentence
The update fixes an issue where column headers were printed twice. A printedHeader variable was added to ensure headers are only printed once by tracking if the header has already been output.
How to use
Reviewers should run the affected commands to confirm that column headers are now printed correctly and only once.
Testing done
Commands tested:

$ ./kubectl-gadget run snapshot_process:latest
$ sudo -E gadgetctl run snapshot_process:latest
$ sudo -E ig run snapshot_process:latest

Hi @andoriyaprashant, welcome to the community!
Thanks for your contribution. I noticed there are still a couple of situations where we still have issues with the header. Could you please check if we can address them also in this PR?

Yes sir, I will try to make the changes you mentioned 👍🏻

@andoriyaprashant do you have any plans to complete it?

Superseded by #3641.


MB-13773 Pre-work to consolidate some definitions before re-using them for the actual ticket work

Jira ticket for this change
Summary
Is there anything you would like reviewers to give additional scrutiny?
this article explains more about the approach used.
Setup to Run Your Code

💻 You will need to use three separate terminals to test this locally.
Terminal 1
Start the Storybook locally.
make storybook

Terminal 2
Start the UI locally.
make client_run

Terminal 3
Start the Go server locally.
make server_run

Additional steps

Access the
Login as a

Verification Steps for Author
These are to be checked by the author.

[ ] Tested in the Experimental environment (for changes to containers, app startup, or connection to data stores)
[ ] Request review from a member of a different team.

Verification Steps for Reviewers
These are to be checked by a reviewer.

Frontend

[ ] User facing changes have been reviewed by design.
[ ] There are no aXe warnings for UI.
[ ] This works in Supported Browsers and their phone views (Chrome, Firefox, IE, Edge).
[ ] There are no new console errors in the browser devtools
[ ] There are no new console errors in the test output
[ ] If this PR adds a new component to Storybook, it ensures the component is fully responsive, OR if it is intentionally not, a wrapping div using the officeApp class or custom min-width styling is used to hide any states the would not be visible to the user.

Backend

[ ] Code follows the guidelines for Logging
[ ] The requirements listed in Querying the Database Safely have been satisfied.

Screenshots

Warnings

:warning:

Files located in legacy directories (src/shared or src/scenes) have
been edited. Are you sure you don’t want to also relocate them to the new file structure?
View the frontend file org ADR for more information

Messages

:book:
:link: MB-13773

  Generated by :no_entry_sign: dangerJS against 97d42aa5489d4abeb68d30d4dd6c67beafde2cb0


Fixed typo

@dbenner : Thanks for your contribution! The author(s) have been notified to review your proposed change.


resources.resources paging error

The resources.resources Swagger file has an error
The latest codegen enforce the check for paging extension to make sure there has a property in the response to match the property name specified in extension.
This file presents an error on Travis.

ResourceProviderOperationDetailListResult
ResourceProviderOperationDetails_List

@vivsriaus - Please send a PR for deleting the provider operation paths from the swagger spec as they rightly belong to Authorization.

@amarzavery Here you go: https://github.com/Azure/azure-rest-api-specs/pull/357


Is there a way to ensure_installed non-LSPs?

If I wanted to automatically install DAP, Linters, and Formatters?
Listing them in the ensure_installed won't work.

Need to ensure install formatters.

Need to ensure install formatters.

That's not a feature yet is it?

you can try this https://github.com/WhoIsSethDaniel/mason-tool-installer.nvim

@nshen this is perfect, thank you!

you can try this https://github.com/WhoIsSethDaniel/mason-tool-installer.nvim

It will be great if no plugin is required. The last update time of the linked project is in January. @hahuang65 Maybe it would be great to keep this issue open.

Well this is my solution for the moment.
vim.api.nvim_create_autocmd('VimEnter', {
  pattern = '*',
  callback = function()
    local masonpath = vim.fn.stdpath 'data' .. '/mason/bin/'

    for _, format in pairs(ide.formattings) do
      if not vim.loop.fs_access(masonpath .. format, 'X') then
        if type(format) == 'string' then
          vim.cmd('MasonInstall ' .. format)
        else
          vim.cmd('MasonInstall ' .. format.name)
        end
      end
    end
  end,
})

@maxmx03 You probably want to use Mason's Lua API instead, something like so:
local registry = require("mason-registry")

for _, pkg_name in ipairs { "stylua", "prettier" } do
  local ok, pkg = pcall(registry.get_package, pkg_name)
  if ok then
    if not pkg:is_installed() then
       pkg:install()
    end
  end
end

Then I will go for this method instead. I tried the linked plugin and I have to use names like ltex-ls there, which causes issues like this: #168.

I'm not sure I'm following. ensure_installed in mason-lspconfig.nvim uses nvim-lspconfig server names whereas mason-tool-installer uses mason.nvim package names. If you use mason-tool-installer you don't really have any reason to also use the ensure_installed feature in mason-lspconfig.nvim.

Does it check for updates btw?

Nope but you can customize it further with pkg:check_new_version() (mason-tool-installer does this already)

ensure_installed in mason-lspconfig.nvim uses nvim-lspconfig server names whereas mason-tool-installer uses mason.nvim package names.

Sorry for commenting on a closed issue, but I found your comment and it seems to conflict with the names in nvim-lspconfig, namely the bashls server per the screenshot below.

Can you provide a link to the correct LSP server names? :pray:
Thank you!

@jc00ke https://github.com/williamboman/mason.nvim/issues/1045#issuecomment-1454013966

@williamboman yep, that was it. I now remember either you telling me that or figuring it out, and then I went and fixed it on a branch... only to be totally and utterly forgotten :rofl:
I had all sorts of stuff wrong, which I mostly fixed in https://github.com/jc00ke/dotfiles/commit/4db691365cfbbe0f88c3d8417c1af253e412522d


Feat/protocol kit/get modules paginated

What it solves
Resolves #
How this PR fixes it

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.9%) to 74.502%

Totals

Change from base Build<PHONE_NUMBER>:
0.9%

Covered Lines:
481

Relevant Lines:
597

💛  - Coveralls


Issue in deploying it in K3S cluster

I'm not quite sure how to write the yaml file to deploy this docker image correctly.

Hello @JokerMirage
Thank you for your inquiry. However, it would be more appropriate to ask your question on https://stackoverflow.com/.


Appstudio update test-component-pac-ocse

Pipelines as Code configuration proposal

Pipelines as Code CI/test-component-pac-ocse-on-pull-request has successfully validated your commit.

StatusDurationName

✅ Succeeded
7 seconds
init

✅ Succeeded
16 seconds
clone-repository

✅ Succeeded
44 seconds
build-container

✅ Succeeded
20 seconds
deprecated-base-image-check

✅ Succeeded
18 seconds
sanity-inspect-image

✅ Succeeded
3 minutes
clair-scan

✅ Succeeded
49 seconds
clamav-scan

✅ Succeeded
15 seconds
sbom-json-check

✅ Succeeded
8 seconds
sanity-optional-label-check

✅ Succeeded
7 seconds
sanity-label-check

✅ Succeeded
8 seconds
show-summary

Pipelines as Code CI/test-component-pac-ocse-on-pull-request has successfully validated your commit.

StatusDurationName

✅ Succeeded
7 seconds
init

✅ Succeeded
16 seconds
clone-repository

✅ Succeeded
44 seconds
build-container

✅ Succeeded
20 seconds
deprecated-base-image-check

✅ Succeeded
18 seconds
sanity-inspect-image

✅ Succeeded
3 minutes
clair-scan

✅ Succeeded
49 seconds
clamav-scan

✅ Succeeded
15 seconds
sbom-json-check

✅ Succeeded
7 seconds
sanity-label-check

✅ Succeeded
8 seconds
sanity-optional-label-check

✅ Succeeded
8 seconds
show-summary

Pipelines as Code CI/test-component-pac-ocse-on-pull-request has successfully validated your commit.

StatusDurationName

✅ Succeeded
10 seconds
init

✅ Succeeded
15 seconds
clone-repository

✅ Succeeded
40 seconds
build-container

✅ Succeeded
13 seconds
deprecated-base-image-check

✅ Succeeded
14 seconds
sanity-inspect-image

✅ Succeeded
31 seconds
clamav-scan

✅ Succeeded
9 seconds
clair-scan

✅ Succeeded
10 seconds
sbom-json-check

✅ Succeeded
19 seconds
sanity-optional-label-check

✅ Succeeded
18 seconds
sanity-label-check

✅ Succeeded
6 seconds
show-summary


Add tools list for web development

Add the list of the tools with link useful for the developers for their day to day work

@shrutikapoor08 I will add a section for this

Thankyou!

PR raised https://github.com/shrutikapoor08/Learn-Web-Dvelopment-Checklist/pull/15


Unification with \or patterns

Fixes #699
Reviewer checklist

[ ] Test coverage: stack test --coverage
[ ] Public API documentation: stack haddock

Does this have any changes that aren't in #1299?

Yes, a few lines

It's hard to check the delta between this and #1299 so I'll postpone the review.

Thanks to @virgil-serbanuta 's #1299 this pr is no longer needed


feat: support iso language codes

The library currently indexes by lowercase BCP 47 locale names. It'd be great if one could use ISO-639-1 codes like en or de too! A region information is not common to be used in many languages.

Great, thank you <3 That would be "dargmuesli" too.

Hey @Kimi-Gao, the required Travis status check never finishes and blocks everything. I have no permission to remove it from being required. Could you please do so?
I'd migrate to GitHub Actions then.

@dargmuesli Sure, it has been removed. In addition, I grant you the permission as the admin as well.

Thank you! :)


Option to disable confirmation prompt

I am trying to automate pcsync pushAll as a workaround while pcwatch is not working on Windows.
So far I've managed to call pcsync pushAll automatically, but I still need to confirm the upload with "y" every time.
It would be great to provide an option to skip the confirmation like -y or --headless.

If you agree with that feature and tell me which parameter you would like, I am willing to implement it and create a PR :)

pcwatch should now work on Windows, please try the latest version of playcanvas-sync

Closing since pcwatch now works on Windows. Please reopen if you continue to experience problems with this.


EKS Pod Identity Module [DONT MERGE YET]

What this PR does / why we need it:
Which issue(s) this PR fixes:
Fixes #
Quality checks

[ ] My content adheres to the style guidelines
[ ] I ran make test module="<module>" it was successful (see https://github.com/aws-samples/eks-workshop-v2/blob/main/docs/automated_tests.md)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

To align with our intent in #714 lets install the pod identity addon using the CLI instead of Terraform. You can see an example of where we've done this in the EBS lab https://www.eksworkshop.com/docs/fundamentals/storage/ebs/ebs-csi-driver

To align with our intent in #714 lets install the pod identity addon using the CLI instead of Terraform. You can see an example of where we've done this in the EBS lab https://www.eksworkshop.com/docs/fundamentals/storage/ebs/ebs-csi-driver

Fixed

Looks good to me. @rodrigobersa let us if you're good for this to merge.

Thanks everyone!


[Feature Request] Padding Parameter for BottomBar

Hi, I noticed that the padding value of the BottomBar has increased in the latest version and this caused mismatch in our app's UI. Could you please add padding parameter to the BottomBar widget?
Thanks for your work on this great package.

Thanks for filing the feature request :)
^2.0.1 now has a padding parameter. Have a great day!


Worker Role permission reduction

Describe the bug
Attached is the review of enhancements (please check column status) for the WorkerRole. There are two items that were deemed as good enhancements to minimize the permissions. Attached is the xls with details.
How to Reproduce
*P.S. Please do not attach files as it's considered a security risk. Add code snippets directly in the message body as much as possible.*

Expected behavior
The worker role post enhancements should have least privilege permissions.
Your project
No response
Screenshots
worker_role.xlsx
OS
All
Python version
3.10
AWS data.all version
v1.3,v1.4,v1.5
Additional context
No response

Hello @manjulaK, thanks for opening the issue. It will be added to the V1.6 project and if there is enough bandwidth it will be part of that release.


Transition gap between routes

Version
3.0.1
Reproduction link
https://codesandbox.io/s/yy0ovx9rz
Steps to reproduce
This is the example at https://github.com/vuejs/vue-router/blob/dev/examples/transitions/app.js with some styles changes.

In the codesandbox app, navigating between /parent, /parent/foo and /parent/bar
There is white gap between the screen components.

What is expected?
There is no white gap between the screen components.
What is actually happening?
There is white gap between the screen components.

I am creating an mobile application and want to mimic the native screen navigating behaviour. The white gap between screen makes it not really working.

A quick fix is to set the transform to a lower amount, try for example transformX(-70%).
But I'm also curious to know what is causing the white gap.

I believe the issue here was some confusion in the transition classes. There was no active transition during the enter hook.
The transition classes should look something more like
.slide-left-enter-active,
.slie-left-leave-active,
.slide-right-enter-active,
.slide-right-leave-active {
  transition: transform 1s, opacity 1s;
}

.slide-left-enter,
.slide-right-leave-to {
  opacity: 0;
  transform: translate3d(100%,0,0);
}

.slide-right-enter,
.slide-left-leave-to {
  opacity: 0;
  transform: translate3d(-100%,0,0);
}

https://github.com/vuejs/vue-router/tree/dev/examples/transitions
Using an example straight from the transitions example still shows a gap when transitioning between / and /parent. Instead of fading simultaneously, it waits till the first fades out completely before starting the next. The only time it doesn't show a gap is when you try transitioning between the parent and one of the children paths: /parent and /parent/foo and /parent/bar.

If there is going to be child router views with transitions, the router view element must have a parent element; it cannot be the root element of the component.
In other words, this won't work:
<template>
    <transition>
        <router-view class="child-view"/>
    </transition>
</template>

Must be something like this:
<template>
    <div>
        <transition>
            <router-view class="child-view"/>
        </transition>
    </div>
</template>

I completely missed this one, sorry about that. It seems to be a CSS problem rather than a bug coming from the Router


curious about the plans for supporting PEFT and LoRa.

Feature request
I need to be able to apply lora adapter to local llm
Motivation
lora is a good tool to lightly go through and check your current llm tuning
I think you need it in the local model api
Your contribution
...

plz~

Hey, do you know about https://huggingface.co/docs/peft/main/en/package_reference/tuners#peft.LoraModel.merge_and_unload
Basically, you could
model = model.merge_and_unload()
model.save_pretrained("mynewmergedmodel")

which will "write" the peft weights directly into the model, making it a regular transformer model, which text-generation-inference could support.
@younesbelkada in case I write something wrong.
It would be nice to add full blown support, but it means reintegrating of lot of peft logic into tgi, this seems like the easier route for the time being.

I second what @Narsil said, you can do that to merge the LoRA weights, we should indeed add more documentation on PEFT

Hey, do you know about https://huggingface.co/docs/peft/main/en/package_reference/tuners#peft.LoraModel.merge_and_unload
Basically, you could
model = model.merge_and_unload()
model.save_pretrained("mynewmergedmodel")

which will "write" the peft weights directly into the model, making it a regular transformer model, which text-generation-inference could support.
@younesbelkada in case I write something wrong.
It would be nice to add full blown support, but it means reintegrating of lot of peft logic into tgi, this seems like the easier route for the time being.

@Narsil
If, i want to use like this,
# model_name is request body data....
if model_name == "chat":
    with self.model.disable_adapter():
        model_output = self.model.generate(
            input_ids=input_ids.cuda(), generation_config=generation_config
        )[0]
else:
    self.model.set_adapter(model_name)
    model_output = self.model.generate(input_ids=input_ids.cuda(), generation_config=generation_config)[0]

it is impossible that using that way... isn't it?

I've wondered about this as well.
Is there a way to have plug and play finetuned adapters on specific tasks?

if task == "chat":
    self.model.set_adapter("chat_model")
elif task == "data_extraction":
    self.model.set_adapter("extraction_model")
elif task == "classification":
    self.model.set_adapter("classification_model")
elif task == "FAQ":
    self.model.set_adapter("faq_model")
else:
    self.model.disable_adapter()

I have finetuned falcon-7B and merged my LORA with the pretrained model. But when I try to load it I have the following errors:
Torch: RuntimeError: weight transformer.word_embeddings.weight does not exist
Safetensors: RuntimeError: weight lm_head.weight does not exist and indeed in config there is no lm_head filed.
Any clues on what should I do ?

@PitchboyDev  i think that is not in this issue agenda... i think that is simply huggingface's model problem. not peft.
because your error msg is not raised in peft.
how about moving your question to huggingface forum or someting?

It's neither peft nor transformers. It's an error from this repo's code cause I can load it and use it with a transformers Gradio app.
I found an issue about it: #541


Questions on RYScaler()

Hi Georgy,
I have 2 questions about the function utils.RYScaler():

For input data of cv2.calcOpticalFlowPyrLK() or cv2.optflow.createOptFlow_DIS(),
why don't you directly use rain rate (mm/h) scaled to [0,256]?  If use dbz, I suspect the intensity conservation assumption will be less obeyed than linear rain rate.

inv_RYScaler() has converted data back to mm/5 min. But when doing metrics such as CSI, the threshold is set using the unit of mm/h.  I guess there is an issue here.

Thanks.
Yuchao

Hi Yuchao,

Our preliminary work shows that the provided scaling strategy is more beneficial than linear scaling to [0, 255]. In your workflow, you can use any scaler you think fits your needs the best.

You are right, before metrics calculation, you should convert depth back to intensity. There is utils.depth2intensity() method for this purpose.

Best regards,
Georgy


[Feature] refactor pdml code.

[Feature] refactor pdml code.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.3 out of 4 committers have signed the CLA.:white_check_mark: fengzunbao:white_check_mark: mrdrivingduck:white_check_mark: HBKO:x: polardb-bot[bot]You have signed the CLA already but the status is still pending? Let us recheck it.

/close


EPIC: Curve Exponential Leverage Bot

Curve Exponential Leverage Bot:
We will create a new Curve leverage bot that will just continously add new leverage as price goes up. Like a limit order, but if price falls it won't add to leverage.
NOTE: The bot will still allow for automatic repayments and stop losses on health and on dollar amount.

https://www.notion.so/volumefi/EPIC-04-12-24-Curve-Exponential-Leverage-Bot-b301931963344de286beeddc4addc898

Closed as finished.


[FEATURE] Leave lobby endpoint

I'll close this because it's duplicated in #409 and being done in #456


Run other code?

How did I obtain the values of evaluation metrics such as PNSR, SSIM, and LPIPS for NICE-SLAM, Vox Fusion, ESLAM, Point SLAM, and SplaTAM * in the paper? I downloaded the corresponding source code from GitHub and ran it on my computer, but I couldn't obtain the specific values of these evaluation metrics.

Hey there. Thanks for your interest in our work. Some of the metrics we took from this survey. Concurrent methods we ran ourselves. To obtain results we ran each method three times with the seeds 0, 1, 2 and averaged them out.

These ESLAM, Splatam, nice slam can run but cannot obtain specific values for evaluation metrics such as PNSR, SSIM, and LPIPS in papers. Point slam can obtain these values in the Replica dataset, but cannot in other datasets. Do you want to add code for evaluation metrics such as PNSR, SSIM, and LPIPS to these source codes?
------------------ 原始邮件 ------------------
发件人:                                                                                "VladimirYugay/Gaussian-SLAM"                                                                                @.>;
发送时间: 2024年6月28日(星期五) 下午4:36
@.>;
<EMAIL_ADDRESS>主题: Re: [VladimirYugay/Gaussian-SLAM] Run other code? (Issue #28)
Hey there. Thanks for your interest in our work. Some of the metrics we took from this survey. Concurrent methods we ran ourselves. To obtain results we ran each method three times with the seeds 0, 1, 2 and averaged them out.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>

We didn't plan to add rendering evaluation code for other datasets.

Closing it for now. Feel free to re-open if you have any other questions!


Support writing fmp4 for Media Source Extensions

Sorry about the massive PR, this really started off with me having to make some small tweaks and blew up in size from there.

Put serde dependency behind a feature flag
Many fixed point numbers from the spec were previously interpreted as fractions, handle those as fixed point numbers instead. This also removes the dependency on num-fractions.
Make a couple of boxes read and write their flags properly. Previously this would fail to read many mp4 files/write invalid mp4 because of this.
A couple of boxes were missing readers and writers for some fields, this caused some browsers to fail to interpret the files because of invalid defaults.
Support a few new boxes required for mp4 segments in browsers.
Add a new mp4tree example that simply dumps the raw box tree.
Expose box apis externally.
Enable writing of avc2 and avc3 variants. Of particular interest is avc3 which stores AVC SPS and PPS in the NAL stream instead of in a separate box.
Probably many more small things.

I am writing mp4 segment files that work from a separate project, if it's wanted I can make a minimal example that writes a simple set of fragmented mp4 files.

Awesome, thanks for this! Looking forward to trying this out when it's ready to review! Good call on placing serde behind a feature flag too.

Hi @hansihe, is this still being worked on?

Yep, this is still on my backlog as part of a project. Added a comment for what I believe is the biggest blocker for merging.

Ok, thanks for the update!

Closing. Feel free to re-open with updated changes if you wish to continue this PR. Thanks.


Setting marks on the graph region

This issue is somewhat related to the #26.  Once we get the coordinates from the event in #26, using the “instance” returned from calling the functionPlot() function, is there another function that I call that I can passing in the X and Y coordinates values, so that the FP library can “mark” an X or something similar on anywhere on the graph region and not only be limited to the plotted charts?  Hope this function has a Boolean parameter called “override” so that when it’s set to true, it will clear all the “marks”, and then place the new “mark” based on the new X, Y coordinates.

have a look at the homepage and look for the annotations example

Annotations are great for x or y labeling at the time when graph is getting plotted by calling the functionPlot() function. This marking should take place after the graph has been plotted.  Lets say that the end-user has click on a plotted line, is there and event or a function that we can call so that it can put a mark on the line, as seen below?

how about using another datum which keeps track of the points? e.g.
var options = {
  target: '#demo',
  data: [{
    points: [],
    graphOptions: {
      type: 'scatter',
      sampler: 'builtIn'
    }
  }, {
    fn: 'x^2'
  }]
}
functionPlot(options)

And then whenever you get a coordinate just update the points option e.g.
var x = // some value
var y = // evaluation of a datum with x = some value
options.data[0].points.push([x, y])
functionPlot(options)

I haven't tested this, hopefully it works

I'll try it first thing tomorrow morning and get back to you.  Thanks

I tried your code below on the JS Bin on your site and it worked.
var options = {
target: '#demo',
data: [{
points: [],
graphOptions: {
type: 'scatter',
sampler: 'builtIn'
}
}, {
fn: 'x^2'
}]
};
functionPlot(options);
var x = 1.367;
var y = 1.870;
options.data[0].points.push([x, y]);
functionPlot(options);
Is there a way to make the mark bigger? like a bigger circle?

mmm I think you can override the styling of the  element on your spreadsheet e.g.
.function-plot .graph circle {
  r: 5;
}

Let me try this right now (CSS style).

After a "marking" has been placed on the graph using the code below:
options.data[0].points.push([x, y]); functionPlot(options);
the goals I am trying to achieve is that on every click on a plotted line:

the previous marking should be removed
then a new marking should be displayed (using above code).

how can I clear a "marking" (No 1) once the end-user has clicked on another x, y coordinates on a plotted line?

It worked; I was passing in some wrong options.  Based on the code you provided, when clicking on the next coordinates, the previous ones (points: []) gets removed first.  The CSS provided worked great as well.
Thanks

.function-plot .graph circle {
r: 5;
}
Works in Chrome 48.0.2564.103 m. Not FF 41.0.2 or IE 11.0.9600.18163CO (didn't test others) I think that the idea is that r is a 'physical' property not a style.
See http://stackoverflow.com/a/32410381
So it would be nice if  we could add an r property to the datum just like color.
Or maybe even just expose all svg attributes. Meaning just check if they are on the datum. I don't know just throwing out an idea.

@meeple142 thanks for the info! I'll add this in a future relase


[ci] vendor Nix packaging - [merged]

In GitLab by @perarnau on Oct 2, 2020, 10:41
Merges shell-pin-pkgs -> staging
Localize the nix pkgs definitions to allow for more fine-grained dependency management.
Also updates some of our dependencies, breaking + updating a bunch of things in the process.

hwloc is kept at a version old enough to make it easy
doxygen is kept in line with previous CI step until we check the rtd infra

In GitLab by @freuk on Oct 2, 2020, 11:27
added 1 commit
85264667 - [ci] Separating shell and packages in two Nix files.
Compare with previous version

In GitLab by @freuk on Oct 2, 2020, 11:37
added 1 commit
5d77177f - [ci] Reverts doxygen to 1.8.14.
Compare with previous version

In GitLab by @freuk on Oct 2, 2020, 11:44
changed title from [ci] {-pin version of argopkgs we are us-}ing to [ci] {+vendor Nix packag+}ing

In GitLab by @freuk on Oct 2, 2020, 11:44
changed the description

In GitLab by @perarnau on Oct 2, 2020, 11:45
changed the description

In GitLab by @perarnau on Oct 2, 2020, 12:01
added 1 commit
719e2de8 - [fix] missing _GNU_SOURCE definition in places
Compare with previous version

In GitLab by @perarnau on Oct 2, 2020, 12:14
@NicolasDenoyelle can you help with the last valgrind errors? I'm not sure I understand that test.

In GitLab by @NicolasDenoyelle on Oct 2, 2020, 15:01
added 1 commit
0fb45c0e - valgrind fixes
Compare with previous version

In GitLab by @NicolasDenoyelle on Oct 2, 2020, 15:02
added 1 commit
084fb7e7 - clang format
Compare with previous version

In GitLab by @perarnau on Oct 2, 2020, 15:17
merged

In GitLab by @perarnau on Oct 2, 2020, 15:17
mentioned in commit 0827bffb90ab2d3930d57eda238b8c6963ceae89


Result<T>.Error throws when calling it with an already thrown exception

Only happens on .NET 5 and later.

The question is of course: How should this function behave? There are several options:

A: Throw if the stack trace is already set
B: Do nothing if the stack trace is already set
C: Replace the stack trace

The current "fallback" implementation for .NET Standard implements C: Replace the stack trace since it sets the private field via reflection.
On .NET 5 and later we use ExceptionDispatchInfo.SetCurrentStackTrace (which I used under the wrong assumption that it behaves exactly like our reflection hack) implements A: Throw if the stack trace is already set.
One unfortunate thing about replacing the stack trace is that it's a very easy footgun for unsuspecting users (I have seen code like this):
var result = originalResult.Match(
  ok: v => v.ToString(),
  error: Result<string>.Error);
  // This replaces the stacktrace but is the only way to implement a Match like this outside of Funcky. (we don't expose the constructor)
  // (internally we use `new Result()` in our `Select` implementation for this exact reason)

I think we should change the behaviour to A: Throw if the stack trace is already set

On an unrelated note: I did an experiment where I reimplemented our reflection hack for setting the stack trace in IL:
https://github.com/polyadic/funcky/blob/experiment/set-stack-trace-without-reflection/Funcky.Unsafe/Funcky.Unsafe.il

Per discussion IRL: We change the behaviour to B: Do nothing if the stack trace is already set
Even though this is technically a breaking change in an implicit contract, we don't treat it like one.


Memory leak in saving and loading a keras model containing CategoricalEncoding and Lookup layers

System information

Have I written custom code (as opposed to using a stock example script provided in Keras):  Yes
OS Platform and Distribution: CentOS Linux release 7.9.2009 (Core)
TensorFlow installed from: binary
TensorFlow version: 2.6.0 (v2.6.0-rc2-32-g919f693420e), 2.7.0 (v2.7.0-rc1-69-gc256c071bb2)
Python version: 3.7.11, 3.8.12, 3.9.6
Bazel version: not applicable
GPU model and memory: no GPU available
Exact command to reproduce: See code below

Description of the problem
To solve a binary classification problem, I have a keras model that processes categorical input (as as well as numeric input).
I need to save (model.save) and load (tf.keras.models.load_model) the model multiple times (performig training of the model inbetween).
I expect that the model consumes constant disk space and constant RAM everytime I load the model since the architecture does not change (only the parameter values change).
This does not happen when the model contains an IntegerLookup layer followed by a CategoryEncoding layer.
The issue can be reproduced without training the model at all.
Here is a minimal code example that creates a model and saves it to disk:
import tensorflow as tf
import numpy as np

input_layer = tf.keras.Input(shape=(1,), dtype="int32")
index = tf.keras.layers.IntegerLookup(max_values=2)
index.adapt(np.array(range(2)))
encoder = tf.keras.layers.CategoryEncoding(max_tokens=index.vocab_size())
encoded_layer = encoder(index(input_layer))
output_layer = tf.keras.layers.Dense(2, activation=tf.keras.activations.softmax)(encoded_layer)
model = tf.keras.Model(input_layer, output_layer)
model.compile(optimizer="adam", loss="sparse_categorical_crossentropy")
model.save("model")

Every time I execute
model = tf.keras.models.load_model("model")
model.save("model")

the space that the model consumes on disk increases by approx. 8 kB.
The even worse: When I load the model, the RAM useage increases by approx. 9 MB in each iteration.
So after 100 iterations, the model needs approx. 1 MB on disk and 950 MB RAM (which is problematic).
This also happens if I start a new python process in each iteration.
In my application, the memory consumption grows even faster because the model has several input layers and also several inner layers.
This makes the model unusable after some iterations because I cannot load it anymore.
Additionaly, of course, the load and save cycles are getting slower with each repetition.
So far, I could reproduce this issue on tensorflow versions 2.6 and 2.7 running on python 3.7, 3.8 or 3.9. The behavior is identical.
I originally posted this problem as a tensorflow issue.

@AdrianFuchsData Sorry for the late response!
I tried to replicate the issue on colab using TF v2.7.0,  tf-nightly (2.9.0.dev20220126) and faced different error.Could you please find the gist here for reference and confirm the same ?
Thanks!

@sushreebarsa
Indeed, if you execute the above code like in the linked google colab document, you do not see much.
The issue becomes visible if we keep track of the memory consumption when saving and loading the model repeatedly.
Save the following python script to load_and_save_model.py
import tensorflow as tf
import psutil

# Open the saved model
model = tf.keras.models.load_model("model")

# Write memory consumption to a file
with open("memory_consumption_when_loaded.txt", "a") as file:
    file.write(f"{psutil.virtual_memory().used}, ")

# Resave the model whitout any modifications
model.save("model")

In a notebook (or a seperate python script) do the following:

Create the model
 import tensorflow as tf
 import numpy as np

 input_layer = tf.keras.Input(shape=(1,), dtype="int32")
 index = tf.keras.layers.IntegerLookup(max_values=2)
 index.adapt(np.array(range(2)))
 encoder = tf.keras.layers.CategoryEncoding(max_tokens=index.vocab_size())
 encoded_layer = encoder(index(input_layer))
 output_layer = tf.keras.layers.Dense(2, activation=tf.keras.activations.softmax)(encoded_layer)
 model = tf.keras.Model(input_layer, output_layer)
 model.compile(optimizer="adam", loss="sparse_categorical_crossentropy")
 model.save("model")

Initialize memory counter lists
 num_cycles = 50
 memory_size_not_loaded = [0 for cycle in range(num_cycles)]
 saved_model_size = [0 for cycle in range(num_cycles)]

You can change the parameter num_cycles to an arbitary integer value.

Run the subscript load_and_save_model.py repeatedly and track memory consumption. This might take some time.
 import psutil
 import os

 for cycle in range(num_cycles):
     memory_size_not_loaded[cycle] = psutil.virtual_memory().used
     os.system("python3 load_and_save_model.py")
     saved_model_size[cycle] = os.path.getsize("model/saved_model.pb")
     print(cycle)

If you are running tensorflow in a python virtual environment, you might need to adapt the os.system call accordingly.
By performig the operation in the subscript, we gurantee that the memory leak is not caused by insufficient garbage collection.

Load the memory data tracked by the subscript
 with open("memory_consumption_when_loaded.txt", "r") as file:
     memory_size_loaded = [int(value) for value in file.read().split(",")[:-1]]

 # Clean up directory
 os.remove("memory_consumption_when_loaded.txt")

Visualize the memory consumption
 import matplotlib.pyplot as plt

 ax = plt.subplot(1, 1, 1)
 ax.plot(range(num_cycles), memory_size_not_loaded, color="green", label="Memory used when model is not loaded")
 ax.plot(range(num_cycles), memory_size_loaded, color="red", label="Memory used when model is loaded")
 plt.legend(loc="upper left")
 ax.set_xlabel("Cycle")
 ax.set_ylabel("Memory consumption (bytes)")
 ax2 = ax.twinx()
 ax2.plot(range(num_cycles), saved_model_size, color="blue", label="Size of model on disk")
 plt.legend(loc="lower right")
 ax2.set_ylabel("Model size on disk (bytes)")
 plt.savefig("memoryleak.png", bbox_inches="tight")
 plt.show()

When I do the above on my system, memory consumptions look like this:

Observations:

The green line (memory consumption when the model is NOT in memory) is constant. This is expected behavior.
The blue line grows linearly. This means that in each cycle, the disk size of the model increases by the same amount.
The red line grows somehow stronger than linear (maybe quadratic). This implies that when repeadetly saving and loading a model, after some more cycles I eventually will run out of memory, which makes this approach unusable to me.

I expect that if I load and save the model without any modification, its size remains constant, both on disk and in memory.
Can you reproduce this pattern?

I faced this issue in loading the model. Found a temporary workaround by using
model2.load_weights()

I faced this issue in loading the model. Found a temporary workaround by using model2.load_weights()

@AdrianFuchsData, Could you please try the workaround suggested by @splendidbug.
Also, try if the issue still exists when you use latest Tensorflow using tf-nightly. Thanks!

Switching to tf-nightly
The nightly build version of tensorflow (version 2.10.0-dev20220419) warns that the identifiers max_values, vocab_size, max_tokens are deprecated and should be renamed to max_tokens, vocabulary_size and num_tokens accordingly. So the code for creating the model should now be
import tensorflow as tf
import numpy as np

input_layer = tf.keras.Input(shape=(1,), dtype="int32")
index = tf.keras.layers.IntegerLookup(max_tokens=2)
index.adapt(np.array(range(2)))
encoder = tf.keras.layers.CategoryEncoding(num_tokens=index.vocabulary_size())
encoded_layer = encoder(index(input_layer))
output_layer = tf.keras.layers.Dense(2, activation=tf.keras.activations.softmax)(encoded_layer)
model = tf.keras.Model(input_layer, output_layer)
model.compile(optimizer="adam", loss="sparse_categorical_crossentropy")
model.save("model")

With this updated code, the result is completely the same as initially described. So the issue has not been fixed by the current version.
Using model.load_weights instead of model.load
It is not obvious how to replace model.load by model.load_weights. Here is what we tried:
Adding model.load_weigths to the subscript load_and_save_model.py
One easy try is to simply add the line model.load_values, i.e., first load the model, then refresh the model values with the loaded values. The subscript then looks the following:
import tensorflow as tf
import psutil

# Open the saved model
model = tf.keras.models.load_model("model")
model.load_weights("model")
# Write memory consumption to a file
with open("memory_consumption_when_loaded.txt", "a") as file:
    file.write(f"{psutil.virtual_memory().used}, ")
# Resave the model whitout any modifications
model.save("model")

Tensorflow accepts this code until saving the model, where it runs into a NotFoundError exception, if the model is saved to a mounted Windows file share. It seems that some files of the model have not been closed properly when loading the model. (Maybe this is a topic for another issue)
When saving to a Linux native file system, we do can save the model, but this does not solve the issue. The memory consumption growth pattern is exactly the same.
Creating the model from scratch, then load the weights into it
On the other hand, we could recreate the model from scratch and load the values into it. The subscript looks the following:
import tensorflow as tf
import psutil
import numpy as np

# Instanciate a model along the same lines as the original model
input_layer = tf.keras.Input(shape=(1,), dtype="int32")
index = tf.keras.layers.IntegerLookup(max_tokens=2)
index.adapt(np.array(range(2)))
encoder = tf.keras.layers.CategoryEncoding(num_tokens=index.vocabulary_size())
encoded_layer = encoder(index(input_layer))
output_layer = tf.keras.layers.Dense(2, activation=tf.keras.activations.softmax)(encoded_layer)
model = tf.keras.Model(input_layer, output_layer)
model.compile(optimizer="adam", loss="sparse_categorical_crossentropy")
# Load the weights from the model
model.load_weights("model")
# Write memory consumption to a file
with open("memory_consumption_when_loaded.txt", "a") as file:
    file.write(f"{psutil.virtual_memory().used}, ")
# Write predictions to a file to be able to compare them (has actually the same model been used in each iteration?)
with open("predictions_when_loaded.txt", "a") as file:
    file.write(f"{model.predict(np.array((0, 1)))}\n")
# Resave the model without any modifications
model.save("model")

Indeed, this now gives the desired constant memory consumption. Predictions are also identical (as expected), which makes this approach a feasible and tolerable workaround.

In my use case, the model architecture depends on the initial data. So using this approach, we would need to save the model architecture in some format, such that we can reconstruct it later (This is why I like and model.load so much, it does this work for me). This means error-prone extra code for my project. Is there a more elegant way?