[PIFO Trees] Telemetry

This is WIP and is not ready for serious review. I am just opening this PR so I can seek help getting it all the way there. This PR will eventually close #1697: the overall plan is exactly as summarized at the bottom of that thread.
Important background:

Recall that what we call "PIFO" in pifo.py in Calyx is actually a more general thing: it is simply a queue that orchestrates two sub-queues.
As part of this orchestration, it has the ability to divine by itself which flow a given input belongs to: 0 or 1. Based on the flow, it then pushes the input into one of its sub-queues.  We'll call this the "top-level" flow inference to distinguish it from whatever further flow inference the sub-queues may go on to perform.
When a PIFO's sub-queues are simply FIFOs, we get simple PIFO behavior. When the sub-queues are themselves PIFOs, we get more expressive PIFO tree behavior. This can be extended forever, gaining height and therefore expressivity.

A summary of what I have done in this PR so far:

The general PIFO-adding Python script now optionally takes as argument stats, which is a handle to a stats component. If passed a stats component, the PIFO component will, after each successful push, report the top-level flow of the recently pushed item to the stats component.
The stats component is a little funny, and I am seeking your feedback re: its design. I currently have it like so:

It takes two 1-bit inputs, report and flow.
It maintains two registers, count_0 and count_1.
If report is high, it also expects two further ref-registers. It drives the values of count_0 and count_1 to those ref-registers and is done. In this case the value of flow is ignored.
If report is low, it actually computes statistics using flow: it increments the appropriate counter and is then done.

The controller component (previously known as mother) invokes the stats component, passing a bogus value for flow, a raised flag for report, and cells into which it wants the stats to be written by reference.
The overall design intent is this: when designing my PIFO tree, I will pass the top-level PIFO queue a stats component, but not pass any of its sub-queues stats components. In this way, our stats component will accumulate a coarse overall view of the top-level statistics of the tree: the very first branching, no more.

Further work is required all over:

I'd like overall stylistic feedback on my Calyx.
I'm looking for feedback re: the design of the stats component in particular. Is my use of inputs and refs reasonable? Should some things just be continuous or non-continuous outputs?
I'm wondering how to make the controller component repeatedly poll the stats component instead of doing it just once.
It would be nice (and not too hard) to give the controller the ability to zero-out the stats component. Maybe that can just be the default behavior upon report.
It would also be nice (and maybe a bit tricky) to eke out the finest possible flow inference instead of the coarsest possible as I have it now. To do this, I would need each sub-queue to report to its caller what its flow inference was. The top-level PIFO, which has at hand a stats component, would then need to somehow digest this fine-grained info and report that to the stats component, not the coarse info that is has to hand.

Thanks for getting us started on this @anshumanmohan! What do you think about formalizing some of the details of the architecture in a system diagram? This would also give a head start for the figure needed in the paper and act as a nice at-a-glance reference for what the architecture is doing

Sure thing! Here's a quick sketch. It's actually an animation that I'll present when we meet, and hopefully it'll make more sense then.
Also, I spoke to @sampsyo and he suggested I can probably do some cleanup in the stats component: instead of an internal count_0 and a separate ref_count_0 that I occasionally drive to, I can just maintain count_0 (resp. count_1) as an out port all along.

A key you will need to understand my digram:

Awesome, the visual is helpful but also leads me to a question: does each "managing queue" have access to its own stats component? If that is the case, is the controller accumulating this information and generating some form of cumulative statistics to report to the outside world?

Right now only the root "managing queue" has a stats component, and it is not learning anything from its sub-queues about their flow-splitting decisions. This means that the (sole) stats component is only ever getting the most coarse-grained info, from the root.
This is an interesting space to explore, though. What I have above is a little unsatisfying. Two further options that will get us finer-grained info:

We keep the setup where only the root talks to the stats component, but we make it so that the root queries its children about their flow-splitting decisions. Perhaps is not a query per se, but just an automatic report from any queue to its caller upon push. Then the root converts this info into some sort of compact scalar and reports that to the stats component.
We change the setup, hooking up all the leaves of this tree to a stats component. These leaves can "just know" their place in the tree. So when a leaf tells the stats component, "Hey, I am leaf 001 (read, 'left, left, right') and I just accepted a packet", the stats component increments some special counter that is reserved for that leaf. Over time, this stats component accretes fine-grained information.

I think this will now pass the compiler tests, which is a step in the right direction. We're not done yet though.
@calebmkim I am wondering if I can get your eyes on something. Let's look at the Calyx file generated for now, which I've pushed temporarily.
My question is this: are you thinking that we do currently have the issue where two different copies of the static component get stamped out by controller and pifo_root?

It looks to me like the two uses are getting at the same stats component, but I could be wrong. To see what I'm saying, search for invoke stats in the file.
Although maybe I'm wrong and your warning is right: if you search for stats = stats(), one can believe that this is exactly where a discrete copy gets stamped out.

In case you are right, would you remind me what the fix was? I seem to recall you saying something about passing the stats component by reference?

Hmmm, I still think there's the problem. If I'm understanding correctly, here's basically what's going on.
component stats (...) -> (...) {
  // define stats component
}
component pifo_root(...) -> (...) {
  cells {
    stats = stats() // 1st instantiation of stats
    ...
  }
  // invoke stats()
}
component controller(...) -> (...) {
  cells {
    stats = stats() // 2nd instantiation of stats
    ...
  }
  // invoke stats()
}

I think you only want one instantiation of stats() throughout the entire design; otherwise you're creating duplicates. Think of the cells section as instantiations of hardware: we could replace stats = stats() in the controller with stats_different_name = stats(), resulting in the following controller
component controller(...) -> (...) {
  cells {
    stats_instance_diff_name = stats() // 2nd instantiation of stats
    ...
  }
  // invoke stats_instance_diff_name()
}

This is the same program as before, but very clearly, there are two different(ly named) stats components.
Solution
Sadly the solution is probably going to be very ugly, since we have given up on passing components by reference (bc we can't invoke things because we need both to run for the exact same time). It seems like dataplane will only ever need to write to the stats component (through the pifo tree), while controller will only ever need to read from stats.
I think it would look something like this:
component main(...) -> (...) {
  cells {
    stats_instance = stats();
    dataplane_instance = dataplane();
    controller_instance = controller().
  }
  wires {
    groups rtl_group {
       stats_instance.go = dataplane_instance.stats_instance_go;
       stats_instance.flow = dataplane_instance.stats_instance_flow;
       stats_instance.flow = dataplane_instance.stats_instance_report;
       controller_instance.stats_instance_count0 = stats_instance.count0; //
       controller_instance.stats_instance_count1  = stats_instance.count1;  //
       ...
    }
  }
}

component dataplane() -> (stats_instance_go: 1, stats_instance_flow: 1, stats_instance_report: 1) {
    cells {
        my_queue = pifo_root();
    }
    control {
        // named ports differently for clarity.
        invoke my_queue(...)(stats_instance_go = pifo_param_stats_instance_go, stats_instance_flow = pifo_param.....)
    }
}

component pifo_root() -> (pifo_param_stats_instance_go: 1, pifo_param_stats_instance_flow: 1, pifo_param_stats_instance_report: 1) {
    ...
    group can_write_to_stats_now {
        pifo_param_stats_instance_go = xxx
        pifo_param_stats_instance_flow = xxx
        // etc.
    }
}

Thanks Caleb!
Re: why it's not working, I think I get it 100%. Those two instances of stats() are totally different, just like two different std_adds would have been different.
Re: the solution, let me try to say it at a high level first. Please nitpick me ruthlessly.

In component main, you have defined the three players as cells. I'll add the three external memories here too, no problem.

Also in main, you have this group where you've written a faux-RTL group. This is hooking up the stats component to the controller_instance and the dataplane_instance like a puppet: all of its in- and out-ports are now managed by one of those two other components, via some port of their own. So playing with stats_instance is just a matter of playing with the appropriate puppet-master ports in controller_instance or dataplane_instance.
Question: I'm guessing we want this group to be constantly enabled?
So the rest of the rtl_group (after your five lines) would have:

dataplane_instance.go = HI.
... a whole buncha desugared stuff to pass memories by reference to dataplane, but without using invokes.
controller_instance.go = HI.
rtl_group.done = dataplane_instance.done.

And main's control program would just be one thing: rtl_group.

I'm a little confused by invoke my_queue(...)(stats_instance_go = pifo_param_stats_instance_go, stats_instance_flow = pifo_param.....). Is the point that:

Invoke the queue normally, passing inputs as usual (...).
But then in the outputs, capture info about how the queue would like to interact with the stats component (stats_instance_go = pifo_param_stats_instance_go, stats_instance_flow = pifo_param.....).
And because the faux-RTL group in main is always enabled, capturing the info in this way automatically communicates our wish to the stats component as well.

And then in group can_write_to_stats_now you are showing me how to drive values to the queue's ports.

1

Re: why it's not working, I think I get it 100%. Those two instances of stats() are totally different, just like two different std_adds would have been different.

Yep
2

Question: I'm guessing we want this group to be constantly enabled?

Yeah, and I think this kind of goes hand in hand with your other comment:

And main's control program would just be one thing: rtl_group.

Since main's control program is just one group ("faux-rtl"), that group will always be enabled.
3

But then when the invoke returns, capture its outputs in local variables. (stats_instance_go = pifo_param_stats_instance_go, stats_instance_flow = pifo_param.....). This precisely captures info about how the recently-returned queue would like to interact with the stats component.

To nit-pick, this isn't necessarily "when the invoke returns".
Idk if you know this already, but invokes are just compiled into groups: see this file and its corresponding .expect file.
So what we're really saying is that, for the entire time the invoke is running (i.e., when my_queue's control is executing), anytime my_queue writes to its output parameter pifo_param_stats_instance_go, that value will be written to dataplane's output parameter stats_instance_go, which (in main) is hooked up to stats_instance.go, hence triggering an execution of stats_instance. Does that make sense?
4

And then in group can_write_to_stats_now you are showing me how to drive values to the queue's ports.

Yeah basically.
5
One last thing: it seems like controller doesn't even have to invoke the stats component. If all it has to do is read from stats.count0 and stats.count1 (which are assigned to in continuous assignments and therefore can be read from anytime, even when stats is not active)., the controller can just periodically poll by reading from stats.count0 and stats.count1. Does that make sense (I could be missing something tho)?

This all sounds great, Caleb! Thanks!! I'll get to it, and we'll see what comes up.

This is now working! I have not implemented the approach that Caleb and I shook on just above, opting instead for a lighter-weight version that @sampsyo and I discussed.
The high-level intent earlier was for main to call controller and dataplane in parallel and watch for dataplane to be done. This is tricky and is being discussed in its own issue. Trying to do it here would have required us to go to a low-level RTL style and there is much pain there. See above.
We have instead opted for a style where main just runs a while loop, conditioned on a kill signal from dataplane, and inside the whole loop it allows dataplane and controller to do quick pieces of work.
To make this work, I had to adapt dataplane to perform its work in this piecemeal way. That component is used to just having a list of commands and running free with it. Now it still has the list of commands, but on every invocation is processes one more command. When it is done with all the commands, or when running a command returns an error, the dataplane raises its kill signal and the main loop respects it by ending the whole program.
 I will clean that up shortly. For now I just wanted to mark this PR as ready for review.

I have pushed the .yx file generated by the eDSL just for ease of review. I will remove it afterwards, and before merging. The point of pushing it is to demonstrate that we are no longer instantiating multiple copies of stats; we are instantiating one and passing it around by refernece.

I have tidied up the eDSL code some, which is nice at the Python level and has led to a leaner hardware design. Kinda fun to see the changes here. This is now ready for review.
I would like to catch the other queues up to this nicer style, but that's going to be a separate issue.


Fix composition logic for Firefox

Fixes: https://github.com/microsoft/vscode/issues/106392
In Firefox when inserting Emoji's using the macOS emoji input or using the compositor for characters with accents è, ë, î, etc. the character is inserted twice. Additionally, when adding an accented character the base character was also inserted.
Examples:
🏅 inserts 🏅🏅
è inserts eèè
This change does two things:

It removes this._onType.fire(typeInput); from the compositionupdate event listener. If you are in the middle of a composition there is no need to fire off the event. This change fixes the unaccented character from being inserted. è no longer inserts eè.

In the composition end function browser.isFirefox is added to the conditional that resets this._textAreaState to the value of the text area. This is needed as the insert of the composition is handled by L294. The textarea needs to be updated correctly when the compositions ends.

Tested in Firefox (Windows 10, Ubuntu, macOS), Safari (macOS), Chrome (macOS), Edge Legacy (Windows 10), Edge (Windows 10)

This PR fixes #

Just a reminder, for changes in this area we need to go through all the input modes documented at https://github.com/microsoft/vscode/wiki/IME-Test
@rebornix Could you please also help to test this? I think removing this._onType.fire(typeInput); from the compositionupdate will break CJK input.

Alex, thank you for taking a look. I will work through the tests in that doc as well.

I gave it a test on macOS and found it changed the behavior of Japanese IME in both Safari and Firefox, meaning that removing this._onType.fire(typeInput); changes the behavior
One example:

Install Japanese IME
Switch to Hiragana
Open VS Code Insiders, type watashi, press enter, type watashi again, press enter
You will get 私私
With this change, type watashi, press enter, you will get 私. And then press watashi again, press enter, it ends up with 私. Somehow the second 私 is omitted.

I'll take a closer look at why this breaks CJK in October, thanks for your help @belcherj !

Tested that if (browser.isEdge || browser.isChrome|| browser.isFirefox) { can solve the Firefox but without breaking other browsers. Thanks for your help!


add more metrics in node status collection

Closing this as nothing zkapp related to add to node status. Updated node error collection to send transaction inclusion failures


Old Harmonised sumstats named incorrectly

The harmonised sumstats for Suhre et al  https://www.ebi.ac.uk/gwas/publications/28240269 are incorrectly named tsv.h.gz rather than h.tsv.gz. This is a typo arising from when they were manually harmonised, and is an isssue because they do not appear on the list of all harmonised files. All harmonised files in this publication should be renamed h.tsv.gz.

We should also check if there are any other files on the ftp in this state. James thinks unlikely, but we should confirm.

Number of files matching 'GCST*-GCST*/GCST*/harmonised/GCST*tsv*': 28308
Number of files matching 'GCST*-GCST*/GCST*/harmonised/GCST*.h*tsv*': 28065

Using the code here /homes/spotbot/goci1196_incorrect_hm_name, I checked the files. There are no files named tsv.h.gz but there are files named tsv.gz. Attached is the list.
@ljwh2 Please let me know if they should be renamed too.
list_of_harmonised_tsv_gz.txt

@earlEBI confirmed the files named tsv.gz should also be renamed

Renaming done. Here is the list.


Problem with consent.google.com on the links

Hi!
I'm trying to resolve the internal links from Google news like this:
`def scrape_headlines(keyword):
gnf = GoogleNewsFeed(language='en', country='US', resolve_internal_links=True)
results = gnf.query(keyword)
headlines_links = {}

for result in results:
    title = result.title
    link = result.link
    headlines_links[title] = link

return headlines_links`

But keep getting errors like this when I want to get the article content through newspaper3k:
Error getting article content for https://consent.google.com/ml?continue=https://news.google.com/rss/articles/CBMiSGh0dHBzOi8vY2Euc3R5bGUueWFob28uY29tL29wcmFoLW9uY2UtZGVlbWVkLWJ1dHRlcnktc29mdC0xMjMwMDA3ODcuaHRtbNIBUGh0dHBzOi8vY2Euc3R5bGUueWFob28uY29tL2FtcGh0bWwvb3ByYWgtb25jZS1kZWVtZWQtYnV0dGVyeS1zb2Z0LTEyMzAwMDc4Ny5odG1s?oc%3D5&gl=NO&hl=en-US&cm=2&pc=n&src=1
It seems like all the links from Google gets sent to consent.google.com. Have you had this problem?

To be frank, I haven't utilized this library for several months, and there's a possibility that Google may have modified their internal routing. Once I return from work, I'll investigate the issue briefly. If the resolution is straightforward, I will take care of implementing it. However, if the fix is more complex, I'll inform you and it may be necessary for you to devise a workaround. In the event of an actual internal routing change by Google, other libraries may also encounter similar difficulties.

Indeed, there has been a change in their internal logic. However, this has been addressed and updated in version 1.1.0, which now correctly resolves the links.

That's great! I actually came up with a different solution as the base64 part of the Google news URL actually hides the re-direct URL, here is my code if you would like to do something similar:
`def get_first_url(original_url: str) -> str:
encoded_url_prefix = "https://news.google.com/rss/articles/"
if not original_url.startswith(encoded_url_prefix):
    return original_url  # URL is not a Google News encoded URL

encoded_url = original_url[len(encoded_url_prefix):].split('?')[0]
encoded_url += "==="  # Fix incorrect padding
decoded_bytes = base64.urlsafe_b64decode(encoded_url)

decoded_url_re = re.compile(rb'^\x08\x13".+?(?P<primary_url>http[^\xd2]+)\xd2\x01')
match = decoded_url_re.match(decoded_bytes)
primary_url = match.groupdict()["primary_url"].decode()

return primary_url`

Well, that's actually a pretty clever solution, kudos to you for discovering it!  Thanks for sharing!


The edge fields grouping does not update on selecting different keys in aggregate

Create a Kafka -> Aggregate topology, selecting some field as the key
Edit the Aggregate and select a different field as the key and save
The edge fields grouping still refers to the old field as the key.

It should be updated to the new key (or keys).
Please check similar behavior in other processors like join.

Fixed this bug and added supported for Real-Time Join processor in https://github.com/hortonworks/streamline/pull/948#pullrequestreview-66067904


Overwrite Data?

Is it possible to get the program to overwrite data if you go back? I tend to save and reload a lot but it keeps data from the first save at that specific time and so can become inaccurate pretty quickly. Is there a way to overwrite data at specific time points?

Hello. Currently the Stellaru data model is append-only when new saves are loaded. With some work it could probably be made to support purging of out-of-order snapshots. I can take a look into that sometime this week.

Newer save files with older in-game dates will now result in the sequence of data being rewritten. Published https://github.com/benreid24/Stellaru/releases/tag/v3.1.0 with the change.


Type Casting

Description 📜
This documentation contains detailed explanation of  Type Casting in Java and its types with examples.
Fixes # 1533

Type of change 📝

[x] Documentation (Content Creation in the form of codes or tutorials)

Domain of Contribution 📊

[x] Java_Domain

Checklist ✅

[x] I follow Contributing Guidelines & Code of conduct of this project.
[x] I have performed a self-review of my own code or work.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] My changes generates no new warnings.
[x] I'm GWOC'21 contributor

Screenshots / Gif (Optional) 📸

/assign

/assign document

https://github.com/girlscript/winter-of-contributing/pull/3811 Review my PR

/assign

@jaigora24 It is a PR, not an issue that can be assigned.

@ketkikaushik Please do all the changes suggested by Saksham and Me

@ketkikaushik Please do all the changes suggested by Saksham and Me

ok I'll work on that.

@AkMo3 I've made all the changes that you requested. Please review it.


python38Packages.jupyterlab_server: 2.8.2 -> 2.9.0

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for python38Packages.jupyterlab_server is: "JupyterLab Server"
meta.homepage for python38Packages.jupyterlab_server is: "https://jupyter.org"
meta.changelog for python38Packages.jupyterlab_server is: ""
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact

Checks done (click to expand)

built on NixOS

The tests defined in passthru.tests, if any, passed

0 of 0 passed binary check by having a zero exit code.

0 of 0 passed binary check by having the new version present in output.

found 2.9.0 with grep in /nix/store/pgc825qmpigk24vxw798gsjjw9y5afn4-python3.8-jupyterlab_server-2.9.0

found 2.9.0 in filename of file in /nix/store/pgc825qmpigk24vxw798gsjjw9y5afn4-python3.8-jupyterlab_server-2.9.0

Rebuild report (if merged into master) (click to expand)

16 total rebuild path(s)

9 package rebuild(s)

9 x86_64-linux rebuild(s)
7 x86_64-darwin rebuild(s)

First fifty rebuilds by attrpath
docbookrx
python38Packages.jupyterlab
python38Packages.jupyterlab-git
python38Packages.jupyterlab_server
python38Packages.jupytext
python39Packages.jupyterlab
python39Packages.jupyterlab-git
python39Packages.jupyterlab_server
python39Packages.jupytext

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/pgc825qmpigk24vxw798gsjjw9y5afn4-python3.8-jupyterlab_server-2.9.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A python38Packages.jupyterlab_server https://github.com/r-ryantm/nixpkgs/archive/27f4341c1d54b0cd310da69bf139899fe3cbf65e.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/pgc825qmpigk24vxw798gsjjw9y5afn4-python3.8-jupyterlab_server-2.9.0
ls -la /nix/store/pgc825qmpigk24vxw798gsjjw9y5afn4-python3.8-jupyterlab_server-2.9.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

8 packages built:

python38Packages.jupyterlab
python38Packages.jupyterlab-git
python38Packages.jupyterlab_server
python38Packages.jupytext
python39Packages.jupyterlab
python39Packages.jupyterlab-git
python39Packages.jupyterlab_server
python39Packages.jupytext

Maintainer pings
cc @costrouc for testing.

Updated in #151861


advanced profiler describe + cleaned up tests

Before submitting

[x] Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
[x] Did you read the contributor guideline?
[x] Did you make sure to update the docs?
[x] Did you write any new necessary tests?

What does this PR do?
Fixes #836
PR review
Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

agreed, i'll clean the tests up a bit

@jeremyjordan really nice refactoring, just a minor comments :rocket:

@jeremyjordan have made your last changes or @williamFalcon was faster with his merge?

@Borda the latter. i had planned to extract _get_total_duration as a module function and update the docs to be sentences

@jeremyjordan sorry, i thought this was ready because there was no WIP. Can you open another PR to finish these changes? :)

@williamFalcon yeah no worries that was my bad! sure thing :)


How to set the path of functions_dir on mac?

Because there will be spaces in the path name on Mac, there are spaces in the functions_dir content displayed by the .info command, so the actual functions_dir directory cannot be located when executing %functions%. In this case, does it need to modify functions_dir as the correct content? Please give me more advice! thx～

see https://github.com/sigoden/llm-functions?tab=readme-ov-file#getting-started-with-aichat
ln -s "$(pwd)" "$(aichat --info | grep -w functions_dir | awk '{print $2}')"
# OR
argc install

I have executed this command. When I executed the aichat -r %functions% command, it did not take effect, and the output was still the answer of the model. (I also installed argc and jq.)

The following is my directory structure and aichat settings information:

@butchoosed deepseek-chat does not support function calling.

@sigoden
i got it,Thank you very much!


Add warnings if forward URL is non-standard (Electron deep link formats like electron-fiddle://)

Plex Forums link:
https://forums.plex.tv/t/forwardurl-not-redirecting-for-non-http-https-urls-eg-my-app-plex-for-third-party-mobile-client/795347
The forwardURL cannot be anything other than http/https or the redirect is blocked by plex, additionally, if you provide a forwardURL that is invalid it scraps the entire authentication method and fails back to the main plex.tv homepage

Good catch, I'll look into adding a protocol check to this.
Thank you


Package structure and services

What services does the COSORE package provide?

Developer: build database from raw datasets
▪	Parse metadata and raw data files, standardize columns, etc. - csr_build()
▪	Make a release - csr_build()
▪	Run reports - run_single_report(), run_combined_report()
▪	Rmarkdown vignette
Developer: tools to modify
▪	Create new dataset
▪	Modify format of DESCRIPTION, COLUMNS, etc. files
▪	Generate emails?
Developer: package testing
▪	testthat
User:
▪	Download release
▪	List available datasets - summary
▪	Reports on database and individual datasets (using above)
▪	Extract individual tables
▪	Run arbitrary functions on database
▪	disk.frame seems ideal for this, but a heavyweight dependency…maybe just provide examples using it

Now that we've shifted to standardized data being stored in the repository and package, some of the considerations of this and #70 (with which it's linked) shift.
Right now we're just saving the data and diagnostics objects, but probably we want to save all tables as (individual?) RDS objects, right? Then it becomes easy to provide functionality independent of drake.
In particular, user-side functionality...

csr_database(): Get the combined DESCRIPTION tables - this should include first and last timestamp?
csr_table(): get combined anything
csr_dataset(): Get a single dataset, standard list structure
Reports on database and individual datasets (using above)

Developer

Builds using drake pipeline
Calls function to save objects inside of package
Rebuild package
Now the user-side functions above have access to new data. They no longer take an all_data parameter, obviously


Handle Amazon Aurora/RDS DNS "Connection Lost" messages

Using Amazon Aurora, laravel throws PDO occasionally like:
SQLSTATE[HY000] [2002] php_network_getaddresses: getaddrinfo failed: Name or service not known
When these happen, existing retry logic should be engaged.  We're using 5.4 (for php compatibility reasons), so PR is against that version.

Thanks for the PR. But 5.4 is not maintained anymore. We can accept this to 5.5 and 5.6+ though. I'll prep a PR to that affect.


Set semver prerelease as pre-release

Would like an option to to autmatically mark the file as a pre-release if versioned as a semver pre-release. (eg 1.1.0-beta.2)
Don't know whether there exists another way to accomplish this?

Something like this might work for you:
name: Create a GitHub release
on:
  push:
jobs:
  release:
    name: Create a Github release
    if: startsWith(github.ref, 'refs/tags/')
    runs-on: ubuntu-latest
    permissions:
      contents: write
    steps:
      - name: Check if tag is a SemVer (pre-)release
        run: |
          # SemVer regex adapted for bash from:
          # https://semver.org/#is-there-a-suggested-regular-expression-regex-to-check-a-semver-string
          SEMVER_REGEX="^(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)(-((0|[1-9][0-9]*|[0-9]*[a-zA-Z-][0-9a-zA-Z-]*)(\.(0|[1-9][0-9]*|[0-9]*[a-zA-Z-][0-9a-zA-Z-]*))*))?(\+([0-9a-zA-Z-]+(\.[0-9a-zA-Z-]+)*))?$"
          if [[ "$GITHUB_REF_NAME" =~ $SEMVER_REGEX ]]; then  # If it matches, then it's a SemVer name
            prereleasePart="${BASH_REMATCH[5]}"  # The 5th group is the pre-release part
          else
            echo "Version $GITHUB_REF_NAME is not SemVer" 1>&2
            exit 1
          fi
          if [[ -n "$prereleasePart" ]]; then  # If not empty, then it's a pre-release
            echo "is_prerelease=true" >> "$GITHUB_ENV"
          else
            echo "is_prerelease=false" >> "$GITHUB_ENV"
          fi
      - name: Create a GitHub Release
        uses: softprops/action-gh-release@v2
        with:
          prerelease: ${{ env.is_prerelease }}

Disclaimer: I manually edited this from a more complex setup, renaming a few things here and there for the purposes of leaving a comment here that would hopefully be helpful. I can't guarantee that this would actually work as-is in a GitHub Actions workflow: there might be errors, both in the YAML syntax and at runtime.


26: error build: Definition of 'CIFilter' must be imported from module 'CoreImage.CIFilter' before it is required

: error build: Definition of 'CIFilter' must be imported from module 'CoreImage.CIFilter' before it is required

Should be fixed in 5.12.1 + for macOS


chore: Upgraded Spring Boot and Spring to intermediate ver

Fixes #18992
Description
Mandatory upgrade path to allow secrets to be re-encrypted correctly. Will also include other changes here if they end up being made.
Changes so far:

[x] Upgraded to Spring Boot 2.7
[x] Upgraded to Spring Security 5.8
[x] Modified encryption and decryption mechanism in Appsmith
[x] DB migration for re-encrypting already existing secrets in DB

TODO:

[x] Check if we need to or can enable Authorization Manager. This is going to be default turned on in Spring 6. (1)(2)
I've enabled this right now, don't see any problems with our redirection to login page
[x] Check if we need to change the error handling at entrypoint, and for which ones
Essentially just needed to add handling as specified here, to continue to throw the error to the client when such a failure occurs
[x] Do we need to be upgrading the MySQL R2DBC library?
Seems to work okay, we should do a full regression of the plugin though.
[x] Test migration against release data
[x] Figure out a way to ensure this migration path is used
Refer to this thread, I'm going to go ahead and try this out

Notes:
https://stackoverflow.com/questions/63778955/how-to-inject-beans-without-interface-in-mongock-changelogs
Type of change

Chore

How Has This Been Tested?

Manual

Test Plan
We'll need to test the migration by creating DB entries using release branch for all features that use secrets and then running this branch and checking if they all still work.
This will also need regression of any of the Appsmith paths that make use of secrets. For example:

Git
OAuth2 (REST API and Google Sheets)
Anything with DB auth
API key and Bearer token (REST)
Any auth that needs certificate file upload

Regression for MySQL
Regression around anonymousUser/authenticated users/few ACL scenarios. Please make sure to test for all the correct types of redirection when users are not logged in (for Appsmith edit page, public app, private shared app, etc)
Issues raised during DP testing
Checklist:
Dev activity

[ ] My code follows the style guidelines of this project
[ ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes
[ ] PR is being merged under a feature flag

QA activity:

[ ] Test plan has been approved by relevant developers
[ ] Test plan has been peer reviewed by QA
[ ] Cypress test cases have been added and approved by either SDET or manual QA
[ ] Organized project review call with relevant stakeholders after Round 1/2 of QA
[ ] Added Test Plan Approved label after reveiwing all Cypress test

Regression tested this on the instance provided by dev. Few issues found as below  (highlighted to dev too)-

Camera widget does not load - getUserMedia is not implemented in this browser
Map widget does not load - requires google maps api
Sample DBs are not working on Migrated apps
On the share modal, we see a new link You haven’t setup any email service yet. Do not see this on release. Also - Link takes user to incorrect page of how to self host appsmith with docker.
Welcome Tour - When I click on Welcome Tour link, it asks me to provide credentials for the Users DB. As per actual behavior - this should directly proceed to the tutorial screen
Fork Template on the migrated workspace - gives query errors due to Sample DB not configured
Configured OAuth2.0 - but found issues due to http on redirect url

With regards auth that needs certificate file upload - unable to test this area as we dont have a certificate file-- any pointers here would help.

Regression tested this on the instance provided by dev (with migrated data and git connected apps). Few issues found as below (highlighted to dev too)-

Camera widget does not load - getUserMedia is not implemented in this browser
Map widget does not load - requires google maps api
Sample DBs are not working on Migrated apps
On the share modal, we see a new link You haven’t setup any email service yet. Do not see this on release. Also - Link takes user to incorrect page of how to self host appsmith with docker.
Welcome Tour - When I click on Welcome Tour link, it asks me to provide credentials for the Users DB. As per actual behavior - this should directly proceed to the tutorial screen
Fork Template on the migrated workspace - gives query errors due to Sample DB not configured
Configured OAuth2.0 - but found issues due to http on redirect url

With regards auth that needs certificate file upload - unable to test this area as we dont have a certificate file-- any pointers here would help.

On the recreated instance, tested and found Sample DB's to be working on Migrated apps, Welcome Tour functions correctly, Fork Template works correctly, Oauth2.0 works fine.
Rest of the areas are specific to setup and should be fine once merged.
Ran second round of testing on all the areas once again and all works fine.
cc: @nidhi-nair @prapullac

/ok-to-test sha=9796393


Move tests to pytest

The current tests work but they're a mess, fixing things which broke under edge cases with the Python parser of Pandas etc.
Tidying up and simplifying the tests by moving them to pytest would be great

closed by #8


On group chat sometimes characters are confused about who they are

Sometimes they answer like they where someone else

I will just attribute the message to the character


Closes: #1

Updated validations.py python script.
Fixed the behavior of validate_user function in validations.py.
Fixes #<issue_number_goes_here>

It's a good idea to open an issue first for discussion.

[ ] Tests pass
[ ] Appropriate changes to README are included in PR

It is my first pull request...


Help i cant download vsc even though my connection is good enough

VSCode Version:
OS Version:

Steps to Reproduce:

Does this issue occur when all extensions are disabled?: Yes/No

Any error message?


error       Unable to resolve module AccessibilityInfo？

Duplicate of #4968

That is the solution step by step
https://stackoverflow.com/questions/51333897/react-native-build-error-development-server-returned-response-error-code-500/51360535


Fix the installation issue of operator under a specific namespace

Fixes #196

/retest

/cherry-pick release-1.6


A problem caused by the extension.

Excuse me .
I installed this extension , the global replacement function is affected
More info.Search-and-replace in more than one file

Thanks, I have uninstalled.

I think this should be re-opened and taken into consideration, @Lonefy


Leaner travis

Trying to tone Travis builds down a bit so we get them reliably green again until we properly break out our runtimes into their own repos, to test and integrate them seperately.
Solves #1649 for the moment.

Merged as https://github.com/openwhisk/openwhisk/commit/ecdd337d7c9cc30add5fc0c16877514c337c3dca.


Automated cherry pick of #1994: huawei rebuild root fix

Cherry pick of #1994 on release/2.11.
#1994: huawei rebuild root fix

Codecov Report

:exclamation: No coverage uploaded for pull request base (release/2.11@0a80836). Click here to learn what that means.
The diff coverage is 0%.

@@              Coverage Diff               @@
##             release/2.11   #1996   +/-   ##
==============================================
  Coverage                ?   7.19%
==============================================
  Files                   ?     445
  Lines                   ?   69254
  Branches                ?       0
==============================================
  Hits                    ?    4982
  Misses                  ?   63859
  Partials                ?     413

Flag
Coverage Δ

#aFlag
7.19% <0%> (?)

Impacted Files
Coverage Δ

pkg/util/cloudinit/cloudconfig.go
55.13% <0%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0a80836...292ddb3. Read the comment docs.

/lgtm

/approve


Add message change to include data restrictions check.

Refs: PV-768
Description
Context

PV-768
How Has This Been Tested?

Manual Testing Instructions for Reviewers

Screenshots

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (b5a3e8e) 82.52% compared to head (e0c8894) 82.50%.
Report is 2 commits behind head on develop.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #454      +/-   ##
===========================================
- Coverage    82.52%   82.50%   -0.02%
===========================================
  Files          154      154
  Lines         9504     9508       +4
===========================================
+ Hits          7843     7845       +2
- Misses        1661     1663       +2

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


messages sending is not allowed.

If i update thingSpeak and SPIFFS data from web and serial terminal the message sending not allowed error still happened until i reboot the module. So I think when message sending was blocked each time it should check if the thingSpeak or SPIFFS value is updated or not, or it can be overcome if this issue is resolved completely #123

#142 has this solution but it is still not tested completely so closing this for now if any issue occurs then we'll reopen it.


OOM error with just 1 1080p .png using the pre-trained in colab

Hello, im trying to convert 1 1080p pic <2mb
using this:
!python cartoonize.py --styles hayao --batch_size 1 --skip_comparison --input_dir /content/prova --output_dir /content/CartOUT --keep_original_size
but i get this error:
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[1,270,480,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:Conv2D]
which i have read is related to the lack of enough gpu ram, is there a way to get over it without increase gpu ram? im using batch_size = 1 and colab
ram in colab:
Gen RAM Free: 12.6 GB  | Proc size: 415.4 MB
GPU RAM Free: 7611MB | Used: 0MB | Util   0% | Total 7611MB
without --keep_original_size it works fine.
any idea?

Apparently 1080p is too memory-consuming for our model and colab GPU. If you really want to keep the original size, try GPU with more memory like P100, V100, which you can find at GCP or Azure, etc. We cannot guarantee that you can successfully transfer 1080p image on P100 or V100 since we do not own a P100 or G100 vm.


Fix fork-tree descendent check

Fork tree nodes below the fork-tree roots should not be finalized.
Previous code was preventing finalization of roots children descendants but was allowing finalization of root children.

Could you add a test?

Could you add a test?

@bkchr I've added a couple of assertions to an existing test to exercise the fix


Lock sides/prevent resizing OR don't erase after resizing

This is an amazing little tool, however countless times I would be working on something near the edge and accidentally grab the side and pull the size of the frame in, only to loose my entire paint!
Maybe a little checkbox that prevents edges from being resized and locks in the size of the frame.

Does the new undo functionality adequately solve this for you? Trying to keep the interface as simple as I can!

Yup it does, solves and then some! Great work!!


error happed in example code mnist_mlp.py

I runned example code mnist_mlp.py, getting following warning and error. I wondering what leads to these error. Thanks in advance.
UserWarning: MRG_RandomStreams Can't determine #streams from size (Shape.0), guessing 60*256
nstreams = self.n_streams(size)
ValueError: size must remain unchanged, changed from 1280 to -10
Apply node that caused the error: GpuReshape{2}(GpuElemwise{Add{output_types_preference=transfer_type{0}}}[(0, 0)].0, MakeVector.0)
Inputs shapes: [(128, 10), (2,)]
Inputs strides: [(10, 1), (8,)]
Inputs types: [CudaNdarrayType(float32, matrix), TensorType(int64, vector)]
Use the Theano flag 'exception_verbosity=high' for a debugprint of this apply node.

Do you have the latest Keras and Theano (from the Github repos)?

No, I installed the Keras 1.2 and the Theano 0.7.  I can running successfully with these two on other computer, but failed on my computer.

I made a mistake, I install both Theano 0.6 and 0.7, and actually run on Theano 0.6, which leads to these error.


Make NFL data go back beyond 1999

Currently ESPN API only goes back to 1999
Find another source of data for below 1999 that can go back further

The Sports DB looks good: https://www.thesportsdb.com/league/4391-NFL?a=1#allseasons


Allow Duplicate Account Names when previous account is suspended

Describe the outcome you'd like
AFT would allow duplicate account names if another account with the same name has been decommissioned and whose status is SUSPENDED in organizations.
Is your feature request related to a problem you are currently experiencing? If so, please describe.
A deprovisioned account must reside within an AWS Organization for 90 days during the post-closure period. As far as I can tell, there is no limitation within AWS Organizations to have duplicate account names, and AFT is imposing this in the account request
Additional context

AWS Organizations allows multiple accounts with the same name
as far as I can tell, there is no use of account name as a primary key within AFT

seconded!


Search paths and features defined

Search paths developed through formative evaluation with group of researchers (all experience levels)

advert for intern now live

When this story is reviewed, note that @FreecenBren indicated in email on 2016-10-11 that place of birth (in addition to county of birth) may also be desirable on the search form.

Questionnaire now ready to go out for this. To prepare email template/mesage to send out.

To review current responses and assess if ready to send out to wider sample of users/respondents.

To revise survey where applicable and then discuss who we send it out to.

Project document for @richpomfret : https://docs.google.com/document/d/1Le4pLD5GSM1p4x7iMKOj0KY1vteCF1XwkQQir0plac4/edit?usp=sharing


Change 'Please Log in to comment' to a button [YEP-66]

@bruskiza @miltontony ready for review

:+1:


Use XTENSION=FOREIGN for ASDF-in-FITS

Use the FOREIGN keyword in ASDF-in-FITS HDUs purely in an effort to pass fitsverify.
The new HDU does not actually conform to the FOREIGN convention as described here. However, the fitsverify program provided by NASA's FITS office appears to validate HDUs with the FOREIGN keyword if they otherwise conform to the IMAGE standard.
To make things even more confusing, fitsverify does NOT appear to validate an HDU that actually conforms to the FOREIGN convention. It also does not validate an HDU that uses XTENSION=ASDF. We hope to eventually sort things out with the FITS office, but in the meantime, we want users of fitsverify to be able to validate our files.
If you need to convince yourself that this actually works now, open a Python interpreter and try the following:
from asdf.tests.test_fits_embed import create_asdf_in_fits
asdf_in_fits = create_asdf_in_fits()
asdf_in_fits.write_to('asdf.fits')

You can open the new file asdf.fits to confirm that there is in fact an HDU with XTENSION=FOREIGN, and it holds the contents of the ASDF file. You can also run fitsverify against this file to confirm that it passes validation.
cc @stscicrawford

If we move forward with this change we will need to update the change log.

Coverage remained the same at 94.024% when pulling a56e007f9218b8cbb8aa3c9158ce6e5ed16cc65e on drdavella:use-foreign-keyword into 3cec7ff7301b28d003fd39375a8a4163a9300a4f on spacetelescope:master.

Thanks @drdavella -- this looks good to me especially with the extensive note in the documentation.  If this is merged, an issue should be created to track this as well.

@stscicrawford, @hbushouse after several weeks and several emails with no response from NASA's FITS office, should I go ahead and merge this?

Is it the case that this change is still creating an extension that does NOT actually conform to the official FOREIGN convention, but instead creates one that will pass fitsverify? If so, I'm not comfortable with forcing our code to do something wrong just so it matches something else that's equally wrong. We should be doing the right thing and telling folks that fitsverify is not to be used, because it's wrong.

I agree that I do not like this as a long term solution either.   An issue open about this definitely needs to be kept open if this is accepted and it should be regularly revisited, but considering how widely used CFITSIO is, I do not think it is a realistic option to ask people to stop using it.
As completely alternative thought, would it be possible to store the ASDF specific stuff inside a FITS table?   This might be a little awkward but it would get around this problem of using the Foreign extension.

Like how putting into a table? If just stuffing it as one big binary item, we have the same issue as using the image format don't we? It's not really a table then (or whatever kind of column item it becomes).

@perrygreenfield Yes it  was just a thought -- I didn't know if it was stored as a table (even a 1x1 table with a big binary blob) that might be a more 'FITS-like' solution.   Then again, I also don't remember from the FITS standards if binary blobs can be stored in FITS tables, so this might all moot.

Has anyone tried contacting Bill Pence directly? (He's shown as emeritus, so he may be still active with FITSIO<EMAIL_ADDRESS>

@stscicrawford You are probably right expect weirder things from tables than images. Column name of ASDF_CONTENT or something like that provides some clue.

Okay, I'm gong to recommend merging this so it fixes the issues faced by the other groups and we have a mostly compliant FITS file and then opening a follow-up issue for how to handle the third version of this.

@perrygreenfield I have not contacted him directly: I just tried using the FITS email alias. I can give him a try though, unless you want to.

@drdavella which email address did you use specifically? I ask since getting cfitsio to adopt the changes is more important than the FITS office specifically.

I got<EMAIL_ADDRESS>from https://fits.gsfc.nasa.gov/.

try<EMAIL_ADDRESS>

Okay I'll give it a try. @stscicrawford should I still merge or hold off to see if I get a response?

Okay, give it another couple of days to see if you hear a response from them.

Closing since this does not seem to be the right solution. Discussion will continue at https://github.com/STScI-JWST/jwst/issues/1807.


Finalize template feature.

Document and announce it.

I want to change the template identifiers from snake_case to kebab-case. I think that is more natural


Add sortBy and partition (fix #4274)

fix #4274

Pre-submit checklist:

Branch

[x] Tests are provided (if possible)
[x] Commit sequence broadly makes sense
[x] Key commits have useful messages
[x] Relevant tickets are mentioned in commit messages
[ ] Formatting, materialized Nix files, PNG optimization, etc. are updated

PR

[ ] (For external contributions) Corresponding issue exists and is linked in the description
[x] Self-reviewed the diff
[x] Useful pull request description
[x] Reviewer requested

Oops, sorry! git didn't find stylish so I committed without it...


[Bug] Treat null or undefined in table data's binding as valid empty data, instead of as an error in the debugger

Description
When the result of the binding in Table Data is null or undefined, the debugger shows an error saying this is not valid data. However, there's cases where a table is bound to a request's data, like so {{myApiRequest.data}}, but the API action is only executed after clicking a button. In that case, until the button is clicked and the API action is executed, this binding will evaluate to null or undefined. This is not an error and the debugger should accept any empty values in the table data.
Steps to reproduce the behaviour:

Important Details

Version: Cloud
OS: macOS
Browser: Firefox
Environment: https://app.appsmith.com.

@somangshu @vicky-primathon can we tried empty / undefined as a valid input to the table?


Project 30 timer code not matching with book text

Code in project file:
https://github.com/twostraws/HackingWithSwift/blob/db581a1aee7c08d387addc12564f717751550e29/project30-files/Project30/ImageViewController.swift#L38
Code in book text:
self.animTimer = Timer.scheduledTimerWithTimeInterval(5, target: self, selector: #selector(animateImage), userInfo: nil, repeats: true)

Looks like the book text needs updated since scheduledTimer(withTimeInterval:repeats:block) is the current syntax.

Thanks! This will be fixed in the next update.


fix(types): queryCache.getQueries

There were some missing types in the getQueries method's type definitions. I copied the parameter types from the other methods that allow AnyQueryKey | string | function, and added boolean as it looks like that's also a possible parameter type, looking at the implementation.

queryKeyOrPredicateFn:
      | AnyQueryKey
      | string
      | boolean
      | ((query: CachedQuery<unknown>) => boolean)

this type should be added to almost all queryCache methods: getQueryData, setQueryData, getQuery, getQueries
make sense to extract it into separate type:
type QueryKeyOrPredicateFn:
      | AnyQueryKey
      | string
      | boolean
      | ((query: CachedQuery<unknown>) => boolean)

@smashercosmo Sounds good! I wasn't sure how far I should go with it. I just pushed an update.

Sorry about all of the weird extraneous commits, I was just trying to get the checks into a good state, there seems to be a preexisting race condition in a test or two 😢

@all-contributors Please add @jgettings for code

:tada: This PR is included in version 2.4.12 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Update ptransform.py error message

Improves PTransform mixed pipelines error message.
R: @pabloem

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[X] Choose reviewer(s) and mention them in a comment (R: @username).
[- ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[-] Update CHANGES.md with noteworthy changes.
[-] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

LGTM. There's a lint failure from a line-too-long:
00:02:44.972 apache_beam/transforms/ptransform.py:602:0: C0301: Line too long (82/80) (line-too-long)

You can let yapf autoformat the code: tox -e py3-yapf, and you can run linters with tox -e py37-lint I believe


memory leak bug

Steps to reproduce the behavior.
# coding=utf-8

from memory_profiler import profile
from rocketry import Rocketry
from rocketry.conds import every

app = Rocketry()

@profile
@app.task(every("1 seconds"),  execution="async")
async def job():
    print(1)

if __name__ == '__main__':
    app.run()

View the dat file in the same directory,memory leaks will be found

OS: [win11]
python 3.9
rocketry 2.4.0

Does your result show memory usage between two different runs of this task? If so, this most likely is not an issue: the accumulation of memory is actually expected.
By default, Rocketry logs the tasks (ie. when a task started, when it succeeded/failed/etc.) to an in-memory list. This is fundamental part of Rocketry as most of the scheduling is based on whether a task started/finished at a given period of time, as does the condition every (though there is some optimization in place by default).
What does the results show if you disable the logger?

from memory_profiler import profile
from rocketry import Rocketry
from rocketry.conditions.api import scheduler_cycles
from rocketry.conds import every

import logging
logger = logging.getLogger("rocketry.task")
logger.disabled = True

app = Rocketry()

@profile
@app.task(every("1 seconds"),  execution="async")
async def job():
    print(1)

if __name__ == '__main__':
    app.session.config.shut_cond = scheduler_cycles(more_than=10)
    app.run()

How did you run the memory_profiler? I tried to reproduce this but couldn't get the output out of the library.

mprof run test.py

I ran it with logging disabled and your results are indeed because the logs are accumulated in memory (which is expected).
I ran Rocketry with the same setup as above but allowing scheduler to be running for 2 min and this is the results:
MEM 1.777344<PHONE_NUMBER>.4039
MEM 3.437500<PHONE_NUMBER>.5066
...
MEM 3.437500<PHONE_NUMBER>.4000
MEM 3.394531<PHONE_NUMBER>.5079
...
MEM 3.394531<PHONE_NUMBER>.0621
MEM 3.394531<PHONE_NUMBER>.1707
MEM 3.394531<PHONE_NUMBER>.2785

So for some reason the memory consumption actually decreased after 30 seconds, maybe due to some representation change but after that, it stayed completely the same for the remaining time.
Therefore it seems Rocketry does not leak memory based on this example. It does accumulate memory by default but this is intended. The logs can be manually cleaned (ie. using another task preferably with execution as main) to avoid accumulating those indefinitely.
And thanks for the observations and the effort! This was a good thing to check.

I didn't disable logging as shown in my code
Runs every second, I ran maybe 2 minutes, memory is growing and not falling. I can try to disable logging to see if I can solve this problem

Yes, but there is no problem from Rocketry's point of view. If you append an item to a list, you cannot expect your memory consumption to go down:
l = []
l.append("item")

Appending to the list increases your memory usage and this is not a memory leak, it's standard behaviour. What you observed was exactly the same: Rocketry adds a new log record to the logs which is a list in memory (if no other destination is set). The past logs are vital for Rocketry as many of the core conditions are dependent on the run history. This is by design and it's not a bug. My example proved it's exactly the logging that causes the accumulation of memory thus it's exactly due to this.
Of course, you can direct the logs to a disk or to a database to not keep them in RAM or you could create a task that periodically deletes too old records. Or disable the logging if you don't mind that some of the conditions might not work. The user is free to choose the option that makes the most sense in their situation.


fix: update 'transaction_hash' import path

Please check if the PR fulfills these requirements

[ ] The formatter, linter, and tests all run without an error
[ ] Tests for the changes have been added (for bug fixes/features)
[ ] Docs have been added / updated (for bug fixes / features)

What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)
fixes: #93

What is the current behavior? (You can also link to an open issue here)
Unable to use Client or other stuff due to an outdated import statement

What is the new behavior (if this is a feature change)?
Things work again! :)
Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)
No

Could you also change fix the version of cairo-lang to 0.7.1 in pyproject.toml (and generate new lock file as well)? This bug seems to have come from unexpected implementation change on upgrade from0.7.0 to 0.7.1


Problem of Installation

Dear Authors,
Thanks for your work and releasing code. But I encountered some problems during installation and would like to ask for your help.
The first problem is about the installation of pytorch_scatter. If using conda install pytorch-scatter -c pyg, the lower versions of pytorch and cuda will be automatically installed. And command pip install torch-scatter -f https://data.pyg.org/whl/torch-2.0.0+cu118.html works fine on my PC.
The second is about the installation of DROID-SLAM. Since the project relies on the old Eigen, a lot of compatibility errors were reported. I checked the original project and couldn't find a solution. If possible, could you please share some experience on installing DROID-SLAM?
A error I encountered recently is /usr/local/cuda-11.8/include/crt/common_functions.h:74:24: error: token ""__CUDACC_VER__ is no longer supported.  Use __CUDACC_VER_MAJOR__, __CUDACC_VER_MINOR__, and __CUDACC_VER_BUILD__ instead."" is not valid in preprocessor expressions
Thanks in advance

Thank you trying out the code.

"conda install pytorch-scatter -c pyg" seems to work fine for us. It's tested with torch-2.0.0 and Cuda-11.8 on ubuntu.
I updated the "Eigen" submodule. Please clone the repo again.
In addition, I updated "install.sh" a little, to install cuda-11.8 (if you needed) and fix detectron2 version to counter a recent bug.
Please give it another try.

Now the installation runs smoothly. For me i have to install lietorch using pip, otherwise it will report ImportError: cannot import name 'SE3' from 'lietorch'. Anyway, the code produces good result on the demo video.  Thanks a lot for your help!

E:\project\tram\thirdparty\DROID-SLAM\thirdparty\eigen\Eigen\src/Core/PlainObjectBase.h(963): error: function "Eigen::Block<XprType, BlockRows, BlockCols, InnerPanel>::operator=(const Eigen::Block<Eigen::Matrix<int, -1, 1, 0, -1, 1>, -1, -1, false> &) [with XprType=Eigen::Matrix<int, -1, 1, 0, -1, 1>, BlockRows=-1, BlockCols=-1, InnerPanel=false]" (declared implicitly) cannot be referenced -- it is a deleted function
Eigen have problems


[v2.12.0] Ensure CI/documentation reflect changes to default admin credentials

Background
Previously, when installing the security plugin demo configuration, the cluster was spun up with the default admin credentials, admin:admin. A change was made in main and backported to 2.x for the 2.12.0 release, which now requires an initial admin password to be passed in via the environment variable OPENSEARCH_INITIAL_ADMIN_PASSWORD. This will break some CI/testing that relies on OpenSearch to come up without setting this environment variable. This tracking issue is to ensure compliance with the new changes.
Coming from: https://github.com/opensearch-project/security/issues/3624
Acceptance Criteria

[ ]  All documentation references to the old default credentials admin:admin are removed
[ ]  Ensure that CI/testing is working with main and 2.x branches

Closing this as complete: https://github.com/opensearch-project/opensearch-net/pull/480


Use Error classes in Aggregate

Hey there! First off: thanks for this awesome repository, this and your great sample project helped me a lot.
I  have a question though: I prefer  using Error classes over pure string messages. I found out that I can basically just use Fail and Ok with these classes. However that doesn't work for Aggregate (I assume  same for Entity) as it implements the create method as follows:
static create(props: any): IResult<Aggregate<any>>;

that falls back to the default string parameter D of IResult so the create method doesn't allow me to use anything else. Is this intended or a bug? Maybe I'm using it wrong so some advice  would be great here!
export interface IResult<T, D = string, M = {}> {
    value(): T;
    error(): D;
    isFail(): boolean;
    isOk(): boolean;
    metaData(): M;
    toObject(): IResultObject<T, D, M>;
    execute: <X, Y>(command: ICommand<X | void, Y>) => IResultExecute<X, Y>;
}

Cheers

Hey @mmintel, thanks for reporting...
check that I understand correctly.
You want to return a custom error and not just the standard error message, correct?
see if the example below helps you
import { Result, Aggregate, UID, Ok, Fail } from "types-ddd";

// My custom error, any type you want
interface CustomErr {
  params: any;
  message: string;
}

// Aggregate props
interface Props {
  id?: UID;
  name: string;
}

// Aggregate class
class User extends Aggregate<Props> {
  private constructor(props: Props){
    super(props);
  }

  // method to validate some business rule
  public static isValidProps({ name }: Props): boolean {
    return this.validator.string(name).hasLengthBetween(5, 40);
  }

  // factory method
  public static create(props: Props): Result<User, CustomErr> {

    // business validation
    const err = { param: props.name, message: "invalid name" };
    if(!this.isValidProps(props)) return Fail(err);

    // return success
    return Ok(new User(props));
  }
}

// failure case
const empty = "";
const failResult = User.create({ name: empty });

console.log(failResult.error());
// { param: '', message: 'invalid name' }

// success case
const name = "Jane Doe";
const result = User.create({ name });

console.log(result.error());
// null

console.log(result.value().toObject());
/**
{
  id: '47c27bbf-f708-4117-9c68-0c29e9819b9e',
  createdAt: 2023-07-07T15:50:57.464Z,
  updatedAt: 2023-07-07T15:50:57.464Z,
  name: 'Jane Doe'
}
*/

@4lessandrodev thanks for your explanation but that's exactly what I was trying to do. However this throws something like
Class static side 'typeof Foo' incorrectly extends base class static side 'typeof Aggregate'.
  The types returned by 'create(...).error()' are incompatible between these types.
    Type 'CustomError' is not assignable to type 'string'.ts(2417)

because string is the default and there is  no way to pass my custom error to Aggregate.

Are you using types-ddd version 3.6?

@mmintel please check version 3.6.2

@4lessandrodev 3.6.2 fixed it, thanks a lot for your quick support!


chore(controller): update Django to 1.6.10 security release

See https://www.djangoproject.com/weblog/2015/jan/13/security/

Code LGTM.  Given our use of the WSGI headers mentioned in the CVE we should consider a patch release.

Code LGTM


OutOfMemoryError in gradlew process

Expected Behavior
gradlew should work fine
Current Behavior
While running a build generating a lot of logs (around 50MB in a single task when the build succeeds).
gradlew generates:
java.lang.OutOfMemoryError: GC overhead limit exceeded
It is the gradlew process (the one run from the gradlew script and that has by default a 64MB max heap size) that generates the error. It is NOT the gradle daemon itself.
Increasing the heap size using GRADLE_OPTS solves the issue, but how can we set a reasonable size?
gradlew is called with the --no-daemon option.
Your Environment
Here is a heap dump from the gradlew process when the error occurs:
https://gist.github.com/emmanuelguerin/374d04e91b3c0bbb870a5878e5df6189/raw/efc12c1880c7497a63551c4643a1a69b0c1c1f41/java_pid30749.hprof

I can't quite tell from the heap dump.. Which version of Gradle is this?

I tried with gradle 5.1.1 and 5.3.1, they both seem to have the issue. The dump is from gradle 5.1.1.

Another thing that may not show in the logs is that --parallel is also set, and other tasks are running when the client crashes.

We haven't see that particular issue for quite a while. I'll close this.


Add settings alert page

Closes #36

Tests to come.

Coverage increased (+0.1%) to 93.376% when pulling aeeee2a83105502886e22771594479bb10c03f1f on eatonphil:master into 2115a68c85c092738e3dfde90060c5954b634302 on linode:master.

Coverage increased (+0.1%) to 93.376% when pulling 645bc191bea258bfd62454db7714fcf7ef5ece12 on eatonphil:master into 2115a68c85c092738e3dfde90060c5954b634302 on linode:master.

API support for this exists. Please wire it up.

IO Ops/sec looks really weird, I believe you could get away with IOPS instead.

@SirCmpwn ok!
@na3d ok!

Coverage decreased (-1.08%) to 92.617% when pulling 2aa140815e03b74391fc2c398694afd2a4c09460 on eatonphil:master into 78fac4d23750f5c9b57e8a5ebd4235ef566dafe4 on linode:master.

Coverage decreased (-0.6%) to 93.065% when pulling ecd2149b5b8185069ec26c0f71d8d7c0862b2fc6 on eatonphil:master into 78fac4d23750f5c9b57e8a5ebd4235ef566dafe4 on linode:master.

Coverage decreased (-0.7%) to 93.021% when pulling c45fd5fc27257f8fa76ffb91578a133ffd5c0efb on eatonphil:master into 78fac4d23750f5c9b57e8a5ebd4235ef566dafe4 on linode:master.

Coverage decreased (-0.7%) to 93.021% when pulling cdfd56ca9873472ab6f52ea798edff1b956e1bf5 on eatonphil:master into 78fac4d23750f5c9b57e8a5ebd4235ef566dafe4 on linode:master.

Coverage decreased (-0.7%) to 93.021% when pulling f07f978f1b9012fbf127c5aadc7bf26568515036 on eatonphil:master into 78fac4d23750f5c9b57e8a5ebd4235ef566dafe4 on linode:master.

Coverage decreased (-0.7%) to 93.021% when pulling 4cb0561ff30f32dcfa3f2015d04279b3ac599aca on eatonphil:master into 78fac4d23750f5c9b57e8a5ebd4235ef566dafe4 on linode:master.

Coverage increased (+0.4%) to 94.105% when pulling ce3fe86c0cf45ba07458dd3b95eee4ab9ca2ce44 on eatonphil:master into 78fac4d23750f5c9b57e8a5ebd4235ef566dafe4 on linode:master.

Coverage increased (+0.5%) to 94.214% when pulling 5ea69fac4d5f5f76dbe15c6864552f5d4981d0d3 on eatonphil:master into 78fac4d23750f5c9b57e8a5ebd4235ef566dafe4 on linode:master.

Coverage increased (+0.5%) to 94.214% when pulling d573fb4bbdade6cb4cf6a8737b907389152dab55 on eatonphil:master into ba98747955561f2d6285520782995314055b015f on linode:master.

Coverage increased (+0.5%) to 94.214% when pulling d573fb4bbdade6cb4cf6a8737b907389152dab55 on eatonphil:master into ba98747955561f2d6285520782995314055b015f on linode:master.

Ready for review/merge

Coverage increased (+0.5%) to 94.214% when pulling d573fb4bbdade6cb4cf6a8737b907389152dab55 on eatonphil:master into ba98747955561f2d6285520782995314055b015f on linode:master.

Coverage increased (+0.5%) to 94.214% when pulling b7e162968441070c72ab8c4faa70a5ae8c79e25a on eatonphil:master into ba98747955561f2d6285520782995314055b015f on linode:master.

LGTM once I hear some good arguments in favor of formsy-react.

Adding to my response above which may or may not have been sufficient.
Good argument: I was writing a lot of crappy form code and it did a better job and took care of the boilerplate I was starting to write.
Bad argument?: It was easier for me to pick up compared to react-forms.
Are there better libraries out there? Maybe. But it is a huge improvement compared to my first approach (looking up DOM node values) and a decent improvement to my second (potential) approach which would have been to use component state to track form values and I would have had to write onChange boilerplate for every input in every form.

That second approach is probably the strategy I would have used, for what it's worth. this.setState over the alert object with a common onChange event handler for the form fields.

I understand trying to keep down the dependencies in general... But we haven't really taken that philosophy so far. By that I mean this is 1 dependency out of 876. I'm not trying to defend the use in general, but I felt more comfortable offloading this effort into a 3rd-party library. Of course it's all a trade off time for quality for control. Perhaps we can have another cleanup smaller task for keeping the interface and removing the dependency if we get to it?

Most of our dependencies are dev dependencies, not runtime ones. I'm just not convinced that this UI would be messy or hard to write without.

Coverage decreased (-2.2%) to 91.64% when pulling f97ead2e3d848083d20f8c0c532d337b1534c690 on eatonphil:master into 0098ed14e145098410603ce9954b49726d8601f0 on linode:master.

Coverage decreased (-2.1%) to 91.744% when pulling fa59c69ce9425d92fa6ace1b7c989901477e5b61 on eatonphil:master into a379d03889318370ca3b96a4d36db892d5d95d24 on linode:master.

Just got tests passing after rebase. Will put up another commit with no 3rd-party form lib tomorrow morning.

Coverage increased (+0.5%) to 94.336% when pulling 8772e08291835361168a20eea27fe02033edd2f7 on eatonphil:master into a379d03889318370ca3b96a4d36db892d5d95d24 on linode:master.

Coverage increased (+0.6%) to 94.45% when pulling d06a10b8e43d6e4ae66bc76da97056ae1f3ab567 on eatonphil:master into a379d03889318370ca3b96a4d36db892d5d95d24 on linode:master.

👍

:+1: mergable when build passes

Coverage increased (+0.5%) to 94.366% when pulling 891a0ee8b77bf3afce16656e42402e0914365c83 on eatonphil:master into a379d03889318370ca3b96a4d36db892d5d95d24 on linode:master.

Coverage increased (+0.5%) to 94.366% when pulling dfb7c64f990d70a0835d437c0ebd701067942d66 on eatonphil:master into a379d03889318370ca3b96a4d36db892d5d95d24 on linode:master.

Coverage increased (+0.5%) to 94.366% when pulling dfb7c64f990d70a0835d437c0ebd701067942d66 on eatonphil:master into a379d03889318370ca3b96a4d36db892d5d95d24 on linode:master.

Thanks!


Scaling out BeeGFS

How can I add the new storage or metaserver to the cluster?
I have tried to follow official documentation here, e.g. scale out the compute/beegfssm VMSSs, restarted the services but the nodes are still not recognized by the BeeGFS manager.
Am I am missing something?

The following procedure worked for me.

Have existing beegfs (4xL8sv2), but want to increase storage and metadata servers from 4 to 6.
-azhpc-resize beegfssm 6

[hpcadmin@beegfsm beegfs]$ beegfs-check-servers
Management
beegfsm [ID: 1]: reachable at <IP_ADDRESS>:8008 (protocol: TCP)
Metadata
beegfa57e000000 [ID: 1]: reachable at <IP_ADDRESS>:8005 (protocol: TCP)
beegfa57e000004 [ID: 2]: reachable at <IP_ADDRESS>:8005 (protocol: TCP)
beegfa57e000003 [ID: 3]: reachable at <IP_ADDRESS>:8005 (protocol: TCP)
beegfa57e000001 [ID: 4]: reachable at <IP_ADDRESS>:8005 (protocol: TCP)
beegfa57e000006 [ID: 5]: reachable at <IP_ADDRESS>:8005 (protocol: TCP)
beegfa57e000005 [ID: 6]: reachable at <IP_ADDRESS>:8005 (protocol: TCP)
Storage
beegfa57e000001 [ID: 1]: reachable at <IP_ADDRESS>:8003 (protocol: TCP)
beegfa57e000003 [ID: 2]: reachable at <IP_ADDRESS>:8003 (protocol: TCP)
beegfa57e000004 [ID: 3]: reachable at <IP_ADDRESS>:8003 (protocol: TCP)
beegfa57e000000 [ID: 4]: reachable at <IP_ADDRESS>:8003 (protocol: TCP)
beegfa57e000006 [ID: 5]: reachable at <IP_ADDRESS>:8003 (protocol: TCP)
beegfa57e000005 [ID: 6]: reachable at <IP_ADDRESS>:8003 (protocol: TCP)
We can see that 2 extra storage and metadata servers have been added.

It worked. Thanks. However, this is strange that I don't see the performance improvement when doubling the size of beegfsm. I am testing the performance by copying the 24GB folder between two locations: time cp sim sim3 -R
The folder contains ca. 120 directories with several files in each in MB range (2.2M, 119MB, 47MB).
For small and bigger beegfsm I get the same result.
real    2m26.809s
user    0m0.461s
sys     0m29.615s
vs
real    2m32.859s
user    0m0.440s
sys     0m28.253s
IO Pattern: 55k reads, 50k writes, summing up to 90% of execution time.
I also tried to change the chunk_size with beegfs-ctl --setpattern --chunksize=1m --numtargets=8 /beegfs/chunksize_1m_4t to 1m, 64kB and 4m size with 8, 1, 8 targets, respectively.
This did not affect the results much.

Have you tried multiple cp's ? Maybe each cp to a different target. May need to determine if the source data is on 4 storage targets or more. Need to determine if reading or writing is slowing the performance.

Try to maximize the number of disks working on the I/O operation. beegfs-df can help to see what disks/targets are active.

First feedback: This is my first attempt to parallelize cp operation:
for i in {0..N}
do
  cp -r $sourcedir/processor$i/* $destination/processor$i  &
done
wait # wait for cp threads to finish

With this code I was able to reduce the copying time from 1m41sec to 58 secs. Now I will test the same code after doubling the size of the cluster.


html5validator: init at 0.3.3

Motivation for this change
Add new package
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

@FRidh done

Please follow CONTRIBUTING.md and manual#submitting-changes-making-patches  and squash the commits.
This can be done with:
git reset HEAD~<number_of_commits - 1>
git add -- pkgs/
git commit --amend --no-edit
git push ... ... --force

@jonringer suggestions applied


Make Pomodoro Timer

Because focusing for 25 minutes isn’t as easy as it sounds.
Build a Pomodoro timer that follows the Pomodoro technique, breaking work into intervals with short breaks in between.

Assign me @sumanth-0


[TC-CNET-4.4] YAML script has to be removed from the repo

Description:

Python automation script is available for TC-CNET-4.4. So the YAML script has to be removed for the same.

PR
TC-CNET-4.4 - Removes yaml script #36209


pymbar-examples is not available

hi,
I am not sure where I should ask, so I post the issue here (which relates to pymbar + choderalab). I cc @davidlmobley, since you are the corresponding author.
I've read https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4420631/
(Guidelines for the analysis of free energy calculations) and the authors stated that the example are available in https://github.com/choderalab/pymbar_examples
Any idea why you remove it? And where I can see the code?
cheers.

We have nothing to do with this paper or tool.

thanks. So I close it.

I suggest you contact the corresponding author directly:

Contributor Information.
Tel.:<PHONE_NUMBER>, Fax:<PHONE_NUMBER>, Email<EMAIL_ADDRESS>

thanks.

I think I found it: https://github.com/MobleyLab/alchemical-analysis

Great!
@davidlmobley: Do you want to amend the URL in your paper, or do you want us to put a pointer to that URL in the README.md here?

A pointer would be great, @jchodera. (We discussed this previously but I
guess it never happened.)
On Wed, Oct 12, 2016 at 2:30 PM, John Chodera<EMAIL_ADDRESS>wrote:

Great!
@davidlmobley https://github.com/davidlmobley: Do you want to amend the
URL in your paper, or do you want us to put a pointer to that URL in the
README.md here?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/choderalab/pymbar/issues/236#issuecomment-253345448,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGzUYaW1YnWasf2_6OeD_N04L9Sdca9Gks5qzVFkgaJpZM4KVOuB
.

--
David Mobley
<EMAIL_ADDRESS>949-385-2436


String interpolation in media queries

It would be nice to be able to use constants:
`@media (min-width: ${theme.breakpoint.mobileMin})`: {}

Is this possible?

of course ...
export default {
  button: {
    width: 100
  },
  [`@media (min-width: ${theme.breakpoint.mobileMin})`]: {
    button: {
      width: 200
    }
  }
}

its just plain es6 ... has nothing to do with jss)

thanks


Split this page per command

Please split this page into subpages, one per command.
It's really hard to see all the parameters relevant to a given command. Sometimes you have to read the description of a parameter to realize that that parameter has no effect on the command I'm looking at.
For example, 'configuration' works for 'restore', but not for 'build' (there '--configuration Release' must be passed as 'arguments'). And with 'pack' there is 'configurationToPack' for that.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 5825539f-634b-03d6-35f0-b532177a5e3b
Version Independent ID: 5825539f-634b-03d6-35f0-b532177a5e3b
Content: DotNetCoreCLI@2 - .NET Core v2 task
Content Source: task-reference/dotnet-core-cli-v2.md
Service: azure-devops-pipelines
Sub-service: azure-devops-pipelines-tasks
GitHub Login: @steved0x
Microsoft Alias: sdanie

Thank you for your feedback. We are migrating this issue to our new documentation feedback system, and closing this issue.
As we phase out GitHub issues for this repository, we encourage you to propose changes using public PRs or submit feedback through our updated control at the top of the page. For more information on our feedback systems, see https://aka.ms/contentuserfeedback.


Adds service for custom commands

Adds a service call to allow executing custom commands with the ipmi tool.
Only works with the addon/docker for now.
This fixes #41 and #36

@flipper Thank you for your contribution!


added japanese translation

Japanese translation(ja.json)

Hi @toyoh3232, thank you for your contribution and apologies for the delayed review. It looks great, we just have a snapshot failure. Would you mind updating it? Try cd client && npm run coverage -- -u. Thank you!


Use of the word Pseudonymized vs. De-identified

We need to ensure that these terms are defined and consistent within the docs for WpA.   could we please have someone review all the docs and fix this.   Pseudonymized in particular should be defined for the audience who maybe learning about WpA and looking for how we handle this data.   This has been escalated several times to WpATRUST team.

@JeremyTyers - Thanks for your feedback! I'm going to close this issue and ask you for more information via email.


Expanded search icon

Adds expanded search icon. Search icon touch functions the same as the 'Search' soft keyboard button. Search icon only visible if search box has text inside of it.

Hi sahildave. This is my first open source project which I've worked on so please let me know if I'm doing anything wrong on the Github side of things. Still learning about pull requests etc on Github. Let me know if there's something specific you want me to look into!

Merged. Thanks for the pull request and bug fix.


feat(ckb-contracts): add simple bridge typescript

Codecov Report

Merging #141 (4e243ff) into main (1d42644) will decrease coverage by 0.03%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##             main     #141      +/-   ##
==========================================
- Coverage   22.50%   22.47%   -0.04%
==========================================
  Files          20       21       +1
  Lines        4292     4298       +6
==========================================
  Hits          966      966
- Misses       3326     3332       +6

Impacted Files
Coverage Δ

...ib/simple-eth-bridge-typescript-lib/src/adapter.rs
0.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1d42644...4e243ff. Read the comment docs.


SAK-32035 : Replace courier service in chat with some other service

More info in JIRA : https://jira.sakaiproject.org/browse/SAK-32035

LGTM, terrifying work @frasese & @josecebe  !

Updated according previous comments :

Removed TimeService (uses java8 time and preferences service instead)
Using lombok's EqualsAndHashcode
Renamed pdate, ptime, pid with localizedDate, localizedTime, dateID

Just a few more minor questions/suggestions and we can get this one in! Again big thanks to @frasese and @josecebe

Anything else I can do here?


Cython/C bindings

In a nutshell, something similar to the numpy <-> cython bridge, which allows you to "cimport" numpy, giving you typed and direct access to numpy.
In this module, this would mean there is a cython api, through which you work directly with the C structures and types (therefore no conversion between C and python - which is, and probably will always be, the slowest part of this module).
I don't know enough about cython to even know how feasible it is to achieve this, or how much work it is.
I can just say, that in my project I am using cython (gil-less) threads to use multiple CPU cores to then feed data into vulkan, the bottleneck in my case is the conversion from cython/c to python back to c structures.
I know that it is possible to bind to vulkan directly from cython, but then you don't get this modules error checking.

Hello @Berserker66, I didn't think about Cython and I don't know much about it.
In my opinion (but I may be wrong), you can directly use the Vulkan header in Cython (like you say) and by passing the VkInstance and VkDevice, it should work.
What do you mean by modules error checking ?
I don't do error checking in the module, that's the purpose of Vulkan layers.
Can you help me to better understand the issue ?
Thanks

If you use cython directly, you lose this section of features:
https://i.imgur.com/MpHTagU.png
and have to wrap it yourself.
Just - considering this wrapper is already automatically generated, I was kind of hoping it might not be too much work to add a cython output.

Indeed @Berserker66,
Since the model is generated, it's only a matter of jinja2 template to write a new wrapper.
So in theory, it can be done easily, but it must be done by someone who know Cython pretty well.
Do you know someone who could help ?

I don't really know anyone - I use cython a lot though, - but never really messed with C libraries directly. If I find a spare week somewhere I might be able to do it.

If you need information to use the generator. You can talk to me by email (in git commit list) or on IRC. I'm connected to the #vulkan channel and I am realitix

I updated the SDK to 1.0.61

I tried for a bit, didn't get very far.
My first shot is to try and just compile the existing template with cython and see if that already gives a benefit or not - sometimes it does.
Well, first snack the cpp command does not exist on windows and I've looked around a short bit for an alternative and it looks like there are, but then I don't know how to translate some of the cmd line arguments and .. ugh anyway, I just grabbed the already compiled parts to skip over that over now.
I got cython to compile "something". Can't be loaded and it clearly fails in the link stage:
C:\Program Files\Python36\lib\distutils\dist.py:261: UserWarning: Unknown distribution option: 'include_package_data'
  warnings.warn(msg)
C:\Program Files\Python36\lib\distutils\dist.py:261: UserWarning: Unknown distribution option: 'install_requires'
  warnings.warn(msg)
C:\Program Files\Python36\lib\distutils\dist.py:261: UserWarning: Unknown distribution option: 'setup_requires'
  warnings.warn(msg)
running build_ext
building 'vulkan.__init__' extension
E:\Visual Studio 2017 Community\VC\Tools\MSVC\14.11.25503\bin\HostX64\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD "-IE:\Visual Studio 2017 Community\VC\Redist\MSVC\14.10.25017\onecore\x64\Microsoft.VC150.CRT" "-IC:\Program Files\Python36\include" "-IC:\Program Files\Python36\include" "-IC:\Program Files (x86)\Windows Kits\10\Include\10.0.15063.0\ucrt" "-IC:\Program Files (x86)\Visual C++ BuildTools\VC\Tools\MSVC\14.11.25503\include" "-IC:\Program Files (x86)\Windows Kits\10\Include\10.0.15063.0\shared" /Tcvulkan\__init__.c /Fobuild\temp.win-amd64-3.6\Release\vulkan\__init__.obj
__init__.c
E:\Visual Studio 2017 Community\VC\Tools\MSVC\14.11.25503\bin\HostX64\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO "/LIBPATH:C:\Program Files\Python36\libs" "/LIBPATH:C:\Program Files\Python36\PCbuild\amd64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\Lib\10.0.15063.0\ucrt\x64" "/LIBPATH:E:\Visual Studio 2017 Community\VC\Tools\MSVC\14.11.25503\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\Lib\10.0.15063.0\um\x64" /EXPORT:PyInit___init__ build\temp.win-amd64-3.6\Release\vulkan\__init__.obj /OUT:E:\vulkan\vulkan\__init__.cp36-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.6\Release\vulkan\__init__.cp36-win_amd64.lib
LINK : error LNK2001: unresolved external symbol PyInit___init__
build\temp.win-amd64-3.6\Release\vulkan\__init__.cp36-win_amd64.lib : fatal error LNK1120: 1 unresolved externals
error: command 'E:\\Visual Studio 2017 Community\\VC\\Tools\\MSVC\\14.11.25503\\bin\\HostX64\\x64\\link.exe' failed with exit status 1120

Now, googling for this particular error seems to indicate that cython really does not like */.init files and you should really put your stuff elsewhere. Which might be a good idea for the base vulkan implementation too, dropping it all in init is somewhat odd. We can still get the same behavior by importing * into the init. Anyway, I made a fork of the project and my progress for today is on there.

I don't know if you guys heard of pybind11.
https://github.com/pybind/pybind11
I'm not a C\C++ programmer but I tired it few days ago.

#include <vulkan\vulkan.h>
#include <pybind11\pybind11.h>
#include <pybind11\stl.h>

namespace py = pybind11;

class Instance
{
public:
	Instance();
	~Instance();
	bool createInstance();

private:
	VkInstance instance;

};

Instance::Instance()
{

}

Instance::~Instance()
{
	vkDestroyInstance(instance, nullptr);
}

bool Instance::createInstance()
{
	VkApplicationInfo appInfo = {};
	appInfo.sType = VK_STRUCTURE_TYPE_APPLICATION_INFO;
	appInfo.pApplicationName = "Test vk";
	appInfo.applicationVersion = VK_MAKE_VERSION(1, 0, 0);
	appInfo.pEngineName = "no engin";
	appInfo.engineVersion = VK_MAKE_VERSION(1, 0, 0);
	appInfo.apiVersion = VK_API_VERSION_1_0;

	VkInstanceCreateInfo createInfo = {};
	createInfo.sType = VK_STRUCTURE_TYPE_INSTANCE_CREATE_INFO;
	createInfo.pApplicationInfo = &appInfo;
	createInfo.enabledExtensionCount = 0;
	createInfo.enabledLayerCount = 0;

	auto result = vkCreateInstance(&createInfo, nullptr, &instance);
	if (result == VK_SUCCESS)
		return true;
	else
		return false;
}

void testList(const std::vector<const char*> &l) {
	for (auto i : l)
		std::cout << i << "\n";
}

PYBIND11_MODULE(pyvk, m) {
	m.doc() = "vulkan python binding.";

	py::class_<Instance>(m, "Instance")
		.def(py::init<>())
		.def("createInstance", &Instance::createInstance);

	py::enum_<VkStructureType>(m, "StructureType")
		.value("STRUCTURE_TYPE_APPLICATION_INFO", VkStructureType::VK_STRUCTURE_TYPE_APPLICATION_INFO)
		.value("STRUCTURE_TYPE_INSTANCE_CREATE_INFO", VkStructureType::VK_STRUCTURE_TYPE_INSTANCE_CREATE_INFO)
		.value("STRUCTURE_TYPE_DEVICE_QUEUE_CREATE_INFO", VkStructureType::VK_STRUCTURE_TYPE_DEVICE_QUEUE_CREATE_INFO)
		.export_values();

	m.def("mylistfunc", &testList);
}

Thanks for sharing.
Can you tell me what's the diff. btw pybind11 and cffi?
Also, how does C++11 compare with C, which Vulkan is written in?
Thanks.
On 05/10/2017 17:41, mack stone wrote:

I don't know if you guys heard of pybind11.
https://github.com/pybind/pybind11
I'm not a C\C++ programmer but I tired it few days ago.
#include  <vulkan\vulkan.h>
#include  <pybind11\pybind11.h>
#include  <pybind11\stl.h>
namespace  py  =  pybind11;
class  Instance
{
public:
Instance();
~Instance();
bool  createInstance();
private:
VkInstance instance;
};
Instance::Instance()
{
}
Instance::~Instance()
{
vkDestroyInstance(instance,nullptr);
}
bool  Instance::createInstance()
{
VkApplicationInfo appInfo = {};
appInfo.sType  = VK_STRUCTURE_TYPE_APPLICATION_INFO;
appInfo.pApplicationName  ="Test vk";
appInfo.applicationVersion  =VK_MAKE_VERSION(1,0,0);
appInfo.pEngineName  ="no engin";
appInfo.engineVersion  =VK_MAKE_VERSION(1,0,0);
appInfo.apiVersion  = VK_API_VERSION_1_0;
VkInstanceCreateInfo createInfo = {};
createInfo.sType  = VK_STRUCTURE_TYPE_INSTANCE_CREATE_INFO;
createInfo.pApplicationInfo  = &appInfo;
createInfo.enabledExtensionCount  =0;
createInfo.enabledLayerCount  =0;
auto  result =vkCreateInstance(&createInfo,nullptr, &instance);
if  (result == VK_SUCCESS)
return  true;
else
return  false;
}
void  testList(const  std::vector<const  char*> &l) {
for  (auto  i : l)
std::cout << i <<"\n";
}
PYBIND11_MODULE(pyvk, m) {
m.doc() ="vulkan python binding.";
py::class_(m,"Instance")
.def(py::init<>())
.def("createInstance", &Instance::createInstance);
py::enum_(m,"StructureType")
.value("STRUCTURE_TYPE_APPLICATION_INFO", VkStructureType::VK_STRUCTURE_TYPE_APPLICATION_INFO)
.value("STRUCTURE_TYPE_INSTANCE_CREATE_INFO", VkStructureType::VK_STRUCTURE_TYPE_INSTANCE_CREATE_INFO)
.value("STRUCTURE_TYPE_DEVICE_QUEUE_CREATE_INFO", VkStructureType::VK_STRUCTURE_TYPE_DEVICE_QUEUE_CREATE_INFO)
.export_values();
m.def("mylistfunc", &testList);
}
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/realitix/vulkan/issues/32#issuecomment-334414272,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AWWLzu5XacPx33YuzN8v2fuWDNGIJKsVks5spKRVgaJpZM4Pn3N_.

Can you tell me what's the diff. btw pybind11 and cffi?

I think the main diff is pybind11 can very easy to create Object-oriented code in C++. cffi can't do that in C correct me if I'm wrong. With pybind11 all your code write in C++. cffi can be python or C+python.
Above using in python
import pyvk
ins = pyvk.Instance()
ins.createInstance()
>>> True

Also, how does C++11 compare with C, which Vulkan is written in?

I'm not a C\C++ programmer. Can't answer that.

Hello,
Please open another issue if you need to talk about another subject. This issue is about Cython.
Thanks.

So, I dug some more into this. After I assembled some more experience assimilating a simpler C library with cython. Happened to be this one: https://github.com/smcameron/open-simplex-noise-in-c
Some things I learned -
The speed advantage is worthwhile, compared to python's opensimplex
https://pypi.python.org/pypi/opensimplex/0.1
it is over a magnitude faster and multithreads very well.
So, that is the pro, which was the expected outcome.
The con:
I don't see how it could ever integrate with the current API.
If you define a cython function, you do so with C types as input and output. As a result, it does not accept CFFI objects, you'd need to peel off the cffi wrapping every time you want to interact between the two and you'd need to cffi wrap anything you return from cython. This would likely smash any hopes of faster performance.
The other way, which is also awkward, is to wrap the cython functions; which are also accessible from the C layer with cffi.
Cffi was however never meant to access the python C layer, in its own documentation you can find that you will likely mess up reference counting and bleed memory all over the place.
So, bottomline -
It is certainly possible to make a cython interface. I don't see a good way to make cython and cffi interact in a reasonable fashion however, so both interfaces would be exclusive to each other, you can use either one or the other, possibly with converter functions to wrap/unwrap with cffi to share a few objects, like logical_device and stuff like that.

Thanks @Berserker66 for this nice explanation.
It would be exclusive but do you see a use case where you would want to use both ?

Well, my own project would be such a use case. I can't vouch for how many other projects there  might be.
First off, there is the use case of having a replacement/alternative to the cffi module. Mostly useful for CPython. Where, as far as python and the "application programmer" is concerned the API is the same, basically a cython api that mimics the current module completely. Such a module should perform faster on CPython, probably not on PyPy. Most importantly, it could release the GIL earlier. Let's look at a trace of my program, to see where that could be useful
Let's get some actual data in here.

You may need to download the file and zoom in, I doubt github handles it well.
So this is a multithreaded environment. So 100% time spent just means that there is always at least one thread spending time in there. Almost all bottlenecks in my project release the GIL, so as a result they quickly exit.
We can see that such a 100% time spent functio nis _callAPI. to be fair, we are also spending 109 ms in vkWaitForFences, which this profiler sees as cpu time spent. (I hope it doesn't actually do that, maybe someone should investigate).
A cython version of such a function would still need the gil for parts of it - to decrease the reference count on the incoming variables, but past that it can assemble the vulkan structures, call into vulkan and almost completely check the output and assemble it without the GIL and so so in C code, as opposed to using the cffi library.
I expect a decent, but not mind blowing speed advantage here.
More importantly, a full cython-c API, that you call from Cython user-code, could just never use the GIL (except to generate exceptions). Completely unlocking the multi-threaded potential of vulkan.
To be fair the cffi version could also be improved, by not handling every argument the same, but having a handler for each argument and in the parser step of this module, dropping all the handlers into the function. Completely erasing the dict comprehension in _call_api and also removing lots of if statements.

Hello @Berserker66 , I close this issue for now.
Maybe someone would create a Cython interface one day !


Correct the version

Latest improvements:
MSFT employees can try out our new experience at OpenAPI Hub  - one location for using our validation tools and finding your workflow.

Contribution checklist:

[ ] I have reviewed the documentation for the workflow.
[ ] Validation tools were run on swagger spec(s) and have all been fixed in this PR.
[ ] The OpenAPI Hub was used for checking validation status and next steps.

ARM API Review Checklist

[ ] Service team MUST add the "WaitForARMFeedback" label if the management plane API changes fall into one of the below categories.
adding/removing APIs.
adding/removing properties.
adding/removing API-version.
adding a new service in Azure.

Failure to comply may result in delays for manifest application. Note this does not apply to data plane APIs.

[ ] If you are blocked on ARM review and want to get the PR merged urgently, please get the ARM oncall for reviews (RP Manifest Approvers team under Azure Resource Manager service) from IcM and reach out to them.
Please follow the link to find more details on API review process.

Automation for azure-sdk-for-python

Automation for azure-sdk-for-go

@hitenjava, please help review this since your are the code owner of this one.

Can one of the admins verify this patch?

@ash1625, could you help review this change? Thanks.