turn pseudobulk from scRNAseq with logCPM value back to a seurat object with counts value?

May i ask 1 question:
if data is already pseudobulk object from scRNAseq data with logCPM value, how can i change it back to a seurat object with counts value? Can i still use above method to turn data back to a seurat object?
My data is normalized to become a pseudobulk data as following:
"Normalizing count data
After excluding poor quality cells, we normalized the sequencing depth of each cell by dividing
each cell’s counts by the total counts in that cell, resulting in a matrix where the entries represent
the proportion of a cell’s reads allocated to each gene (i.e. values in the range [0,1]). To estimate
a library size for each dataset, we summed the total counts in each cell, and then we took the
median as the library size for the dataset. Next, we multiplied the proportions by the library size
to get a count matrix that was normalized for sequencing depth. Finally, we transformed the
normalized count matrix with log2(1 + count). We referred to this log-transformed quantity in the
figures as log2CPM.
We created pseudobulk expression (L. Lun, Bach, and Marioni 2016) for the cells in a
cluster for each donor such that the pseudobulk matrix had one row for each gene and one column
for each cluster from each patient.
We normalized the pseudobulk counts to log2CPM as
described in the previous section. Then we use limma::lmFit() to test for differential gene
expression with the log2CPM pseudobulk matrix (Ritchie et al. 2015). We also use
presto::wilcoxauc() to compute the area under receiver operator curve (AUROC or AUC) for the
log2CPM value of each gene as a predictor of the cluster membership for each cluster
(Korsunsky, Nathan, et al. 2019).
"
thanks best wishes
J.

How can i assign cell subtype to cluster defined by this pseudobulk matrix with logCPM value, after subset major celltypes defined by this pseudobulk matrix? Could i use markers expression profile in different clusters to redefine cell subtypes? thanks lot! best wishes,jiang

Hi, apologies on the delay on this, not sure how I missed it.
I'm not sure of the best approach to go from pseudobulk to single nuclei - you will lose some information given you won't be able to work out what counts belong to what cell from the aggregated counts per individual so you would end up just having separate cells with an averaged value which I don't think will work well. The raw count matrix with metadata from our analysis is available through Synapse so you shouldn't need to convert from pseudobulk to raw counts.


Windows binaries?

If its possible, please.

Sure, since CI tool was migrated to Github Actions should be not a problem. Stay tuned.

Sure, since CI tool was migrated to Github Actions should be not a problem. Stay tuned.

Released on v1.14.0
Feedback welcome.


Improve speed of generate_index_dates

Currently, the new generate_index_dates action is extremely slow to run on dummy data for anything N > 20,000. I am conscious how long this action could take to run on real data. Potentially worth trying to rewrite this script using data.table to see if it speeds things up?

There is a data table back-end for dplyr if you don't want to do a full rewrite?

To clarify, I am happy for stuff to be written in data.table but I know many of our team members are big tidyverse fans!

@venexia the script runs locally in RStudio in a matter of seconds. Do you know of any reasons why it could be taking so long when executing the script via OpenSAFELY?

Issue addressed in this PR in the GI repo. Changes to be ported over.


feature request: an modern android data/time picker ?

first of all thx for the plugin - its in action in the ng-cordova wrapper for the ionic framework:
https://github.com/driftyco/ng-cordova/commit/d59a5881ae84ec1ebe92dd5d82ae3c3662c7dc54

it would be niece with a more modern android picker:

this is lined out in http://developer.android.com/guide/topics/ui/controls/pickers.html

When I started developing my application I created a Multi-Device Hybrid App in Visual Studio 2013 and @joshbuckley182 solution worked like a charm. Later I migrated my project to the Telerik Platform and it stopped working. I tried the @vosManz  solution and it worked. I had to add another change tho.
I changed line 131/132
final DatePickerDialog dateDialog = new DatePickerDialog(currentCtx, dateSetListener, mYear, mMonth, mDay);
To:
final DatePickerDialog dateDialog = new DatePickerDialog(currentCtx, DatePickerDialog.THEME_DEVICE_DEFAULT_LIGHT, dateSetListener, mYear, mMonth, mDay);
Now the datepicker dialog and the timepicker dialog theme is set.

Thanks a lot guys.
Your solution in my fork repository: https://github.com/loiclefloch/cordova-plugin-datepicker.
Test on Android 4.2.2 and 5.1
@vosManz there is actually a pull request to change to 24h picker.

@loiclefloch Your fork looks great, care to turn it into a pull request?

@vbraun with pleasure. But in my fork, I also add the https://github.com/VitaliiBlagodir/cordova-plugin-datepicker/pull/68 pull request. Should I remove that and just put the new picker design ?

Any news about the merge ? without the #68 pull request

nothing above worked for my angular ionic app. I ended up altering the plugin code directly:
platforms > android > platform_www >  plugins > cordova-plugin-datepicker > www > android > DatePicker.js
There you can see where the theme is being defaulted to THEME_TRADITIONAL, and can change it to THEME_HOLO_LIGHT or THEME_DEVICE_DEFAULT_LIGHT.


Ghosting

I am having a real issue getting past the ghosting when commenting on posts. We're talking about NON spam posts, completely on topic with the current conversation and it just vanishes like a shadow ban, yet the account is still fine for real browser session posts. I've tried routing and binding through legit residential IP space, proxies, user behaviour movements, amazon aws IPs, new accounts old accounts, nothing is working. I must be missing something. Any tips you may have tried that I haven't would be greatly appreciated my friend. This is messing with my day LOL.
Cheers,
wuzzy

After dumping the page I can now see that the comment text box just never appears, no matter how long I wait and how much "user behaviour" I do...  I am 100% logged in. So that would explain the issue. Hmmmmm

It was clearing the browser data. I thank you for your code it really helped. Question, how come you're not using **kwargs for your bot init, that way if you're utilizing threading on 1000s of accounts it would make the process I think much smoother. Either way, great job.

Hey there, thanks for reaching out )
Glad you figured everything out. I guess there are not that many args yet so explicitly using them does not seem problematic (just yet). As for threading, I thought about integrating threading but that messes with selenium sometimes. For example focusing on another window will execute interactions from another thread, so you would need multiple chromium instances open which 1st isn't optimal for hardware performance, 2nd Reddit spots multiple accounts logged in at the same time from the same IP address. And while there are solutions to possibly any problem, this worked for me and wasn't highly requested (lol), so I called it a day as is.
If you feel like there is a thing that will certainly improve the code, let me know, or open a PR.
Cheers mate

P.S. Check out this discussion if you've got the time


Report Grid: cannot remove filter (but can disable it)

Description
More of an annoyance than a breaking bug.
Clicking the 'x' to remove a filter (or toggling from the filter list) no longer works.
Worse, the selected filters are remembered even when switching between grids.
Waltz Version
1.48.1
Steps to Reproduce

...
Expected Result
No response
Actual Result
No response

fixed as part of MVA work (#6359)


Document tasks

Need
This is a collection of pages that describe some useful procedures:

how to organise an event
how to reserve a room for an event
how to promote an event

The pages should be editable and accept markdown for easy writing.
Deliverables

[ ] add db schema for Pages: id, name, content, last_edited_by
[ ] add page to manage Pages: list all, add page, edit page
[ ] include links to Pages when adding events

This page offers short lists to tasks that need to be done in certain situations. E.g.:

how to organise an event
how to reserve a room for an event
this seems out of scope and not related with the TODO list.

One referred the implementation and the other the objective. I have updated correspondingly.

We have a wiki on GitHub were we strived to document all of these things. I think we should use that and add a link to the wiki page to the project, which can be seen by rosedu members.
This issue and the one with doodle are reinventing the wheel, which is already built by others (and better than we can do it).

Don't you need some different access rights to edit those wiki pages ?

You need to be a ROSEdu member ( and this also includes people who are
supporters), hence have access to the Github organisation.
On 11 March 2015 at 10:51, Marius Coțofană<EMAIL_ADDRESS>wrote:

Don't you need some different access rights to edit those wiki pages ?
—
Reply to this email directly or view it on GitHub
https://github.com/rosedu/events/issues/33#issuecomment-78225309.

That works for me. I have added the corresponding page.


app-router integration

My app component contains my UniFlow dispatchers. It also contains my <app-location> and some of my <app-route> components. In Where to call server API, google/uniflow-polymer#1, we determined that we could emit a save action from a page such as a Profile Edit page. Some dispatcher would take care of calling the back end server API to save the data and emit its own action for success or failure. Another dispatcher (or the same) would receive the success action and handle routing.
How do you suggest setting that up? Would you pass the route into the dispatcher and put an <app-route> component in the dispatcher to be manipulated?
<my-app-dispatcher state="{{state}}" route="{{route}}"></my-app-dispatcher>

Then in <my-app-dispatcher> have a <app-route route="{{route}}"> in the template section so that it can be manipulated in the action methods that need to handle routing?
(I'll try this myself but) Can an observer be used to watch a specific portion of the state tree? For my application, my main app needs to know when the user is logged in or not and route to either the login screen or, depending on the logged in user's roles, a specific page. Can I use a simple
observers: [
  'onLoggedIn(state.auth.user.token)'
]

Is that the best method to listen to different branches of the state tree? In a lot of cases the state-path will be a section of the tree specific to the page but the page will still need access to other sections of the tree such as the current user's information.

A few quick tests of using observers show that you must observe a specific value in the state tree and not an entire branch. If I have a tree state.auth.user then:

onStateChanged(state) : Does not get called when state.auth.user changes.
onStateChanged(state.auth) : Does not get called when state.auth.user changes.
onStateChanged(state.auth.user) : DOES get called when state.auth.user changes.
This should work for my scenario but thought the information might be helpful to someone trying to figure out why their observer doesn't get called.

I also tested adding a state property with its own observer and it did not get called.

I have not used app-route in production app, so might not know all the details about it. However I don't see a reason to put  inside . I would simply make route part of the state (state.route) which would give the ability to manipulate it from the dispatcher.

This does work. I had to do two things to make it work, maybe because of the order things get created, not sure?
First I updated <app-location id="location" route="{{state.route}}">. That alone does not do it. state.route does not get updated. I also had to set state.route to the app-location route object like so:
ready: function() {
  this.set('state', {
    route: this.$.location.route
  });
},

I tried setting state.route to {} in ready but that doesn't work.

That makes sense. <app-location> uses two-way binding for route, which makes it a bit difficult to use in uniflow pattern (in which elements either listen to state changes or mutate state, but never both). Do you have your app on GitHub somewhere for me to take a look and see if I can suggest something for your specific case?

Also using chained routers (passing tail to route of component) we are able to only manage the portion of the URL we are displaying and don't need to worry about where it was called from in the URL chain making it easier to use the same component that has routing from multiple locations in the app.
The app is closed source so I don't have something to show you. I'm going to test not using state in app-router and also not passing the routing information to the dispatchers. That won't work either since the routers need to be chained.
Instead, I'm going to use the emitAction to save data and then use the standard fire method from the action to let my components know to change route if needed. In the original SAVE_USER example that would mean the component's save method would simply validate the form and emit a SAVE_USER message with the model. It (or the parent of it) would listen for a standard 'user-saved' message and handle routing.

I've just done an UniFlow version of the Polymer Starter Kit, where I try to do  the routing operations using actions.
All feedback is welcome...

Looks good. Where I'd want to see more is a link in view1 to view2, a button in view2 to view3, and also a use of the BACK action. I use a type of BACK action in my app when someone hits cancel on some of my pages. Nice that you don't need to worry where exactly they came from when they hit Cancel, especially when that page is called from multiple locations.
The other issue I ran into was nested app-route components. So if view1 had an iron-pages on it and used a route that was passed in from the app's app-route tail property to determine which page to show. How can that be handled with UniFlow cleanly. I'm sure it can, just need something like that level of sophistication in the starter app so people don't hit a roadblock on day 2. So how would /view1/page1 and /view1/page2 get handled.
My current non UniFlow solution differs by returning a promise for its emitAction equivalent. That allows me to do some routing right from the component. Emit save, wait for success, then fire a standard event like this.fire('save-clicked') for higher components. I tried to fire events from my action dispatcher but since it is at the root of the application, the event didn't go anywhere. Events bubble up, not down. I had to pass in the calling element which seemed dirty. With a promise in UniFlow you could call the emitAction CHANGE_PAGE event from the component rather than from the action dispatcher. Of course the initial emitAction to SAVE could include an onSuccessPage value that could be passed to CHANGE_PAGE on success. So many ways to do it :-)

@LostInBrittany Thanks a lot for your work! I made some comments here: code review, but overall looks great as a starting point. I would be happy to contribute by adding more non-trivial scenarios on how UniFlow can be used, but the problem is as usual: finding time to do that. So can't promise anything at this time, but will keep this in mind in case I'll have some spare cycles.

I have also had a go at Polymer 2.0 + Uniflow + app-location, app-route integration. I wanted changing the URL in the address bar,  links to work also and to have some state change with the page (in the real app it would be a rest call to pull back data for the page).
It sort of works but I've come across an issue, it seems that when the app loads (first time, and when the URL is changed) the emit of the page event is firing before the action-dispatcher is ready to receive the event although the code works fine when triggered in other situations without a full app reload. Is there a way to prevent emits of actions until the dispatcher is ready?
Could be my understanding of either polymer or uniflow though, but I'd appreciate some help as I can't see how to fix it.
I did look at what @LostInBrittany had done but I couldn't see how it would do what I wanted.
Repo here -  https://github.com/RobMaskell/PolymerSandbox


Added code for inversion count in a permutation using BIT

Description: Added inversion count in a permutation using BIT
Checklist

[x] I've followed the Contributing guidelines provided in the repository.
[x] I've made the changes which were demanded in the linked issue.
[x] My code gave a clean output on debugging. (NO warnings/errors)

Mention Closes #issue_number in the comment.


Access to flags that indicate which ad types are supported.

Games should not show rewarded ad buttons for instance when the active plugin doesn't support it

Fixed in 96ff3c713d3b52681319ea8cd37e124c33885753


Verify/Lint Markdown files in a proper way

E.g. right now there are multiple #/first-level headlines, which should be avoided.
To avoid something like this, the idea could be to lint all markdown files. I would recommend to use the following tool: https://github.com/DavidAnson/markdownlint-cli2. The authors of this tool also provide a GitHub action, which could validate the markdown files on a PR base 😉

you may provide a PR for this

Of course 🙂 I will try to provide tomorrow a PR for this


unexport is not backwards-compatible

Was playing with the new unexport syntax for vim-just and found some weird-looking backwards-incompatible behavior:
# In previous versions of just,
# these would create a recipe called "unexport" with parameters "QT_QPA_PLATFORMTHEME" and foo.
# In current just, they unexport QT_QPA_PLATFORMTHEME and create a recipe "foo"
unexport QT_QPA_PLATFORMTHEME \
foo:
	echo Not Foo
unexport QT_QPA_PLATFORMTHEME foo:
	echo Also Not Foo

# just 1.29.0 accepts this line as if it were the two lines
# unexport QT_QPA_PLATFORMTHEME
# and
# foo := 'bar'
# but shouldn't this actually be an error, as it is in previous just versions?
unexport QT_QPA_PLATFORMTHEME foo := 'bar'

# previous versions of just allowed "unexport" to be a recipe name or variable name
# these are now an error, even though they don't conflict with the documented unexport syntax
unexport := 'foo'

unexport:
	echo Unexport Recipe

unexport blah:
	echo {{quote(blah)}}

Thanks for reporting this! Fixed in 1.28.1, which I just released.


PromEx.Plugins.Broadway assigns "acknowledger" name to message metrics tag rather than module name

From
https://github.com/akoutmos/prom_ex/blob/3557e828cf3905adaf6719003e68b65294b62288/lib/prom_ex/plugins/broadway.ex#L248
This populates the name tag with the module name of my producer module, eg. in my code App.Producers.TestProducer
It conflicts with the expectation the dashboard has, that message metrics are reported with the name of the Broadway module, eg. in my example code this is filtered with the value App.Broadway and shows no data.

I don't know if there is a different idea at play here than matching the metrics with the processing module, but it seems that the same information is not readily available. Does this need a PR in Broadway to include additional information?
The same issue is present for exception info;
https://github.com/akoutmos/prom_ex/blob/3557e828cf3905adaf6719003e68b65294b62288/lib/prom_ex/plugins/broadway.ex#L347
Edit: I see the batch tag extraction gets the topology_name - is there a way to get the same passed to the messages metrics handler?

Hey Dennis!
I'll take a look at this either tonight or tomorrow and get back to you. I am almost positive that the example app in the repo is working correctly....so I am curious what is going on in your case.

Thanks, I will check the example repo and report back (after this weekend)
On Wed, 2 Feb 2022, 18:23 Alexander Koutmos, @.***>
wrote:

Hey Dennis!
I'll take a look at this either tonight or tomorrow and get back to you. I
am almost positive that the example app in the repo is working
correctly....so I am curious what is going on in your case.
—
Reply to this email directly, view it on GitHub
https://github.com/akoutmos/prom_ex/issues/114#issuecomment-1028172941,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AC2MWEJ7GXTSXOYEE4FVVZDUZFSBNANCNFSM5NJPODKQ
.
You are receiving this because you authored the thread.Message ID:
@.***>

Hi again @akoutmos, you're right, the example applications have the answer. What makes it work is specifically assigning the acknowledger for the pipeline, eg. lines
https://github.com/akoutmos/prom_ex/blob/d4346eb0ca87ffba5276ffaf0723b8f7c4f5b4fd/example_applications/web_app/lib/web_app/temp_processor.ex#L14
and
https://github.com/akoutmos/prom_ex/blob/d4346eb0ca87ffba5276ffaf0723b8f7c4f5b4fd/example_applications/web_app/lib/web_app/temp_processor.ex#L63
It should probably be documented or pointed to as the implementation method. I'll add the fix to my own code and confirm it works, and make a PR with a line in the documentation for the Broadway plugin. Would you prefer in the docs to just point to the example application, or to provide the steps that are needed?

@akoutmos Getting back to my code and prom_ex integration here and I'm finding a potential trap with this transform addition that skips all acks back to the original producer.
Not sure how this is missed, but the impact on the standard Broadway producers is huge, fx. it would result in messages not being dequeued from RabbitMQ (from my reading of the source code apply_ack_func would never be called?) I haven't tested with any actual producers - but you must have some experience here. Maybe I'm way off.
However, working through it, I have made the passthrough ack below as a solution that uses the Broadway ack semantics to group correctly back to the original producer(s). It should be a permanent fix, but it does bloat the code somewhat, and needs comments to explain whys:
def tranform(%Message{data: data, acknowledger: {ack_module, ack_ref, ack_data}}, _opts) do
  # Updating the acknowledger module is needed for PromEx tracking, but we preserve
  # the original and ensure we ack our messages to it in the `ack` function.
  %Broadway.Message{
    data: data,
    acknowledger: {__MODULE__, {ack_module, ack_ref}, ack_data}
  }
end

@impl Broadway.Acknowledger
def ack({ack_module, ack_ref}, successful, failed) do
  restore_original = fn msgs ->
    Enum.map(msgs, fn %Broadway.Message{acknowledger: {_, _, ack_data}} = x ->
      put_in(x.acknowledger, {ack_module, ack_ref, ack_data})
    end)
  end

  # Pass messages on to the producer acknowledger. As we are grouping by the
  # ack_module and ack_ref, we can simply send successes and failures.
  ack_module.ack(ack_ref, restore_original.(successful), restore_original.(failed))
end

I'd like your input on how come this hasn't been an blocker for the standard Broadway producers, and what we should do, if anything? Potentially this code could be dropped into a PromExAcknowledger, making it a bit easier to employ and document usage for.
I can put in some time to do what is needed.
Thanks!

works! Thanks you again.

Potentially this code could be dropped into a PromExAcknowledger, making it a bit easier to employ and document usage for.

👍 from my side

Hi @dsschneidermann ! There's an exception while using PromEx.Plugins.Broadway with Broadway Kafka and adding the transform/2 function.
The exception is:
[error] GenServer :"condition_sync_producer_consumer.Broadway.Producer_0" terminating ** (KeyError) key :topic not found in: %{}     (broadway_kafka 0.3.0) lib/broadway_kafka/producer.ex:557: anonymous fn/2 in BroadwayKafka.Producer.build_allocator_spec_and_consumer_entry/5     (broadway 1.0.3) lib/broadway/topology.ex:228: anonymous fn/3 in Broadway.Topology.build_producers_specs/2     (gen_stage 1.1.2) lib/gen_stage/dispatchers/partition_dispatcher.ex:258: GenStage.PartitionDispatcher.split_events/4     (gen_stage 1.1.2) lib/gen_stage/dispatchers/partition_dispatcher.ex:243: GenStage.PartitionDispatcher.dispatch/3     (gen_stage 1.1.2) lib/gen_stage.ex:2322: GenStage.dispatch_events/3     (gen_stage 1.1.2) lib/gen_stage.ex:2140: GenStage.handle_noreply_callback/2     (stdlib 3.17) gen_server.erl:695: :gen_server.try_dispatch/4     (stdlib 3.17) gen_server.erl:771: :gen_server.handle_msg/6 Last message: {:poll, {1302, "statuses", 0}} State: %{consumers: [{#PID<0.3407.0>, #Reference<0.3463651493.1223163905.98119>}, {#PID<0.3406.0>, #Reference<0.3463651493.1223163905.98112>}], module: BroadwayKafka.Producer, module_state: %{acks: %{{1302, "statuses", 0} => {[], 22476728, []}}, allocator_names: {0, [:"Elixir.condition_sync_producer_consumer.Allocator_processor_default"], [:"Elixir.condition_sync_producer_consumer.Allocator_batcher_consumer_default"]}, buffer: {[], []}, client: BroadwayKafka.BrodClient, client_id: :"Elixir.condition_sync_producer_consumer.Broadway.Producer_0.Client", config: %{client_config: [connect_timeout: 60000, ssl: true, sasl: {:scram_sha_512, _, _}], fetch_config: %{}, group_config: [offset_commit_policy: :commit_to_kafka_v2, rejoin_delay_seconds: 5], group_id: "condition-sync", hosts: [{_, _}, {_, _}], offset_commit_on_ack: true, offset_reset_policy: :earliest, receive_interval: 2000, reconnect_timeout: 1000, topics: ["statuses"]}, demand: 20, group_coordinator: #PID<0.3400.0>, receive_interval: 2000, receive_timer: #Reference<0.3463651493.1223163905.99911>, reconnect_timeout: 1000, revoke_caller: nil, shutting_down?: false}, rate_limiting: nil, transformer: {MyApp.Workers.ConditionSyncProducerConsumer, :transform, []}}
Apparently, the BroadwayKafka.Producer needs some message.metadata keys, like :topic. But, when we run the transform/2 function, we create a new %Broadway.Message{} struct and do not copy the metadata from original message. Therefore, message.metadata is a empty map %{}.
So, I changed to:
def transform(%Message{data: data, metadata: metadata, acknowledger: {ack_module, ack_ref, ack_data}}, _opts) do
  # Updating the acknowledger module is needed for PromEx tracking, but we preserve
  # the original and ensure we ack our messages to it in the `ack` function.
  %Broadway.Message{
    data: data,
    metadata: metadata,
    acknowledger: {__MODULE__, {ack_module, ack_ref}, ack_data}
  }
end

and the exception was fixed.
Please, let me know if I made some mistake.

@marcospgsilva there is a simpler, general way to solve this and other similar issues that may arise: Instead of selecting the fields of the Message struct that you want to copy into the new message, just copy the entire original message and only update the acknowledger field. This is what I do:
def transform(%Message{acknowledger: {ack_module, ack_ref, ack_data}} = message, []) do
    %Message{message | acknowledger: {__MODULE__, {ack_module, ack_ref}, ack_data}}
end

And you're good to go. Whatever fields the producer might expect to be in the message, they'll be there.

@maxmarcon Thank you! I'll try your solution.

@akoutmos Reopen this to welcome a PR?

I have reopened the issue. Any PRs are welcome that address this. I think the best place to start would be to add RabbitMQ and Kafka to the example app in the PromEx repo to have something that tests this E2E. It looks like the GenStage backed Broadway modules that I put together do not replicate the issues that people are encountering with RabbitMQ and Kafka.

@maxmarcon Please do, I've had to leave my Elixir project on the backburner for now.

I took a look at the code and I believe that this is no longer an issue since this commit: d76956385c5622096c88b33545c3998f7cc16de8. Before this commit, the pipeline's name was being extracted from an overwritten acknowledger. Since this commit, the name is extracted from the broadway event's metadata, which is the simple and obvious way of doing it. Overwriting the acknowledger just to make the pipeline's name available was an (unnecessary) hack. The problem is, after this commit the documentation wasn't been updated, and it stills says one should overwrite the acknowledger.
I'll verify the above and then create a PR to update the docs.


Could you add ticker sound for  picker like ios picker

Please add ticker sound for picker like ios native picker.

Closing this issue as it does not follow the issue template. Please file a new issue that follows the template, which gives the team the information needed to investigate.


How plug plugin to the game?

I try to copy example plugins dll files to %temp% but in game is nothing changed.

sorry for late answer..
what is your plugin name ? is that plugin contains DLSP_ prefix ?

and did you run the core app ? for plugin load ??

Core app? I copied example plugins from your repository to %temp%. There is no core app. Plugin name contains LSP_ prefix.

https://memoryhackers.org/konular/pubg-mobile-dword-vip-r3-beta-v2-4-march-2019.70425/
core app is here.
and for Developer Plugins Plugin prefix Should be DLSP_

plugins are loading but not working


User Account Demo

New user created cannot login. It seems like it sticks to the user account demo. Do we have workaround on this in code. Thanks.

Can't re-create this. If you go to sample site (blogifier.azurewebsites.net) and create new user there - are you able to log in?

Downloading latest copy from GitHub resolves the issue. I'm still testing it on localhost but I plan to deploy it on my existing ASP.NET hosting provider not on Azure. Thanks for the update.

Just an observation in the database, user account credentials are stored in AspNetUsers and Authors, I can't see foreign key relationships between the two tables.
Thanks.

This more like two databases running side-by-side, foreign key can be added but not necessary for our usage.


Is there a way to know which process triggered an action?

Let's say the main process triggers an action. Once the action is complete and the store is updated, I need to handle that change in the renderer process. But if an action is triggered in the renderer process, I need to handle it a different way. Is there a way to know which process triggered the action?
Here's an example.
When a menu item is clicked, I need to trigger an action that updates my store from the main process and once the store is updated I need to react to that change on the renderer side. I could use a lifecycle method to react to the props changing but I only need to do so if the change was triggered by the main process. Does that make sense?

Nevermind. Turns out I don't actually need to know which process triggered the action. It's how I handle the change which, while being a little hacky, is pretty simple. Closing the issue.


Stop working after a few seconds

Hi,,
I can see the data points but after a few seconds the frames received do not contain TLVs.
Do you know why this happen?
Thanks
Regards

Hi,
I think that I solved it.
I have to add the following:
if idX > 0 and byteBufferLength>idX: (in line 259) and if startIdx and startIdx[0] < byteBufferLength: (in line 139).
Moreover, I also added:
byteBuffer[byteBufferLength-startIdx[0]:] = np.zeros(len(byteBuffer[byteBufferLength-startIdx[0]:]),dtype = 'uint8')  (in line 144)
byteBuffer[byteBufferLength - shiftSize:] = np.zeros(len(byteBuffer[byteBufferLength - shiftSize:]),dtype = 'uint8') (in line 264)
Since when we are moving the data, there are some data that is replicated if it is not removed.
Finally, I added the other cases (just in case they are considered):
elif tlv_type == MMWDEMO_UART_MSG_NOISE_PROFILE:
idX += tlv_length
        elif tlv_type == MMWDEMO_UART_MSG_AZIMUT_STATIC_HEAD_MAP:
            idX += tlv_length

        elif tlv_type == MMWDEMO_UART_MSG_DOPPLER_HEAD_MAP:
            idX += tlv_length

        elif tlv_type == MMWDEMO_UART_MSG_STATS:
            idX += tlv_length

        elif tlv_type == MMWDEMO_UART_MSG_DETECTED_POINTS_SIDE_INFO:
            idX += tlv_length

        elif tlv_type == MMWDEMO_UART_MSG_MAX:
            idX += tlv_length

Now, it works.
Many thanks

I am glad it works. I will check your code and add update the program with your code.
Thank you for your feedback.
Best regards,
Ibai


Slim 669 improve cucumber scenarios and coverage

Changes for flexible testing on different testsystems by flexible choosing the URL. Updated test scenario description and implementation.

This pull request makes the build fail.

This pull request makes the build fail.

This pull request makes the build fail.

This pull request makes the build fail.


Administrator account issue

Why doesn't the initialization script have an administrator account password？What should I do？

Hi~ The system administrator account needs to be initialized on the login page.


List on hex.pm

This is cool stuff, once we get things cleaned up, we should publish it.

https://github.com/helium/erlang-tc/pull/18


Bootstrap 4.1 Support

Will it ever support Bootstrap 4?

1.13.1 as the author claims already supports bootstrap 4

Hi everyone,
I'm also looking forward to using "bootstrap-select" in the lastest bootstrap version (4.1.3).
The last version of bootstrap-select I found in the repository in 1.12.4... @slidenerd , where did you get this 1.13.1 version ?
Thx

Hi, i try also to make it woks with Bootstrap 4.1.3 is there a workaround or an update ?
thanks

It works fine with 1.13, except I can't get liveSearch to work.
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/bootstrap-select/1.13.1/css/bootstrap-select.min.css">

thanks a lot, liveSearch works fine !

Le 3 août 2018 à 20:42, Jesse Knowles<EMAIL_ADDRESS>a écrit :
It works fine with 1.13, except I can't get liveSearch to work.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub , or mute the thread .

I have it working on 4.1.2 or bootstrap but the dropdowns don't open first time.  I click once nothing then click it again and it works fine.  Does anyone else have this issue?

You can now find v1.13.0+ in this repository. See https://github.com/snapappointments/bootstrap-select/releases.


Root dependencies cleaned up

Closes #194

Coverage decreased (-0.003%) to 97.361% when pulling 621f3566415ed497b2d6b01d4f8be5b637654b91 on deps-cleanup into 032a5575935c7f782dda4f19908891038de997b8 on master.

Changes Unknown when pulling 3db983ee5044438dada211a16e6e17e6791aab52 on deps-cleanup into ** on master**.


libhostfxr.dylib could not be found

libhostfxr.dylib could not be found
$ hello> dotnet new
A fatal error occurred, the required library libhostfxr.dylib could not be found at /usr/local/bin/
Info
So happy to heart that .netcore support macOS sierra, but while I have tried that's still a big disappointment!

Can you describe how you installed .NET Core?

Issue moved to dotnet/core-setup #3400 via ZenHub

Final solution:
Go to default user directory cd ~, and add below export PATH=$PATH:/usr/local/share/dotnet line to the end of ~/.zshrc file
cd ~
vim ~/.zshrc

export PATH="$PATH:/usr/local/share/dotnet" # Append this line to end.


-Zbuild-std gives error: duplicate lang item in crate core:

Problem
Multiple variations of:
error: duplicate lang item in crate `core`: `bool`.
  |
  = note: the lang item is first defined in crate `core` (which `std` depends on)
  = note: first definition in `core` loaded from /media/data/openwrt/build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/angelcam-connector-0.10.1/target/armv7-openwrt-linux-muslgnueabihf/release/deps/libcore-8bec01d570fa74ab.rlib, /media/data/openwrt/build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/angelcam-connector-0.10.1/target/armv7-openwrt-linux-muslgnueabihf/release/deps/libcore-8bec01d570fa74ab.rmeta
  = note: second definition in `core` loaded from /media/data/openwrt/staging_dir/host/lib/rustlib/armv7-openwrt-linux-muslgnueabihf/lib/libcore-533ac1cfe6c35904.rlib

as adding -Zbuild-std adds --extern flags to the invoke when they already exist in the sysroot.
Steps
No response
Possible Solution(s)
No response
Notes
Reference thread https://rust-lang.zulipchat.com/#narrow/stream/122651-general/topic/The.20basics.20of.20debugging.20duplicate.20definitions/near/285316237
In which @jyn514 figured it out.
Version
cargo 1.60.0-nightly

Can you provide some more information on how to reproduce the problem?
It is normal behavior for cargo to use --extern flags with -Zbuild-std, even if there is a sysroot. The --extern ones take precedence.

This is rust 1.60.0 under OpenWrt (x86_64-unknown-linux-gcc Build/mips64-openwrt-linux-musl Host/Target).  My standardized cargo call included a -Zbuild-std, which was causing issues
See the attached build log, and the thread at https://rust-lang.zulipchat.com/#narrow/stream/122651-general/topic/.E2.9C.94.20The.20basics.20of.20debugging.20duplicate.20definitions
Removing the -Zbuild-std flag solved the duplicate definition issue, and it was suggestion (at the above Zulip link) that it probably was a cargo issue, and to report it.
I'm not a rust-dev, but I can get any information you might want me to collect
connector.log

I'm going to close as this seems to be an issue with some integration with the OpenWrt build system.
I understand the duplicate lang item error can be annoying and difficult to diagnose. However, without a specific minimal reproduction, it is unlikely we'll be able to help here. If you can boil it down to a minimal set of steps using a stock Rust installation without using another build system, that might be helpful. However, I understand that might be difficult if you are not a Rust developer.


Readd libpython

Checklist

[X] Used a personal fork of the feedstock to propose changes
[X] Bumped the build number (if the version is unchanged)
[X] Reset the build number to 0 (if the version changed)
[X] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[X] Ensured the license file is being packaged.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

@conda-forge-admin, please rerender

@brandonwillard where does the version number 2.0 come from? And how do we know when it needs to be updated?

@brandonwillard where does the version number 2.0 come from? And how do we know when it needs to be updated?

I don't recall any exact details at the moment, but it doesn't seem like a particularly restrictive bound.

I see now that libpython is not actually a library, but merely sets up some configuration for using m2w64 with Python. Thus I believe we want to always pin the latest version of the package on Conda-Forge (2.2).
I have added both libpython and m2w64-toolchain to the aesara-main package.
@twiecki, if you approve you can add the automerge label from the right column on this page in order to merge once the tests pass.

@twiecki could you please trigger the merge? I don't have permissions.


Vulnerability fix - crypto to 41.0.0

bump cryptography to 41.0.0 for vulnerability fix,
bump pyopenssl to 23.2.0 for compatibility with cryptography 41.0.0

I can't merge this without write access to the repo.


test_shaders.py: unconditionally updates reference files when new results differ

The test reference files are just updated whenever the new results differ.  I'd suggest guarding that behaviour behind an --update flag.  By default the script should fail if results differ from expectations.

Implemented on master, as well as --malisc and --keep flags.


brew list --cask outputting uninstalled casks

brew config output
HOMEBREW_VERSION: 3.4.6
ORIGIN: https://github.com/Homebrew/brew
HEAD: 56415326bb8c9eccc6e53f3997b4e062b36402af
Last commit: 6 days ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: afa4538954ae879c2071d3a16c67600b174d5f32
Core tap last commit: 45 minutes ago
Core tap branch: master
HOMEBREW_PREFIX: /usr/local
HOMEBREW_CASK_OPTS: []
HOMEBREW_CORE_GIT_REMOTE: https://github.com/Homebrew/homebrew-core
HOMEBREW_MAKE_JOBS: 16
Homebrew Ruby: 2.6.8 => /usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.8/bin/ruby
CPU: 16-core 64-bit ivybridge
Clang: 12.0.0 build 1200
Git: 2.35.1 => /usr/local/bin/git
Curl: 7.64.1 => /usr/bin/curl
macOS: 10.15.7-x86_64
CLT: <IP_ADDRESS>.1.1610135815
Xcode: N/A

brew doctor output
Warning: The following directories are not writable by your user:
/usr/local/share/man/man8

You should change the ownership of these directories to your user.
  sudo chown -R $(whoami) /usr/local/share/man/man8

And make sure that your user has write permission.
  chmod u+w /usr/local/share/man/man8

Warning: Broken symlinks were found. Remove them with `brew cleanup`:
  /usr/local/share/man/man8/prl_convert.8
  /usr/local/share/man/man8/prl_disk_tool.8
  /usr/local/share/man/man8/prlctl.8
  /usr/local/share/man/man8/prlexec.8
  /usr/local/share/man/man8/prlsrvctl.8

Verification

[X] I ran brew update and am still able to reproduce my issue.
[X] I have resolved all warnings from brew doctor and that did not fix my problem.

What were you trying to do (and why)?
I am trying to get a list of installed casks to determine if I need to install the casks or not. Chef.io is also using brew list --cask for this reason, as you can see here.
What happened (include all command output)?
As shown by the output below, brew list --cask outputs parallels; however, when brew info parallels is run it shows that parallels is not actually installed:
brew list --cask
parallels parallels-virtualization-sdk vagrant
brew info parallels
parallels: 17.1.2-51548 (auto_updates)
https://www.parallels.com/products/desktop/
Not installed
From: https://github.com/Homebrew/homebrew-cask/blob/HEAD/Casks/parallels.rb
==> Name
Parallels Desktop
==> Description
Desktop virtualization software
==> Artifacts
Parallels Desktop.app (App)
==> Analytics
install: 950 (30 days), 2,572 (90 days), 9,011 (365 days)
What did you expect to happen?
the 'parallels' cask should not be included in the output of brew list --cask as it is not installed.
Step-by-step reproduction instructions (by running brew commands)
`brew list --cask`
`brew info <cask>`

Looks like a duplicate of https://github.com/Homebrew/brew/issues/13139. The fix for it was merged 2 days ago https://github.com/Homebrew/brew/pull/13141.

@amadejpapez I am still hitting the same issue after updating to 3.4.7 which contains #13141.

What's the result of running ls $(brew --caskroom)/parallels? And what happens when running brew uninstall --cask parallels?

@EricFromCanada
ls -l $(brew --caskroom)/parallels # => total 0
brew uninstall --cask parallels # => Error: Cask 'parallels' is not installed.

Running brew list or brew list --cask simply passes the Cellar or Caskroom path to ls. It appears the "parallels" folder wasn't removed from your Caskroom path when the cask was uninstalled, which is why it was listed. In my testing, though, running brew uninstall --cask parallels after installing removed the folder properly, so some other issue must be involved.

@EricFromCanada
Yeah, it looks like it wasn't deleted from the Caskroom path. I'm thinking that the issue might be related to the fact that the 'parallels' directory only contains a '.metadata' subdirectory:
tree -a parallels

parallels
└── .metadata
    └── config.json


Fix crd upgrade step

Signed-off-by: nshankar13<EMAIL_ADDRESS>
Description: The CRD upgrade step for the upgrade-e2e-job was running the CRD cleanup script from the current upstream branch, instead of the upstream branch corresponding to the previous release. This fix ensures that the CRDs are deleted from the "previous" branch, and the installed from the "current" branch.

Affected area:

New Functionality      [ ]
Documentation          [ ]
Install                [ ]
Networking             [ ]
Metrics                [ ]
Security               [ ]
Tests                  [ ]
CI System              [X]
Performance            [ ]
Other                  [ ]

Please answer the following questions with yes/no.

Does this change contain code from or inspired by another project? If so, did you notify the maintainers and provide attribution?

/azp run upgrad-e2e-arc


Fix ldap

Problem

#81

Solution

Use YunoHost permissions

Yolocommit, not well tested lul
PR Status

[ ] Code finished.
[ ] Tested with Package_check.
[ ] Fix or enhancement tested.
[ ] Upgrade from last version tested.
[ ] Can be reviewed and tested.

Package_check results

If you have access to App Continuous Integration for packagers you can provide a link to the package_check results like below, replacing '-NUM-' in this link by the PR number and USERNAME by your username on the ci-apps-dev. Or you provide a screenshot or a pastebin of the results

We should find a way to sync wekan mail with ldap mail

n mail with ldap mail

Mails should be synchronized between ldap and wekan. The only issue is when a user 'like the first one created on YunoHost has several email addresses.
https://github.com/YunoHost-Apps/wekan_ynh/blob/a196efb0622b0118803f1064f3324bbbd0303002/conf/.env#L358-L365


Markdown highlighting not working directly after an HTML tag

VSCode Version: 1.52.0
OS Version: Windows 10

Steps to Reproduce:

In a markdown file write:

# Title

**test**

See that the title and the bold text are not highlighted.

To fix this problem, the user can add an empty line:

But when using an extension which adds an anchor to titles for generating Table of Content, for example, we don't always have the control of where the HTML elements are being added.

Does this issue occur when all extensions are disabled?: Yes

This is still a bug today.  Note that the last line beginning with *Pushed... does not have Markdown highlighting applied:
GHE Dev Terraform Plan 📖success
Show Plan
Some content

*Pushed by: @jonathan-regeimbal, Commit: `blank`, Action: `pull_request`*


⚠ Account has degraded performance

In 330c08a, Account (https://account.wrk.com/health) experienced degraded performance:

HTTP code: 200
Response time: 183 ms

Resolved: Account performance has improved in 10bcd27.


Allow multi-line inputs for form values

Originally reported on Google Code with ID 191
What steps will reproduce the problem?
1. create a post request
2. under form payload add a new entry
3. paste a multiline text for example a small CSV output

What is the expected output? What do you see instead?
the line return should be urlencoded, but because an input is used, they are stripped
from the, using a textarea instead of an input would allow for multiple line form values.

On what operating system, browser and browser version?
Chrome 29.0.1547.22 / OSX 10.8

Please provide any additional information below.

Reported by dcolens on 2013-07-16 10:33:14

This won't be usable. The same effect you'll reach if you edit the values in the RAW tab.


getBoundingClientRect fails when element not in DOM

Repro in IE.

What is the impact of this bug? These issues are hard to triage.

This is IE so P4.

IE11 support removed


Design Meeting Notes, 10/1/2024

The internal Modifier
https://github.com/microsoft/TypeScript/issues/5228

We've discussed an internal modifier on-and-off since 2015.
High 👍 count, but haven't pursued it.
Today you can use /** @internal */ JSDoc comments to mark things as internal, and --stripInternal.
Previously, --stripInternal was not even publicly documented.
Also, have to do hacks to make .d.ts bundling work with this.
Would be nice to have something more formal.

Also, internal allows you to know whether something is getting overridden in a subtype.

Idea:

internal stays in .d.ts output.
Project package or project boundary.

Only operates in the same compilation.

Not allowed to import an export internal.
At the tsconfig.json level, internal access might be granted and asserted in a few ways:

Maybe an option for specific packages
{
    "compilerOptions": {
        // Grant access to 'consuming-package':
        "internalsVisibleTo": ["consuming-package"],

        // Assert access to 'dependency-package':
        "internalsVisibleFrom": ["dependency-package"],
    }
}

Possibly just something on each reference
{
    "references": [
        { "path": "../package", "allowInternals": true }
    ]
}

What are the rules?

Can't mix (e.g. no public and internal in the same declaration).

Why not JSDoc?

Relatively expensive for this.

What should tools do with internal?

Should bundlers limit access to an internal member that's export *'d?

We'd sort of hope "no".

Does the declaration emitter have to worry about this now?
What happens when you run keyof on a type with internal?

Probably should do the same thing that private does.
But does that mean that keyof means something different depending on where it's written.

How does this work for overload resolution?

Do overloads get skipped?
Do they get related appropriately?

The "someone subclassed this" issue is not something we've heard a ton of.
Do most people use .d.ts bundlers that can do this post-processing?

No, mostly not.

Some of our build tooling would be simpler if we had this.
But doesn't it mean that we're just making other stuff more complex?

For example, public and private overloads can't be mixed today.
Well maybe we do need to explore that restriction.

Also, some of the point of --stripInternal is to not make things publicly known and avoid documenting functions to prevent usage.
Back to this question again - what do you do when you have a mapped type?

internal is inaccessible depending on where it's used.
private and protected has this, but it's not witnessable in the majority of code (which is outside of the class).

Coming back to keyof - is this now a location-sensitive operation?

The idea that you will get different types based on instantiation location is not something we've done before.

Really don't like that.

What do you want?

Scope visibility to peers.
Keep existence undocumented.

Do we really want an ergonomic way for people to import something marked internal?

We don't have to do everything listed above.

Can do this just on the module level.

Maybe even just that and the property level.
"Good enough" as readonly.

Cautious but curious around a solution.

Not convinced that we would have a solution that we'd entirely be happy with.
Must preserve existing invariants around how type information appears.

One opinion: sympathetic to a package-like modifier, but doesn't entirely solve the problem for our own usage for hiding entities in the resulting declaration file.

Not allowed to import an export internal.

We are talking here about whole codebase or just entry file?
I think good idea would be have option or just apply this to entry file.
Or is it mentioned at

At the tsconfig.json level, internal access might be granted and asserted in a few ways:

I am not sure i full understand this.


Silence clang warning

Changed implicit char pointer arithmetic to array index notation to stop clang from complaining.

I haven't touched this project in years, but thanks.


Sol2: Add includes_lua option to be able to ship a custom Lua library

In the case of Lua, it's not uncommon to ship a custom Lua library (to update the langage a bit for example), but sol2 forces the lua dependency, this adds an option to disable the default lua shipping.

Thanks!


Colab: Notebook can not run all cells

After Colin completed his session on 2023-06-30, the Colab notebook was unable to run all cells.
Debug.

The issue was that in the version revised by Colin, the name of the file bfactors.npz was changed to b_factors.npz, but this change was not reflected in the dependency in the Download results section. This has been ammended.


Feature request: metadata passthrough / more type support

As my nixpkgs grows bigger, I plan to also generate metadata for each of the packages I defined. Using passthru in sources.toml isn't very helpful because it assume whatever value that it takes is a string, thus preventing greater control of the generated nix expr. I suppose a separate meta attribute would suffice (including more type support).
Maybe Nix could also be used interchangeably with toml? Though I would be fine if I can just evaluate Nix into toml and use that instead.

I also considered about this question. Currently we use string interpolation to generate nix expr, which is not neat and error-prone. If we could create the expression programmatically, i.e. constructing and pretty printing the AST, it won't be hard to transpile toml to according nix expression.


Update the CLI message of the ballerina command

Purpose

Describe the problems, issues, or needs driving this feature/fix and include links to related issues.

Fixes #
Approach

Describe how you are implementing the solutions along with the design details.

Samples

Provide high-level details about the samples related to this feature.

Remarks

List any other known issues, related PRs, TODO items, or any other notes related to the PR.

Check List

[ ] Read the Contributing Guide
[ ] Required Balo version update
[ ] Updated Change Log
[ ] Checked Tooling Support (#)
[ ] Added necessary tests

[ ] Unit Tests
[ ] Spec Conformance Tests
[ ] Integration Tests
[ ] Ballerina By Example Tests

[ ] Increased Test Coverage
[ ] Added necessary documentation

[ ] API documentation
[ ] Module documentation in Module.md files
[ ] Ballerina By Examples

Codecov Report

Merging #19273 into ballerina-1.0.x will not change coverage.
The diff coverage is n/a.

@@               Coverage Diff                @@
##           ballerina-1.0.x   #19273   +/-   ##
================================================
  Coverage            15.73%   15.73%
================================================
  Files                   47       47
  Lines                 1265     1265
  Branches               197      197
================================================
  Hits                   199      199
  Misses                1053     1053
  Partials                13       13

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 95f6d57...e60f614. Read the comment docs.


Request Body Parameters unclear for setting up AWS S3 via API

I'm trying to write an API POST request to connect/save my AWS S3 bucket to my self-hosted Only Office server. This is easy enough to do manually so I presume it can be done via an API call but I'm unclear how to submit my AWS IAM access key / access secret and indicate AWS S3. Using requests my python function will look something like this albeit with a smaller json object, hopefully only 3 items...
def connect_s3(auth_token):
    data = {
        "url": "some text",
        "login": "some text",
        "password": "some text",
        "token": "some text",
        "isCorporate": True,
        "customerTitle": "some text",
        "providerKey": "some text",
        "accessKey": "some text",
        "secretAccessKey": "some text"
    }
    r = requests.post(f'{URL_ROOT}/files/thirdparty', json=data, headers={"Content-Type": "application/json", "Authorization": f"Bearer {auth_token}"})  # noqa: E501
    return json.loads(r.content)

What options do I need to pass in that json object to indicate my access key / access secret and AWS S3 provider to the endpoint POST api/2.0/files/thirdparty? The API docs don't specify a providerKey for S3 despite it being available in the GUI.

Oh thats actually exactly what I'm trying to do. I want to use S3 for backup file storage, not for normal documents... Sounds like I'm targeting the wrong endpoint then. Is there an API endpoint for configuring S3 for backup storage? I see how to do it manually but I need to do it programmatically with the API. I was able to save a backup file on the local filesystem like so...
def start_backup(auth_token):
    data = {
        "storageType": "Local",
        "storageParams": [
            {
                "Key": 'filePath',
                "Value": '/var/www/onlyoffice/Data/Studio/00/00/01/backup'
            }
        ],
        "backupMail": 'true'
    }
    r = requests.post(f'{URL_ROOT}/portal/startbackup', json=data, headers={"Content-Type": "application/json", "Authorization": f"Bearer {auth_token}"})  # noqa: E501
    return json.loads(r.content)

But I'd like to be able to pass my aws credentials, specify S3, and have the backup file go directly to S3 (then restore directly from S3). Is there a separate endpoint to configure S3 or maybe special keys for the json object sent to the POST api/2.0/portal/startbackup endpoint?

You can send it in request, for example
api/2.0/portal/startbackup.json
    "StorageType": "5",
    "storageParams":  [{"key":"module", "value":"S3"},
     {"key":"bucket", "value":"qwe"},
     {"key":"region", "value":"qwe"},],
    "backupMail" : false

But first you need configure keys for S3 in portal settings /Management.aspx?type=9

Thank you! Do you have a link to those Key: Value labels? I couldn't find them anywhere in the API docs...

Yes, the documentation of this method is far from ideal. You can see the parameter set here https://github.com/ONLYOFFICE/CommunityServer/blob/master/web/studio/ASC.Web.Studio/web.consumers.config#L176-L185

Sorry, but this is not currently implemented. We will try to add this in one of the future versions.


revoke permissions for builder

I think that this solves https://github.com/alsuren/cargo-quickinstall/issues/49 without any added complexity.
I stumbled across this setting when trying to set up the webhook based scheme described in #49.
The paper trail went something like: https://github.com/organizations/cargo-quick/settings/actions
https://docs.github.com/en/actions/security-guides/automatic-token-authentication#modifying-the-permissions-for-the-github_token
This allowed me to know that it was possible at the top level, and then I jumped into vscode to see whether it was possible to use it at the job level (using the red hat yaml plugin, which pointed me at https://json.schemastore.org/github-workflow.json)
Turns out it is: https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idpermissions

Don't worry about the failing test. This only exists because I have a special case for the actions branch, which happens to have a race in it.


Tab focusing between widgets doesn't focus text (2017.2)

I expect to tab between widgets and be able to enter text. Same with when the thing opens up.

Looks like a Unity bug: https://www.reddit.com/r/Unity3D/comments/75y2gj/201720f3_pressing_tab_doesnt_select_all_text_in/


Q: Combine with official PoE HAT

Hello,
Can the Witty Pi 4 Mini be used with the official PoE HAT on a Raspberry Pi 4+? I am looking to manage power on a set schedule and would love to explore other functionalities the witty board offers.

Witty Pi 4 Mini doesn't support the official PoE HAT.
Witty Pi 4 Mini doesn't even have a DC/DC converter on board, and it can only accept 5V DC.
The PoE HAT will power the Pi directly and that will bypass Witty Pi 4 Mini.


Capability to publish resource with assigned DOIs

Implement the capability to have HydroShare resources permanently published with DOIs once their metadata meets the requirements in the Formal Publication Specification Document https://my.alfresco.com/share/s/wqJISdf_TQS5lJs32OoYZw.  This should be coordinated with CUAHSI working with Rick Hooper

Details of the process and workflow for formally publishing a resource are available in the Google Doc at:
https://docs.google.com/document/d/1tNFStpEmbB_AC2ICppVSw-024-iTrnqGK9Np7Elw2OM/edit
The following are needed to complete formal publication of a resource:

[ ] Add a "publish" button to the resource landing page
[ ] Add code to validate that the required metadata elements are present and that the resource has content files for resource types that require them. If they are incomplete, notify the user and boot them out of the process.
[ ] Display a message notifying the user that the resource will be permanently published and that they will no longer be able to edit the resources. The message will also state that the user will be able to create new versions of the resource after it has been published. The user must agree to continue.
[ ] Display the text of a data publication agreement that the user must agree to before proceeding.
[ ] Automatically assign a DOI to the resource

[ ] Construct the DOI for the resource using the CUAHSI prefix (see Google Doc)
[ ] Construct the deposit XML for Crossref from the resource's science metadata
[ ] Upload the deposit XML into Crossref for registration
[ ] Display a message that alerts the user that the DOI has been requested and that the resource will be updated when the DOI has been registered.
[ ] Monitor the status of the deposited DOI until it has been registered. This may be instantaneous and contained in the response from the Crossref web services. However, need to make sure that the DOI actually gets registered.

[ ] Set the properties of the resource to make it immutable (disable the edit button on the resource landing page) and set the status of the resource to "published".  Display a status of "Published" on the landing page.
[ ] When the DOI is available, update the resource's science metadata to include the DOI as an identifier element. Update the display of the landing page to use the URL-encoded DOI in the citation instead of the HydroShare URL.
[ ] Send a notification to the user that the resource has been published and that the DOI is available

CUAHSI's prefix for DOI's is set:  10.42111/
Sample of an existing CUAHSI DOI:  http://dx.doi.org/10.4211/his-data-shalenetwork
It is up to HydroShare to generate a unique suffix for each DOI. One simple option for the suffix will be to simply use the HydroShare resource identifier, but I think we want a node in the suffix to uniquely identify HydroShare resources given that the prefix is for anything CUAHSI creates.  This would effectively place HydroShare resources within a DOI namespace of sorts within CUAHSI's unique prefix:
Example:
http://dx.doi.org/10.4211/hs.<resource identifier>

Where:
"hs" is a unique DOI node for HydroShare
<resource identifier> is the HydroShare resource identifier

@dtarb @horsburgh Just to confirm: we don't require resource to be public or discoverable before the resource can be published, right? In other words, as long as the same requirements are met for making a resource public, the owner of the resource can publish a private resource, right?

@hyi - Correct.

Yes, but the act of publishing should result in the resource ending up discoverable or public (with public the default).  Also in some discussions with Rick Hooper there has been suggestion of additional resource metadata completeness rules that may need to be met for publish over and above public and discoverable, so while we do not have such rules now, please recognize that we may need this in the future.

Public and Published are effectively 2 different states.  A resource can be Public without being Published.
Both Public and Published are discoverable.

@dtarb @dtarb @alvacouch So once a resource is published, the resource should be set as Public by default as its sharing status? If so, I would assume it means the owner of the resource can modify the sharing status even after a resource is published. Then I guess the published resource is immutable only in terms of its content files and metadata since we still allow users to change its sharing status after the resource is published. Is this intended? It makes more sense to me NOT to allow users to change sharing status after a resource is published and made immutable, but there might be use cases I am not aware of that requires changing sharing status even after the resource is published. Need explanation or confirmation to make sure I implement it correctly.

I think "Published" should be a sharing status. "Public" and "Published" are not exactly the same.
And yes, permissions for "Published" would be the same as "Public" but should not be revocable (or really editable except that ownership may change?).

@horsburgh @dtarb @alvacouch Making "Published" a sharing status is more complicated from implementation standpoint since access control code has to be modified to consider this additional "Published" sharing status. We can enforce more requirements for publishing a resource on top of requirements for making a resource public, and it seems simpler to make "Published" as a flag on the resource object without making it a separate sharing status. Since we don't allow a resource to be editable after the resource is published (i.e., disabling resource edit button), it makes more sense to me not to allow users to make any edit on a published resource including the resource's ownership and sharing status. To make published resource discoverable, we can add a requirement to only allow a resource to be published after it is made public or discoverable. This means we don't allow users to publish a private resource. This makes sense to me since we need to make published resource discoverable, so why do we allow users to publish a private resource. We can just require owners to make resource public or discoverable before they can see the publish button. What do you all think?

Published resources need to show up in the "Sharing Status" column in the My Resources page with a sharing status of "Published". Also, on the call this week we discussed being able to filter by SharingStatus = "Published".  Also - it would be better if it weren't a two step process to make something public and then have to take a separate step to publish.  Seems like private, public, and published are mutually exclusive states for a resource.
Publishing a private resource would effectively make it publicly viewable and discoverable (there is no such thing as a "private" Published resource).

@horsburgh @dtarb OK. I think I've got the point. I was confused by the comment that publishing a resource makes it public or discoverable (public by default) which made me think public or discoverable sharing status in HydroShare. Thinking "Published" as a mutually exclusive sharing status with Public, Discoverable, and Private makes sense to me now. Basically, we allow to publish a resource with any sharing status (private, discoverable, or public) as long as the resource meets the publishing requirements, and after a resource is published, the resource becomes viewable/discoverable/searchable to anybody, and of course is immutable and not-editable. Since a published resource is not-editable, should we remove uses with editing privileges from access control point of view? Should we allow owners of a published resource change ownership? Adding @alvacouch in the loop for access control related questions for a published resource.

See my response in #889.
While formally publishing a resource, which includes setting the immutable flag, makes having change permissions moot, I do not think that the act of publishing should remove all change and view permissions. If we did this it would then result in removing a resource from a MyResources page for a user that had change permissions (i.e. it disappears from their page).  I prefer to leave the change and view permissions as declared privileges and have these declared privileges serve to support the filtering functionality on the MyResources page.
As to the question about changing declared privileges for published and immutable resources, I think it should be allowed.  This means that the setting and revoking of declared privilege parts of access control for a resource are not frozen, by a resource becoming immutable. So I suggest interpreting immutable as applying to the content, not the ownership and permissions.  I think that all access control changing functionality as currently stated in policy (and implemented notwithstanding bugs still to be resolved) should still apply for published and immutable resources.  This includes

an owner can add or delete owners where there are multiple (not removing the last one).  This enables ownership transfer for these resources should the responsibility for "keeping" change.
a user with change permissions can add or remove change and view permissions.  The effect of this will be leaving control over what appears on MyResources pages with the users, in a similar manner to unpublished resources.
A user with view permissions could remove their view permission.  This would effectively drop the resource from their MyResources page.
We could extend the model a bit to allow any user to claim view permission on a published resource.  This could be the device whereby they add it to their MyResources page rather than tracking separately whether they have added to MyResources.

The rationale behind this is consistency of application, behavior and interface.  A resource becoming published and immutable should not change the behavior users will get accustomed to in sharing of resources (and we simplify the code by reusing what exists).

@horsburgh Question related the 'Publisher' metadata element. As per the current code base the Publisher name sub-element is set to "HydroShare" and the the url sub-element is set to 'https://hydroshare.org". It seems this needs to change. Now it seems the publisher name attribute needs to be set to "Consortium of Universities for the Advancement of Hydrologic Science, Inc. (CUAHSI)" and the url needs to be set to "https://www.cuahsi.org". Is this correct?
Also, CUAHSI becomes the publisher only when the resource has content files. For resources, that does not require content files, the user should be able to set the publisher to something other than "CUAHSI". Is this assumption correct?

@dtarb - Leaving all of the access control open for editing on Published resources seems unnecessary and potentially confusing. It doesn't make sense to me that we would allow users to keep granting "edit" access to something that can't be edited. Also, why do we need to grant "view" access?  If it's published, anyone can view it and add it to their list of resources.  Also - if we disable the edit button for Published resources (which is part of the workflow we laid out), users wouldn't be able to get to the access control dialog anyway to change any of those rules because it's on the edit page.
With respect to owners - the responsibility for "Keeping" published resources resorts to the publication system once something is published.  When I publish a research paper, I don't have any responsibility to to do any "Keeping". So, I'm not sure what "keeping" means in this context.
Published resources should not behave the same as other resources. Buy formally Publishing something, the user has made a final sharing choice.
I think we need some other way to keep published resources in people's lists without allowing modification of permissions after publication.

@pkdash - I made the suggestion that only resources with content files stored in HydroShare can effectively be "published" by HydroShare. The only way HydroShare can guarantee that content to which a registered DOI points does not go away is for that content to be stored in HydroShare and not elsewhere. Plus, for some types of referenced resources, the content to which they point may already be published elsewhere.
There was some discussion on the phone this week about whether referenced resources should be formally publishable in HydroShare.  They would effectively be metadata-only records (pointing at data stored in another system) to which DOIs would be assigned.  I think the BYU team was in favor of this.  I think we need a pow-wow to make some final decisions about this.

@dtarb - @alvacouch 's suggestion in #889 of what to do with permissions for immutable resources seems reasonable. Since the edit page will be disabled for immutable resources, the following changes would be made to access control when resources become immutable:

real privilege
not immutable
immutable

OWNER
OWNER
OWNER

CHANGE
CHANGE
VIEW

VIEW
VIEW
VIEW

@dtarb @horsburgh @alvacouch @mauriel133 @pkdash I have implemented resource publication in the way that after a resource is published, it is made immutable with ownership retained and edit permissions downgraded to view permissions for all edit users. The published resources appear on my resources page since it is public and discoverable with sharing status as "Published". Filters on my resources page still apply but published resources will not show up on "editable by me." After clicking on a published resource from my resources page to get to published resource landing page, it shows its sharing status as "Published and Immutable" with no edit and delete buttons. For a discovered published resource, "Add to my resources" button will still show up to allow users to add a published resource to my resources list, but no edit and delete button will show up preventing users from doing any edit to published resources. This means ownership and permissions are frozen for published resources retaining values at the time when the resource was published although edit permission is downgraded to view permissions automatically when a resource is published. Will deploy the resource publication work to a VM after the DOI assignment work with crossref is done for testing and comments.

@hyi - Awesome!  I'll look forward to testing!

See issue #893 for changes in access control that will benefit this -- forthcoming.

@alvacouch If you can get this done by end of next week, that would be great. I can then integrate my resource publication work on top of your work for issue #893 without doing duplicate work to get user interface working as expected. Thanks.

@alvacouch Based on my testing, what I have done already regarding permissions and access control for published and immutable resources worked well for resource publication purposes, so there is no rush for you to get issue #893 done for resource publication work. You can take your time to get system-wide change worked out as documented in this issue without rushing. We can always integrate our respective overlapping work on top of each other as needed. @dtarb @horsburgh @pkdash @mauriel133

@alvacouch @dtarb @horsburgh @pkdash @mauriel133 I noticed the following potential "issue" based on testing on my resource publication branch:

Although non-owner users with edit permissions can be safely downgraded to VIEW permissions for published immutable resources, retaining ownership of a published resource means all owners have edit permissions in our current access control implementation, so if we retain the owners, we cannot remove owners' edit permissions. This results in published resources ending up in "editable by me" filter in my resources page for owners of published resources.
@alvacouch Do you intend to fix this in access control system as part of work to address issue #893 or we retain the current behavior with assumption that owners have edit permissions but for published resources, the "editable by me" is just a declared privilege, not an effective privilege since they will not be able to edit or delete a published resource from the resource landing page anyway. What do you all think? In any case, this needs to be handled system-wide in access control level, so I will not do any further work for resource publication regarding this specific issue.

@horsburgh Looks like http://test.crossref.org requires login to be able to do anything. Do I really need a login credential on test.crossref.org for testing purpose? I cannot find any user registration info on the site, so not sure how to obtain login account. Need to dig deeper into crossref help system to understand the process, in the meantime, asking you in case you have information handy regarding this. I need to work on creating deposit xml file first, so don't need to use crossref test system now. Just happen to come across this which prompted the question.

@hyi - I don't know if the test site requires it, but we will eventually need CUAHSI's login information.  @dtarb - can you help get the necessary details from Rick Hooper?

@dtarb @horsburgh I have successfully uploaded deposit xml metadata to crossref test system programmatically using crossref https request POST API. After about 10 minutes, I received email notification from crossref indicating success. According to crossref documentation, how long requested DOI can be processed and activated depends on how many requests are in the queue at the time. Although the basic resource publication workflow has been implemented as outlined in the design spec, we need to decide what to do with the resource during the time after the DOI request has been successfully deposited in crossref system and before the DOI has been activated by crossref to take effect (i.e., after we receive email notification from crossref indicating success). In the worst scenario, human intervention may be needed, to contact crossref support if we don't receive email notification for successful DOI activation in a long time or resolve some unforeseen issues. I am currently implementing the workflow as follows: after I receive response from crossref indicating metadata deposit submission is successful, I just set resource status as published and immutable and change science metadata accordingly treating the resource as published. The citation of the resource automatically changes to resource DOI after it is published. If a user happens to click the resource's DOI link before the DOI is activated, the user will see a crossref page indicating DOI is not found with possible reasons including DOI is not yet activated, a sample page can be found here:  http://dx.doi.org/10.4211/hs.a312a8bffd334462b68e732752acdfac. After DOI is activated, crossref should take the user to the published resource landing page if the DOI link is clicked. Is this acceptable? If not, what do you suggest we should do with the resource during the period after we successfully submitted DOI registration to crossref and before the DOI is actually activated?

@hyi - I've had this happen with journal papers before.  The publisher sent me the DOI, which hadn't been registered/activated yet, and when I tried to use the dx.doi.org resolver to access the landing page for the paper, I got the page you sent us the example for.  One option is to just notify the user at the end of the publication process that their DOI will not be available until it has been registered and activated (which can take up to X amount of time).  But we will need some sort of automated process that verifies that the DOI does, in fact, eventually get activated.  If not, human follow up may be needed.

Another option is to disable editing on the resource at the time of publication but have an offline process that monitors the DOI until it is active and then have that offline process update the metadata for the resource to use the DOI.  So, the resource landing page would show the HydroShare URL until the offline process verifies the DOI is active.

@horsburgh I thought about this option as well, specifically using celery worker to query crossref API to check the status of DOI activation in the background thread. However, I am not sure how often we should run this background worker to achieve our purpose without impacting performance too much especially in the cases when human intervention is needed to contact crossref support if DOI activation for a specific resource has issues. I am bringing @ironfroggy in the loop to get his input. @ironfroggy Check info above for some background info on my specific questions.

@horsburgh @dtarb @ironfroggy @mjstealey After giving this some thoughts, I'd like to go with the option @horsburgh proposed above, i.e., notify the user at the end of the publication process that their DOI will not be available until it has been processed and activated (which can take up to X amount of time). I can write a monitoring script to run every midnight as a cron job to check all pending published resources by querying crossref API to see whether the pending published resources have been activated successfully, and compile a list of pending published resources that have not yet been activated as well as x amount of time the resources have been pending since submission and email<EMAIL_ADDRESS>the info as needed. This way, hydroshare support team will get notified each day the pending resources to be published and x amount of pending time if any, and follow up as needed. Let me know whether this approach is acceptable.

I agree with this approach.

I agree with the approach as long as this is not a CRON job but a Celery task as originally proposed.

@hyi - some issues based on testing on dev.hydroshare.org today. Many of these would need to modified in the data publication agreement page since you are displaying that in the pop-up in the publication workflow.

[ ] When I click on the “Terms of use” link in the data publication agreement, it puts the terms of use in the text box, but I can’t go back to the publication agreement.  I think it should open the terms of use in a new page (like it does with the link to NSF’s data sharing policy).
[ ] Same issue with the statement of privacy link.
[ ] In the data publication agreement, change “Tufts” to “Tufts University” and “Purdue” to “Purdue University”
[ ] In what is now the first item in the numbered list – make the word data plural in all cases - “in-situ data were collected”, “ex-situ data were analyzed”, “data are published"
[ ]  In the Disclaimer, change the following text “Resources accessed through the HydroShare…” to “Resources accessed through HydroShare…” (get rid of “the”)
[ ] Change the last statement to “CUAHSI will not accept liability for damages of any kind that result from using HydroShare or its content resources."
[ ] Number one in the numbered list of items should be something like "I hereby grant to CUAHSI and the HydroShare Team a non-exclusive publishing and distribution license for the content of this resource in all forms and media (whether now known or hereafter developed) effective when I accept this publication agreement."

@horsburgh please also make these changes on the Google Doc publication agreement as that is what Rick Hooper has been asked to have CUAHSI approve.

@dtarb - I made the modifications to the Google doc.

@horsburgh @dtarb I have addressed all your comments and redeployed beta.hydroshare.org for your further testing. Please test it out on beta.hydroshare.org and let me know if there are any further comments to be addressed.

@hyi - I just published another resource on beta.hydroshare.org and things looked good to me. See my comments about the DOI registration stuff inline with your code. I'm excited to have this capability in HydroShare!!

Closing this issue since this feature has been deployed.


updating wrong app while trying to get other app

Describe the bug
trying to get .net framework 7 just update win-get not actually installing .net
To Reproduce
Steps to reproduce the behavior:

Go to 'main page'
Click on '.net desktop runtime 7'
click on 'install selected'
See error

output
Chocolatey Already Installed
Checking if Winget is Installed...
Winget v1.6.3482
- Winget up-to-date
Winget Already Installed
Found an existing package already installed. Trying to upgrade the installed package...
No available upgrade found.
No newer package versions are available from the configured sources.
===========================================
--      Installs have finished          ---
===========================================

Well that means you already have this package installed.
So it's not necessary to install it again.

This issue should be fixed when PR #1662 is reviewed & merged to the project, if you're curious about the solution, then I suggest reading the Description of that PR, our apologies for the inconvenience, and hope you've a nice day.

@blueswill
I also thought that the .NET Runtime 7 was installed previously, but after double-checking, it was not installed.
I installed it manually using winget.
Thanks.

@og-mrk
It seems that your commit will fix this issue.
I'm curious to know if there's an extra log file rather than terminal output that is more verbose and helps us fix these types of problems.
If that's not the case, I think we can at least add package name(s) to the terminal log, resulting in something like this:
.NET Runtime is already installed.
No upgrade available for .NET Runtime.
Instead of:
Found an existing package already installed.
No available upgrade found.

@og-mrk
It seems that your commit will fix this issue.
I'm curious to know if there's an extra log file rather than terminal output that is more verbose and helps us fix these types of problems.
If that's not the case, I think we can at least add package name(s) to the terminal log, resulting in something like this:
.NET Runtime is already installed. No upgrade available for .NET Runtime.
Instead of:
Found an existing package already installed. No available upgrade found.

Hello @sharif221,
my bad for the late respond, and this feature you're suggesting is possible to do with a few modification to the one or more script files in the repo.
I personally am not very good at PowerShell, so I won't be working on this feature, maybe one of the Main Contributors or anyone for the task can do this.

[!NOTE]
The project is currently under a Major Re-write, so any new features or ideas will be looked into, but won't be added at the time been...


JIT slows down execution by 10x

Description
While developing some optimisation code I find a case where JIT is 10x slower than not JIT-ing. In making an MCVE I've tried  cutting away as much as possible, however apologies if the example is a little larger than I would like. Essentially, it's just doing a jax.scipy.sparse.linalg.cg inside a lax.scan. The matvec op of CG involves a JVP and VJP (the code of which is provided). Anecdotally, I'm starting to notice JAX perform slower in several situations where lax.scan is involved. What's going on here?
EDIT: added output
import dataclasses
import time
import warnings
from typing import Callable, Any
from typing import NamedTuple, Tuple

import jax
import jax.numpy as jnp

class ProblemState(NamedTuple):
    x: Any  # current solution, may be a pytree
    delta_x: Any  # step, may be a pytree
    damping: jax.Array  # damping factor
    F: Any  # residual, may be a pytree

@dataclasses.dataclass(eq=False)
class Problem:
    residual_fn: Callable
    num_steps: int = 0
    more_outputs_than_inputs: bool = False

    def create_initial_state(self, x0) -> ProblemState:
        x = x0
        delta_x = jax.tree.map(jnp.zeros_like, x)
        F = self.residual_fn(x)
        F_norm = pytree_norm(F)
        F_norm_delta = F_norm
        damping = F_norm_delta

        state = ProblemState(
            x=x,
            delta_x=delta_x,
            damping=damping,
            F=F
        )
        return state

    def solve(self, state: ProblemState) -> ProblemState:
        def build_matvec(J: JVPLinearOp, damping: jax.Array):
            def matvec(v):
                JTJv = J.matvec(J.matvec(v), adjoint=True)
                return jax.tree.map(lambda x, y: x + damping * y, JTJv, v)

            return matvec

        J_bare = JVPLinearOp(fn=self.residual_fn, more_outputs_than_inputs=self.more_outputs_than_inputs)

        def body(state: ProblemState, unused) -> Tuple[ProblemState, None]:
            J = J_bare(state.x)
            JTF = J.matvec(state.F, adjoint=True)
            matvec = build_matvec(J, state.damping)
            delta_x, _ = jax.scipy.sparse.linalg.cg(matvec, jax.tree.map(jax.lax.neg, JTF), x0=state.delta_x)

            state = ProblemState(
                x=state.x,
                delta_x=delta_x,
                damping=state.damping,
                F=state.F
            )

            return state, None

        state, _ = jax.lax.scan(body, state, jnp.arange(self.num_steps))

        return state

def pytree_norm(pytree: Any) -> jax.Array:
    square_sum = jax.tree.map(lambda x: jnp.sum(jnp.square(jnp.abs(x))), pytree)
    leaves = jax.tree.leaves(square_sum)
    total_square_sum = sum(leaves[1:], leaves[0])
    return total_square_sum

def isinstance_namedtuple(obj) -> bool:
    """
    Check if object is a namedtuple.

    Args:
        obj: object

    Returns:
        bool
    """
    return (
            isinstance(obj, tuple) and
            hasattr(obj, '_asdict') and
            hasattr(obj, '_fields')
    )

@dataclasses.dataclass(eq=False)
class JVPLinearOp:
    """
    Represents J_ij = d/d x_j f_i(x), where x is the primal value.

    This is a linear operator that represents the Jacobian of a function.
    """
    fn: Callable  # A function R^n -> R^m
    primals: Any | None = None  # The primal value, i.e. where jacobian is evaluated
    more_outputs_than_inputs: bool = False  # If True, the operator is tall, i.e. m > n
    adjoint: bool = False  # If True, the operator is transposed
    promote_dtypes: bool = True  # If True, promote dtypes to match primal during JVP, and cotangent to match primal_out during VJP

    def __post_init__(self):
        if not callable(self.fn):
            raise ValueError('`fn` must be a callable.')

        if self.primals is not None:
            if isinstance_namedtuple(self.primals) or (not isinstance(self.primals, tuple)):
                self.primals = (self.primals,)

    def __call__(self, *primals: Any) -> 'JVPLinearOp':
        return JVPLinearOp(fn=self.fn, primals=primals, more_outputs_than_inputs=self.more_outputs_than_inputs,
                           adjoint=self.adjoint, promote_dtypes=self.promote_dtypes)

    def matvec(self, *tangents: Any, adjoint: bool = False):
        """
        Compute J @ v = sum_j(J_ij * v_j) using a JVP, if adjoint is False.
        Compute J.T @ v = sum_i(v_i * J_ij) using a VJP, if adjoint is True.

        Args:
            tangents: pytree of the same structure as the primals.
            adjoint: if True, compute v @ J, else compute J @ v

        Returns:
            pytree of matching either f-space (output) or x-space (primals)
        """
        if self.primals is None:
            raise ValueError("The primal value must be set to compute the Jacobian.")

        if adjoint:
            co_tangents = tangents

            def _get_results_type(primal_out: jax.Array):
                return primal_out.dtype

            def _adjoint_promote_dtypes(co_tangent: jax.Array, dtype: jnp.dtype):
                if co_tangent.dtype != dtype:
                    warnings.warn(f"Promoting co-tangent dtype from {co_tangent.dtype} to {dtype}.")
                return co_tangent.astype(dtype)

            # v @ J
            primals_out, f_vjp = jax.vjp(self.fn, *self.primals)

            if isinstance_namedtuple(primals_out) or (not isinstance(primals_out, tuple)):
                # JAX squeezed structure to a single element, as the function only returns one output
                co_tangents = co_tangents[0]

            if self.promote_dtypes:
                result_type = jax.tree.map(_get_results_type, primals_out)
                co_tangents = jax.tree.map(_adjoint_promote_dtypes, co_tangents, result_type)

            del primals_out
            output = f_vjp(co_tangents)
            if len(output) == 1:
                return output[0]
            return output

        def _promote_dtype(primal: jax.Array, dtype: jnp.dtype):
            if primal.dtype != dtype:
                warnings.warn(f"Promoting primal dtype from {primal.dtype} to {dtype}.")
            return primal.astype(dtype)

        def _get_result_type(primal: jax.Array, tangent: jax.Array):
            return jnp.result_type(primal, tangent)

        primals = self.primals
        if self.promote_dtypes:
            result_types = jax.tree.map(_get_result_type, primals, tangents)
            primals = jax.tree.map(_promote_dtype, primals, result_types)
            tangents = jax.tree.map(_promote_dtype, tangents, result_types)
        primal_out, tangent_out = jax.jvp(self.fn, primals, tangents)
        return tangent_out

def main():
    n = 1000
    m = int(n * (n - 1) / 2)
    s = 3

    alpha = jnp.ones((m, s))
    idx1 = jax.random.randint(jax.random.PRNGKey(0), (m,), 0, n)
    idx2 = jax.random.randint(jax.random.PRNGKey(1), (m,), 0, n)

    def forward(params, alpha):
        g1 = params[idx1, :]
        g2 = params[idx2, :]
        return jnp.sum(g1 * alpha * g2.conj(), axis=1)

    def residuals(params):
        return forward(params, alpha)

    x = jnp.ones((n, s))

    lm = Problem(residual_fn=residuals,
                 num_steps=1)

    state = lm.create_initial_state(x)

    solve = lm.solve

    t0 = time.time()
    jax.block_until_ready(solve(state))
    run_time = time.time() - t0
    print(f"Run time (no JIT): {run_time}")

    # AOT JIT
    solve = jax.jit(lm.solve).lower(state).compile()

    t0 = time.time()
    jax.block_until_ready(solve(state))
    run_time = time.time() - t0
    print(f"Run time (JIT): {run_time}")

if __name__ == '__main__':
    main()

Output:
Run time (no JIT): 0.3084075450897217
Run time (JIT): 4.048235177993774

System info (python version, jaxlib version, accelerator, etc.)
jax:    0.4.31
jaxlib: 0.4.31
numpy:  1.26.4
python: 3.11.9 (main, Apr 19 2024, 16:48:06) [GCC 11.2.0]
jax.devices (1 total, 1 local): [CpuDevice(id=0)]
process_count: 1
platform: uname_result(system='Linux', release='5.15.0-117-generic', version='#127~20.04.1-Ubuntu SMP Thu Jul 11 15:36:12 UTC 2024', machine='x86_64')

Solved my own problem, but would still love a kind explanation.
Basically, looks like some closure problem.
def forward(params):
    alpha = jnp.ones((m, s)) # <--- This was brought in via closure before
    idx1 = jax.random.randint(jax.random.PRNGKey(0), (m,), 0, n) # <--- This was brought in via closure before
    idx2 = jax.random.randint(jax.random.PRNGKey(1), (m,), 0, n) # <--- This was brought in via closure before
    g1 = params[idx1, :]
    g2 = params[idx2, :]
    return jnp.sum(g1 * alpha * g2.conj(), axis=1)

Run time (no JIT): 1.3602826595306396
Run time (JIT): 0.0619661808013916

Hello ...
I think...
Closure in JIT: When you pass a function or variable into a JAX operation via a closure, JAX needs to trace the entire computational graph. If the closure captures variables like arrays or random keys, it can cause JAX to recompile each time, leading to significant overhead.
Avoid closures with dynamic variables: When using JIT in JAX, avoid capturing variables in closures if possible. Instead, pass them explicitly or define them within the function.
i hope it might help!

That still doesn't explain the 10x difference in run time. I can't see how capturing a few small arrays leads to such significantly longer execution time. What is that time being spent on? Note this has nothing to do with compile time, as that is done before benchmarking. @jakevdp do you have a quick explanation for this?

I don't know the answer to this specific question, but just chiming in here on some potentially relevant debugging tips. You can always inspect what the function compiles to using the following snippet:
c = jax.jit(f).lower(*args).compiler_ir('hlo')
print(c.as_hlo_text())

This will print out the XLA/HLO and you can inspect the difference between the two cases.

Closing as this is solved


German Language, comma fix

German Language de-DE

wrong commas

Hi @osworx, and thank you so much for your translation!
As it is a JavaScript file, the commas shouldn't be an issue, but you're right, it is cleaner like that :)


Move copy for arg's in-place update right after its last consumer

          Would it make sense to try to move the copy as early as possible by analyzing usage, instead of defaulting to the end, since this will have memory implications?

Maybe we can open an issue for this. Also let's add an issue with the proposal for batchnorm.
Originally posted by @lantiga in https://github.com/Lightning-AI/lightning-thunder/pull/675#pullrequestreview-2147432669

I think this reordering happens because in general fusing passes do topological sorting of operations grouping things together and in practice, I don't think this buffer tensor lives long in the program. But it needs to be checked what actually happens.

functionalize_inplace_ops puts all the required prims.copy_'s right before the return stmt. Luca and I were thinking about moving each copy right after the last consumption of its destination


Workaround for mergeBufferGeometries failure when loading Collada files with inconsistent attributes

Currently mergeBufferGeometries fails when a Collada file is loaded that has an inconsistent set of attributes in the contained geometries. I noticed this when loading a mesh (anymal_base.dae) that contains uv attributes for some of the nodes but not for all of them. In those cases, mergeBufferGeometries would write an error to the console and then return a null object resulting in a load failure.
This PR is a workaround which simply removes all textures from the loaded and merged geometry if this failure would otherwise be triggered. It's not a pretty solution but is better than not showing any geometry at all. Open to hear about better suggestions on how to fix it :-)

In your case, are you loading the DAE using _meshfile_object or _meshfile_geometry?  (See #166 for documentation of each -- on master once that lands.)  The anymal_base.dae looks to me like it has defined material(s), texture image(s), etc. so to me it should be loaded with _meshfile_object and not _meshfile_geometry.  (Currently meshcat-python lacks the ability to send _meshfile_object messages, but support for that really should be added, and should be simple to do.)
At the moment, I think the problem would still occur with _meshfile_object but it does open up another path to a solution: when loading with _meshfile_object we don't really need to call ExtensibleObjectLoader and merge_geometries.  Instead, we can do what the *.gltf loader already does -- load the *.dae file as a scene and just plop the whole scene tree into three.js, without any geometry merging.  In that case, threejs would just show the loaded scene however it got loaded; meshcat doesn't need to get in the way.

Please see #170 for active work on having _meshfile_object support DAE more carefully.  We would welcome any help getting that one over the finish line.  In the meantime, I'll close this PR to help focus our efforts on a single branch.