How to use the Serve object ?

I've been trying to make a hyper server run on the same context/core as a tokio TCP server.
However this kind of usage seems to be poorly documented (or I'm unable to find/read the docs properly). currently my code looks like this:
    let mut core = Core::new().unwrap();
    let handle = core.handle();

    let server_addr = "<IP_ADDRESS>:1834".parse().unwrap();
    let serve = Http::new().serve_addr_handle(&server_addr, &handle, || Ok(DataServer)).unwrap();

    let server = serve.for_each(move |f| {
        Ok(())
    }).map_err(|err| {
        println!("IO error {:?}", err);
    });

    current_thread::run(move |_| {
        current_thread::spawn(server);
        current_thread::spawn(receptor); // <--- This is that other TCP server, it works just fine
    });

This code runs just fine and is listening on port 1832, however the call() method of DataServer is never called.
Any idea what am I doing wrong here and how I should go about this ?

You're right, it is poorly documented. The reason is because it's a newer feature, but I wasn't quite certain how well it would play with the newer tokio and futures. As that becomes more certain, the whooe server module will get some documentation about the 2ish APIs, low level and high level.
For your immediate case: Serve is a stream of Connections, bound to a service. However, each Connection is itself its own Future, needing to be executed. In your for_each call, you should spawn them onto the executor.

Also, you likely didn't show your whole program, so maybe it's not an issue, but that Core you created will need to be running for the IO in hyper to work.

Sorry, the code is actually wrong, I modified it a lot to figure out how to make it run and this version is plain wrong since it doesn't use the core I created for IO but rather the current thread :/
The version of the code I ended up using is:
 let server_addr = "<IP_ADDRESS>:1834".parse().unwrap();
 let server = Http::new().bind(&server_addr,|| Ok(DataServer)).unwrap();

 let movable_handle = server.handle();
 let receptor = listener.incoming().for_each(move |stream| {
     //movable_handle used here to spawn futures
 });

 server.handle().spawn(receptor);
 server.run();

Which basically means I'm taking the handle from the server and running everything on whatever  context/core the server is using (which I assume is on the current thread of execution).
The alternative would be to create the core and use a handle from it to spawn all processes and I assume that would work, though I'm happier with this (less code and as far as I can tell exact same result).
What I'm unsure of is if there is a way to accomplish this (running two servers on the same event loop) w/o this  "hack" and without going into tokio-core, as in, only using things like current_thread and in some way binding the execution of the server to that ?

The unfortunate thing is that you end up with 2 reactors (epoll event notifiers), one for HTTP and one for your TCP server. The TCP one is probably tokio's new default, in a separate thread.
I don't know the best way to get them working together, I probably wouldn't recommend trying to do it just yet. hyper will upgrade to the new tokio soon, once the futures 0.2 release happens.
cc @carllerche

Wait, so the above snippet of code would create tow reactors ? I assumed that in that case the TCP server would just use the hyper's server thread+notification-loop.
Also, a bit ot, but does tokio use epoll instead of poll ? And if so, would this stop hyper from working on non-linux systems ? Or would it just fall back to poll ?

Well, in neither example do I see how you create the TCP server, so I can't say for certain, but seeing the current_thread API suggests to me that you're using tokio, not tokio_core. If you're using both, then yes, they have currently have incompatible reactors, and so likely using 2.
Tokio uses mio, which abstracts over the various selection strategies in a cross platform way (so epoll on linux, kqueue on bsd, IOCP on windows).

I'm not using current_thread anymore, but yes, the TCP server is created using Tokio not tokio_core and ps shows the program apparently spawns to threads... -_-.
I'm quite unsure why the TCP servers would use it's own thread though, I spwan it using the server's handler and I spwan the futures inside it's callback using the same handler (not current thread).
Can a process two separate futures be spawned on the same handler but use different threads to run ? Or does the server just start a new thread by default when I call server.run(); and the main thread stays inactive ?

The new design of Tokio is that by default, it spawns a global reactor, putting it in its own thread. This happens if you use Handle::default() (or functions on things like TcpListener that don't take a Handle argument). You can prevent the global by creating a reactor yourself, and passing references to it (its Handle).
However! hyper does not yet use new Tokio, but instead the older tokio-core, So, it doesn't know about global reactors, and in fact, its Handle is technically a different type, so you cannot share them. This means that to drive your newer-Tokio TCP server, and the older-tokio-core hyper server, you need 2 reactors.
This might seem like poop, but don't worry, you just jumped in right during a transition period. Tokio is transitioning, and the new tokio 0.1 crate is a preview of the new design. The futures crate will see a list of breaking changes to version 0.2 in a couple weeks, and when that happens, tokio will upgrade to 0.2, and hyper will upgrade to both. After that, you no longer have the situation you're in at the moment.

(I'm going to close this issue, since it isn't about a bug in hyper.)


Nouveau salon: media permissions (permission.CAMERA,...) are not checked correctly

The new user is not able to set a picture when he wants to create a new "salon"
When the room is created, he succeeds to add a picture after giving the permissions.

Please consider the PR #212 as it avoids the need of the CAMERA permission.


Improve accuracy of ICA C implementation

ICA C implementation is compiling, running, and producing accuracies between ~25%-70%.  There are plenty of optimizations that can be incorporated and perhaps a missing line or two from the Matlab. Could use another pair of eyes to go through it and see if I missed anything.

Some notes on what Colin and I did today:

We implemented the formula for icasig in the MATLAB code because the C code didn't implement the whole formula; however, we did not see an increase in accuracy in the test run that we used (1-5 removed from each class, 27.5% in both cases
The MATLAB ICA code is actually calling remmean twice: once in run_ica, in which it removes the mean image, and once again in fastica, in which it removes the mean "row" rather than the mean image because mixedsig = X'

Some things that I plan to do once I have time:

move matrix functions from ica.c to the matrix library
remove memory leaks
clean up the code, uniform naming conventions, etc
combine temp_PCA with the main PCA function

Once I have done my clean-up, we should run cross-validation so that we can see how the accuracy of the C code changes as the training set shrinks.
Meanwhile, it might be useful for some people to play around with the parameters in the MATLAB ICA code that were not included in the C code and document their findings so that we know which parameters are worth keeping.

Okay, after a few debug sessions throughout last night and today, I was able to fix the issue we had with NaNs in our matrices. A few points:

The PCA_alt function in ica.c was returning an extra eigenvalue with value 0, because an N-by-N symmetric matrix has at most N-1 eigenvalues greater than 0. The eigenvalues are later diagonalized and inverted, which introduced some very large numbers. I changed PCA_alt to remove the extra eigenvalue, but I may modify m_eigen in the future to automatically remove these zero columns. Anyway, this change got rid of the NaNs.
In the MATLAB version of fpica, the inner loop actually breaks when the w vector converges, but this break statement was never added to the C code because the MATLAB indentation is so bad. Anyway, I added the break and fixed up the convergence code so our C code is now considerably faster.

In summary, the result of my changes is that the ICA C code seems to produce very similar results to the MATLAB code, including intermediate values. However, even with the break statement, the C code is still significantly slower. I would assume that many of the operations in the MATLAB code are multi-threaded, but there may be other differences. We'll need to add timing information to the ICA code so that we can identify any bottlenecks.

Converting the matrix library to single precision seems to have improved the performance of ICA somewhat.

Accuracy is on par with Matlab


Feature request: Full Screen mode / Hide headers / headless mode

Is your feature request related to a problem? Please describe.
When I split my terminal horizontally, I would like to hide the headers (showing shortcuts and cluster information) because I have not so much space for this information frame.
Describe the solution you'd like
I would like a shortcut allowing to toggle the headers area on and off to maximize the table height.
To me, the only information being important at all time in the header is the context, so it may be a good idea to keep it in "full screen mode".
Describe alternatives you've considered
Other (harder) alternative should be a ctrl + and ctrl - to increase/decrease the area and adjust it ( please note there are always 3 empty lines above the title of the table)
I have no workaround for now and I deal with it by putting the terminals on a bigger screen or by lowering down the char size...
Additional context
Similar to https://github.com/derailed/k9s/issues/270
If unclear I'll provide more context
Off topic : Great Work for this wonderful app :clap: :+1:

Hi @viniciusartur , thanks for opening the issue. I haven't worked on wander in the past little bit but hoping to get back to it soon. I'll try to implement this when I do! I think it's a great idea and would use it myself.
Would pressing - with control held down for compact mode and then + with control held down for regular mode make sense to you? I'll add an option to start sessions in compact mode as well.

Sounds great, @robinovitch61.
A little digression: I was wondering that in k8s we can have a view of all pods in all namespaces, so if one pod is not healthy we can see the red color. This is good as we can use k8s as a dashboard to have a big picture of all pods. I was wondering if we could see all tasks in all namespaces in wander.

Sounds great, @robinovitch61.

Sweet.

A little digression: I was wondering that in k8s we can have a view of all pods in all namespaces, so if one pod is not healthy we can see the red color. This is good as we can use k8s as a dashboard to have a big picture of all pods. I was wondering if we could see all tasks in all namespaces in wander.

Sounds reasonable! Opened a separate issue for tracking here https://github.com/robinovitch61/wander/issues/87

Merged to next

Released in v0.10.0


Class and function documentation

Having basic documentation of the functionality of classes and functions would be quite nice.
Following discussions in person, I generated automatically documentation starting for the global constraint classes. The  documentation was auto generated using codiumAI (pycharm extention).
Let's see if it is good enough and if yes we can proceed with more functions and classes.
In the first commit, only the text generated automatically is included. In following commits I will check the texts and maybe change some parts if needed

I both like and hate this: it is 95% correct, and figuring out where it is incorrect is time consuming. I'd allow this if we put an "AI-generated"-tag next to it. People can remove that tag once they checked it manually.
It reminds me of why StackOverflow forbade the use of AI-generated answers: it looks correct, but figuring out whether it is correct is hard.


Microsoft.Azure.Devices.Gateway.Native.Windows.x64 nuget doesnt build

Hi,
I'm trying to create a Gateway in .Net Core application using Microsoft.Azure.Devices.Gateway.Native.Windows.x64 nuget package. However as you can see below it doesn't resolve:

Regards,
Ali

Needed to add a reference to .Net Core binding - https://github.com/Azure/iot-edge/tree/master/bindings/dotnetcore/dotnet-core-binding/Microsoft.Azure.Devices.Gateway


Improve usability of HashSet/Map type aliases

Extracted from https://github.com/ogxd/gxhash/pull/35

Had to fixup an issue with my git config before repo gets too large / too many contributors to do it. It messed up this PR but no worry I have rebased it: https://github.com/ogxd/gxhash/pull/56


Need new SQL dump on request.

I am Master's student writing my thesis on transaction analytics of Casper. I am looking for an indexer project which I can use to collect Casper transaction data for an analytics project. Can admin please provide an updated dump which can be used for academic analytics purpose only ?
Thank.

Hello @Ankan0011 here is database carefully exported for your needs at block 1207880. If we can help on something also please let us know. https://bit.ly/3VW08Se Here is unpublished feature of the platform which can be interesting: https://caspermetrics.io/stake you can test by using any ART3MIS delegator public key as for example: 01a60e0885f4968dd3107e088b4cb5798af665859b769bc1aa86909a5b67f66a66 Good luck on research !

Thanks for uploading the updated dump. I just want to confirm a few things regarding the dump tables. Is the Users tables contains the details of all the validators as well, I also looking for the address/id's of the validators as well ?

@Ankan0011 It is a table for the administation panel. You can keep it empty.

There is no valuable info into it

@Ankan0011 currently you can't easily get the validators details from the database. It exist experimentally for one validator, but it might not be helpfull for your purposes. Let us know please if we can help with something else and thank you!

Thanks a lot for your help. I will keep looking for validators info from public API's.

Hi, can I ask for an up-to-date snapshot of Casper mainnet data?

Hi, can I ask for an up-to-date snapshot of Casper mainnet data?

Hope it helps: <IP_ADDRESS>/mysqldump_22_May_2023_13_35_19.sql

Thanks @matsuro-hadouken.


NRF52840 i2c pin config handles Port1

@FabienLep latest change looks good, however your PR also includes commit than changes repository version to 1.5.0 which is not ok. I suspect you created fix on 1.5.0 and then created PR for master branch. You need create fix on master branch and the you can force push the result to the same branch on GitHub to update this PR.

@andrzej-kaczmarek , updated branch with changes from master. tell me if that is ok


[REQUEST] [BS-TOOLTIP] Option to indicate to a bs-tooltip that the formLayout is vertical

Right now, when using the bs-tootip component with placement='right' in a form, it display the tooltip at the right top of  the control. But initially, it was supposed to be displayed at the right, but this time centered.
This is the code:
{{#bs-form formLayout="vertical" as |form|}}
//                                         ^

  <fieldset>
    <legend>{{Configuration}}</legend>

      {{#form.element controlType="text" label='Name'
                      placeholder='Name'
                      property="name" as |el|}}
        {{el.control}}
        {{#bs-tooltip placement="right" triggerEvents='hover'}}
           {{'Name of the test'}}
        {{/bs-tooltip}}
      {{/form.element}}
 </fieldset>

{{/bs-form}}

And this is what I got:

So an option to indicate to the tootip, if it is in a form, that the formLayout='vertical' can be useful.

Output from ember bootstrap:info (available since 1.0.0-alpha.9):
Npm packages:
ember-bootstrap: 1.0.0-rc.4 -> 1.0.0-rc.4
ember-cli: 2.16.2 -> 2.16.2
bootstrap: 3.3.7 -> 3.3.7
bootstrap-sass: n/a
ember-cli-sass: n/a
ember-cli-less: n/a
Bower packages:
bootstrap: n/a
bootstrap-sass: n/a
ember-bootstrap configuration:
bootstrapVersion: 3
importBootstrapFont: true
importBootstrapCSS: true

I fixed it, but the is a workaround, not the official solution.
Just add in your CSS stylesheet :
.tooltip .right {
  margin-top: 10px;
}

UPDATE Thanks to the slack community, they helped me  understand how to use the bs-tooltip. There is no need to do some hack, you just have to link the element of the {{form.element}} to the bs-tooltip, using an id.
If we take the snippet above:
{{#form.element controlType="text" label='Name'
                      placeholder='Name'
                      property="name" as |el|}}
        {{el.control id="Tooltip-name"}}
        {{#bs-tooltip placement="right" triggerElement="#Tooltip-name" triggerEvents='hover'}}
           {{'Name of the test'}}
        {{/bs-tooltip}}
{{/form.element}}

If you are using a specific components instead of a controlType, the idea is the same:
{{#form.element property="name" label="Name"}}

      {{#power-select
            selected=name
            options=allName
            onchange=(action (mut name))
          as |name|}}
            {{name}}
       {{/power-select}}
         {{#bs-tooltip placement="right" triggerElement="#Tooltip-name" triggerEvents='hover'}}
           {{'Name of the test'}}
        {{/bs-tooltip}}

{{/form.element}}

This is indicated in the doc here: Look at the very end of the page


0.16.0 release

It'd be nice to issue a release soon. We should decide if this is a minor 0.15.2 or major 0.16.0
Please edit this list as you wish.
Must-have

[x] #3936 multiple dims argmin, argmax

[x] #4019, #4088; MultiIndex to sparse DataArray bug

[x] #3922 fix dask array handling in idxmax, idxmin

[x] #4009, #4173 combine_attrs with open_mfdataset

[x] #3824 transpose coords (if major release)

[x] #3926 remove old autocombine

[x] #4135 nD dask arrays in idxmax/min

Nice to have

[x] #4049, #4094 unstacking merge bug
[ ] #4022 apply_ufunc meta vectorize
[ ] #3924 interpolating to coordinates with nans
[ ] #3925 sel along unindexed 1d coordinates
[ ] #3993 replace dim with coord in integrate
[ ] #3594 unit support with pint
[ ] #4003 mfdataset with zarr
[x] #3905 length of dataarray reprs
[x] #3847 error summary for assert_allclose
[x] #4033 infer_freq
[x] #3938 plotting with multiindex
[x] #3816 map_blocks template
[x] #3976 in-place replacement error message
[x] #3975 pint dataset
[x] #4018 netdf3 dtype coercion
[ ] #3985 groupby multi-index bug (ref discussion below & in the issue)

[ ]  #4075 passing boolean weights to weighted mean

would be nice

As discussed on the dev call, I'll take this one.
I think we need to do 0.16.0 as we had some backward-incompat changes (I can't remember now what they were; but can hunt them down if anyone otherwise has a preference for 0.15.2?)

Yes we have some breaking changes so definitely 0.16.0
IMO we should do #4184 #4175 and #4163 and then release. #3985  is still waiting for a minimal example so perhaps we should leave that for the next release.

Agree! I'll keep an eye on those this week.

Thanks for updating @dcherian . Is it worth waiting for #3985? I'm still not sure exactly what the bug is there. I'd vote to leave it until we have a repro.
I was thinking of sneaking in https://github.com/pydata/xarray/issues/3891 but didn't want to bring it up as probably won't be able to; definitely not worth waiting on me for!

I'd vote to leave it until we have a repro.

:+1: I'm also not sure what the bug is. In fact it's possible that a bug was fix and a workaround is what's raising the error now :P
Re: #3891 do you mean set keep_attrs=True by default?

Re: #3891 do you mean set keep_attrs=True by default?

Exactly, so this will push to 0.17

FYI I will do this towards the end of this week or at the latest this weekend

Thanks @max-sixty !


on windows, table breadcrumb has vertical scrollbars

This only seems to affect Windows. Linux and macOS are fine.

Adding overflow-y: hidden to pf-c-breadcrumb__list seems to remove both scrollbars shown in the above image.

Adding overflow-y: hidden to pf-c-breadcrumb__list seems to remove both scrollbars shown in the above image.


Allow Download File Type dropdown to be deselected

Description
Add a default selection to the Download File Type dropdown so that it can be deselected.

Fixes #674
Type of change

Bug fix (non-breaking change which fixes an issue)

Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes

Rebased onto release, hopefully the tests are fixed

@daniel-shuy I'm really sorry about this but can you please update your branch again with the latest code from release? There were some breaking changes introduced in release which is probably causing tests to fail on your branch. We've fixed those tests and the CI code as well.
The tests should then pass on this branch.

@mohanarpit it's passing now, thanks!


Fix for ES Module detection using npm@7 (#4295)

@richardsimko Bunch of previous tests seem to be failing, see https://github.com/knex/knex/pull/4296/checks?check_run_id=1904682577

Yeah I don't think my approach to mocking the env vars was good. I'll try another way but if that doesn't work I'm not sure it's possible to add tests for this code in a good way.

@richardsimko Unit tests are fixed, but it looks like something in migration loading logic might be broken, judging by non-mysql integration tests.

I think what's most likely happening is that I've uncovered latent issues where importFile was not used with await. import(filePath) is an async function and always was - but this function was only used in the ESM case which doesn't seem to be commonly used. Now importFile is always async and this is how that issue manifests. Fixing this would require a lot more in-depth knowledge of knex than what I have but if you have any pointers as to where this might be an issue I'm all ears.

importFile seems to be used here https://github.com/richardsimko/knex/blob/2cc3247d83e17cfb7d5788f6b391ab31faa4758b/lib/migrations/migrate/sources/fs-migrations.js#L84:L84
Which in turn is used in many places without await:
https://github.com/knex/knex/blob/2cc3247d83e17cfb7d5788f6b391ab31faa4758b/lib/migrations/migrate/Migrator.js#L493:L493
https://github.com/knex/knex/blob/2cc3247d83e17cfb7d5788f6b391ab31faa4758b/lib/migrations/migrate/Migrator.js#L145:L145
https://github.com/knex/knex/blob/2cc3247d83e17cfb7d5788f6b391ab31faa4758b/lib/migrations/migrate/Migrator.js#L86:L86

@richardsimko Yeah, noticed same thing: https://github.com/knex/knex/blob/b20a31fe5d0256234e1542c1ec4cbe2082f9970e/lib/migrations/migrate/sources/fs-migrations.js#L84 seems to be the only place where we importFile without awaiting, so I would review all cases where getMigration is used to ensure we await properly.

I can slap awaits on all the above mentioned locations but in my experience fixing these types of issues is never that easy 😅

@richardsimko You might need to make few more functions async (e. g. waterfallBatch), but that should do it. We have pretty good coverage for migrations at this point, if something breaks down, you'll know immediately.

It worked 🎉  I added a note in the changelog as well. Do you want me to squash the commits?

@richardsimko Thanks a lot!
Squashing is not worth the effort, I'll squash on merge anyway.

@kibertoad Is there anything I have to do in order to get this backported to 0.21.x?

@richardsimko There is a 0.21 branch, just open a PR targetting that.

Thanks, I opened #4308


What would the language code be for Indian/hindi/bollywood movies?

Trying to figure out what the language code would be for indian/hindi/bollywood movies

All language codes are normalized to the Alpha3 ISO standard, so you would need to reference that
https://en.wikipedia.org/wiki/ISO_3166-1_alpha-3


example |> getRow(idx: 0) wrong ???

Describe the issue here.
Relevant URLs

https://docs.influxdata.com/influxdb/cloud/query-data/flux/scalar-values/
Provide other relevant URLs (project issues, community threads, etc.)

I wonder if the field values is missing.
|> getRow(idx: 0)
idx:0 is set as default in the declaration, but field is missing
getRow = (tables=<-, field, idx=0) => {
as I'm still a newbee I could be wrong
oh now, it is the other way around. in the function definition field is not needed
again: examples with errors are a pain for beginners

@antonmeyer Thanks! You are absolutely right. I'll update the example.

|> getRow(field: "temp") does not cause an error message,
but field is not used in getRow


Update gif image and readme - Collaborative Stage View

@Harikrishnan-MSFT - Kindly run build pipeline before merging this PR.

Build Pipeline -
https://github.com/OfficeDev/Microsoft-Teams-Samples/actions/workflows/build-complete-samples.yml


[RFC] Sparsity Future Plans

I had a chance to reflect after PTC / CUDA-MODE and wanted to share some thoughts on future plans for sparsity in torchao.
Current State
There are two components of sparsity, accuracy and acceleration.

Accuracy consists of finding which parameters to zero-out/remove and how to recover model task performance afterwards.
Acceleration consists of removing computation from the model after zeroing out parameters to make it fast.

torchao has primarily focused on the acceleration side, with our 2:4 and block sparse kernels and quantization composition. We do offer some prototype features (WeightNormSparsifier, superblock) for accuracy but these are not our main focus. We felt that these GPU sparsity patterns offered a “godilocks” sweet spot in the accuracy/acceleration tradeoff and focused on making them easily accelerable.
We still believe this to be true, but research in accuracy has evolved from when we first released these APIs. While post-training methods continue to be researched, there’s been a lot of development in 1) distillation and 2) activation sparsity, which we’d like to better support moving forward.
Proposal
There are two things concretely that we’d like to do:
1) Run sparsity (2:4, block sparse) + distillation experiments for LLaMA 3.
NVIDIA was able to use distillation + pruning to halve the size of LLaMa 3.1, and the new LLaMa 3.2 1B models were trained with distillation.
Instead of shrinking the model with pruning, we can also do so with weight sparsity. Theoretically, the extra granularity given to us by block / semi-structured sparsity would allow for us to push the overall sparsity level of the models higher. For example, NeuralMagic has trained their 2:4 sparse models LLMs using distillation. We can adapt the torchtune distillation tutorial to get some initial benchmarks.
2) Fast run-time compression routines for  2:4 sparsity
The idea of activation sparsity papers like CATS / TEAL is to swap out the SwiGLU activation for a ReLU-esque (zero-based) activation function, which can yield up to 80% sparsity in the activations.
In the memory bound case, we can accelerate mamtul (GEMV) by selective loading only the rows of W that we need, which is very similar to how we achieve speedups with row-wise structured sparsity.  Researchers have been looking into using 2:4 sparsity instead for speedups in both the memory-bound and compute-bound case.
For activation sparsity, this means we need run-time sparsification routines, since we can’t compress the tensor offline. We’ve written these routines for 2:4 sparse training, but via custom CUDA kernels.
We’d like to see if we can expand these routines to be torch.compile able, so that they could be fused as part of other calculations, like per-row FP8 scaling. This will also allow us to explore future optimizations, i.e. fusing a permutation into the model so that we can accelerate more of the computation, using an approach similar to https://arxiv.org/pdf/2408.11551 or adding support for new dtypes like FP8 / INT8.
An open question here: Would support for more flexible accuracy patterns via a permuted / shuffled 2:4 sparse / block sparse matrix be interesting?
The way I see it, we can potentially get the best of both worlds by shuffling unstructured sparsity to an accelerable GPU sparse format.
Run-time activation sparsity is also important for TTFT for LLM inference. When we are compute-bound in prefill, we can use 2:4 sparsity to speedup up inference. However, once we start decoding we are memory-bound, and would no longer like to use our sparse kernels. A run-time sparsification kernel would let us use the same model for prefill + decode.
I ran some preliminary benchmarks for FP8 + 2:4 sparsity on LLaMa3 and observed the following speedups on TTFT.

Prefill batch size
Sparse TTFT
Dense TTFT
Speedup

8192
0.2643 sec
0.2968 sec
1.12x

16384
0.9132 sec
0.9664 sec
1.06x

32768
3.2265 sec
3.4839 sec
1.08x

Work Items

Core:

[DONE] cuSPARSELt version update  https://github.com/pytorch/pytorch/pull/134022

[INPROGRESS] Pip dependency  https://github.com/pytorch/builder/pull/2010
[INPROGRESS] https://github.com/pytorch/pytorch/pull/137653/files

[DONE] cuSPARSELt FP8 support  https://github.com/pytorch/pytorch/pull/136397
[DONE] Inductor mm lowering for alg_id, split_k params  https://github.com/pytorch/pytorch/pull/137427
[TODO] Rewrite existing fast-sparsification routine to be torch.compilable
[TODO] Add fp8 support for runtime-sparsification kernels
[INPROGRESS] hipSPARSELt support (?)
[TODO] add cuSPARSELt to fbcode build

AO

[INPROGRESS] Add TTFT benchmarks
[TODO] Add load-as-sparse-compute-as-dense LLaMa benchmarks
[TODO] Add distillation experiments

Additional Notes
If we create torch.compilable compression routines, we may be able to use them to accelerate the memory bound case by fusing together a mm with the decompression routine, basically creating a load-as-sparse compute-as-dense kernel.

Nice plan, I am working on hipSPARSELt  support and plan to release that at rocm 6.3/6.4-time frame.

cc @alexsamardzic @supriyar @cpuhrsch
2:4 CUTLASS GEMM work
Previously, we needed to use either cuSPARSELt or the 2:4 CUTLASS kernels in core for accelerated 2:4 sparse matmul, but with the addition of https://github.com/pytorch/ao/pull/880 we have the ability to land custom CUTLASS kernels into torchAO. There are a couple of interesting things we could do:
H100

Add support for row-wise scaled MM for FP8 with 2:4 sparsity. Currently cuSPARSELt lacks the ability to fuse any scalar multiplies into the matmul, so we would need to fuse scaling matmul into the surrounding ops. This works fine for tensor-wise FP8 scaling, but not more granular scaling (row-wise / group-wise) This is a H100 focused feature, and something that would help with the activation sparsity 2:4 work @danthe3rd. I belive that https://github.com/huyz2023/2by4-pretrain also is trying to do something similar.

A100

Enable int8+2:4 sparse attention support Currently we are limited to only applying int8 quantization + 2:4 sparsity to the MLP layers only of ViT-h, because we rely on cuSPARELt for fusing the scalar multiplications into the matmul. We'd like to do the same for the attention blocks, but because those are all fused, we cannot use cuSPARSELt without a large performance penalty. We know that we can recover the accuracy degradation in the attention blocks, as NeuralMagic's 2:4 sparse model sparsified both the attention and mlp weights.

2:4 sparse GEMM using the MARLIN int4 wo packed format Currently we have int4wo + 2:4 sparse (MARLIN) for decode, but we see higher speedups on prefill workloads with the int8 cuSPARSELt / CUTLASS kernels (https://github.com/pytorch/ao/pull/1140). However, the weights are packed in a different format for MARLIN vs cuSPARSELt / CUTLASS, so currently we keep 2 copies of the weights, one for prefill and one for decode. If we can write either a way to translate between packed formats efficiently or a CUTLASS int8int4GEMM that uses the marlin format, we can use the same weights and get max speedups for prefill and decode.


No module named 'model'

Hi,
I tried to run CenterNet-master but I get the error 'ModuleNotFoundError: No module named 'model'' while running demo.py file.
This issue happens in /CenterNet-master/lib/roi_layers/nms.py, line 3 "from model import _C".
I can't find a solution to this problem.

I solved the mentioned problem but yet I get another one :
ImportError: cannot import name 'C' from 'lib.model' (E:\Object-Detection-for-Graphical-User-Interface\CenterNet-master\lib\model_init.py)
How can I solve this one?


add a no-op optimization

Closes #201

@kellyselden I like this, but can you add a quick unit test for that hook, ensuring that this works as expected (and continues to into the future)

@stefanpenner added a test

@rwjblue That's essentially what my test is. What would you like changed?

released as v6.12.0 🎉


Move Save folder Into Assets

The save folder for world data was originally saved outside of the asset folder to prevent it from being targeted by the asset database. However, it would be better to save the worlds inside the asset folder, (as to be more uniform with the standard Unity workflow) while using ~ in the folder name. The Unity Asset Database ignores folders with names ending in ~, so there's no need to worry about "reimporting" the files when the world saves or generating metafiles.

In order to copy the world data to the build client, it would also be a good idea to place the folder inside the StreamingAssets folder in order to have them copied to the BuildData folder when the project is built.
A runtime script can then be written to copy world data from the streaming asset path to the persistant data path if the world is edited and saved in the build.


On chain asset management

@@ -1,4 +1,4 @@
On Chain Asset Management on SUI
On Chain Asset Management on Sui
This implementation is based on Transaction Request - Transaction Approval agreement. Anyone with the Spender role can initiate a transaction by creating a Transaction Request and sharing it on-chain, if the thresholds are met. After a transaction initiation, anyone with the Approver role can review the request and approve it or reject it. By approving a request, a Transaction Approval is created and sent to the spender who uses this approval to execute the transaction.
Set up your asset management application
First step, you have to publish the smart contract:
  sui client publish --gas-budget 30000

With the publication, an administrator capability is created and transfered to the publisher, and a  foundation_balance and roles registry is created and shared. It's a good practice to save those addresses:
  package = "<ID>"
  admincap = "<ID>"
  balance = "<ID>"
  registry = "<ID>"

Next step, administrator has to assign roles to foundation members. Create_spender function transfers to someone the spender capability and specifies the policy thresholds. Create_approver function transfers to someone the approver capability:
  sui client call --package $package --module policy_config --function create_spender --args $admincap $registry <Spender_Address> 10000 10 --gas-budget 30000
  spendercap = "<ID>"
  sui client call --package $package --module policy_config --function create_approver --args $admincap $registry <Approver_Address> --gas-budget 30000
  approvercap = "<ID>"

Last step, top up your foundation balance:
  sui client call --package $package --module policy_config --function top_up --args $balance <Coin_ID> --gas-budget 30000

Transaction Flow

this work is not ready for a PR,
What exactly is the purpose of this @yadavpagare  ??


Menu songs based on date

What is your question?
I'd like to make an addition to my mod so when it's a certain time of the year, a different menu theme will play (e.g. Christmas theme, Halloween theme and a normal theme for every other time)
Is this possible without having to edit source code and having to build the game myself?

No

What is your question?
I'd like to make an addition to my mod so when it's a certain time of the year, a different menu theme will play (e.g. Christmas theme, Halloween theme and a normal theme for every other time)

it might but we ain’t helping you with ur 'oh this website will help me to get people to code complex things for me' bullshit

What is your question?

I'd like to make an addition to my mod so when it's a certain time of the year, a different menu theme will play (e.g. Christmas theme, Halloween theme and a normal theme for every other time)

it might but we ain’t helping you with ur 'oh this website will help me to get people to code complex things for me' bullshit

it isn't complex at all actually, stop bashing on him.
if (date.getMonth() == 11 && date.getDate() == 1)
{
FlxG.sound.music.play(Paths.music("freakyHolidayMusic"));
}
stop being a jerk to people which code isn't complicated.

he wanted it in lua your code is haxe
thats why kitboi said if it works on lua then it is probably  complicated

What is your question?

I'd like to make an addition to my mod so when it's a certain time of the year, a different menu theme will play (e.g. Christmas theme, Halloween theme and a normal theme for every other time)

it might but we ain’t helping you with ur 'oh this website will help me to get people to code complex things for me' bullshit

it isn't complex at all actually, stop bashing on him.
if (date.getMonth() == 11 && date.getDate() == 1)
{
FlxG.sound.music.play(Paths.music("freakyHolidayMusic"));
}
stop being a jerk to people which code isn't complicated.

he wanted it in lua your code is haxe
thats why kitboi said if it works on lua then it is probably complicated

runHaxe() is a command in funkin lua that lets you run haxe code

FunkinLua doesn't run outside of PlayState


Flutter build iOS failed.

Xcode's output:
↳
```
Undefined symbols for architecture arm64:
"OBJC_CLASS$_GeneratedPluginRegistrant", referenced from:
objc-class-ref in AppDelegate.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see
invocation)
note: Using new build system
note: Planning build
note: Constructing build description

I have done all the steps that need to solve it.
I have used these references
https://github.com/flutter/flutter/issues/23556#issuecomment-433817284
https://github.com/flutter/flutter/issues/56567

My Pubspec.YAML

name: ************
description: A new Flutter project.

version: 1.0.0+1
environment:
sdk: ">=2.7.0 <3.0.0"
dependencies:
flutter:
sdk: flutter
The following adds the Cupertino Icons font to your application.
Use with the CupertinoIcons class for iOS style icons.
cupertino_icons:
shared_preferences:
flutter_spinkit:
provider:
geocoder:
http:
geolocator:
location: ^3.1.0
google_fonts:
hexcolor:
fluttertoast:
firebase_database: ^4.4.0
url_launcher:
google_maps_flutter:
google_maps_webservice:
flutter_polyline_points:
image_picker_modern: ^0.4.12+3
firebase_storage:
material_design_icons_flutter:
firebase_auth: ^0.18.4+1
flutter_local_notifications:
latlong:
google_directions_api: ^0.5.0
lottie:
flutter_rating_bar: ^3.0.1+1
country_code_picker: ^1.6.2
search_map_place: ^0.3.0
dev_dependencies:
flutter_test:
sdk: flutter
flutter_launcher_icons:
flutter_icons:
android: true
ios: true
image_path: "assets/Untitled-1-01-01.png"
adaptive_icon_foreground: "assets/Untitled-1-01-01.png"
flutter:
assets:
- assets/
fonts:
- family: Raleway
fonts:
- asset: assets/Raleway-Bold.ttf
weight: 800
- asset: assets/Raleway-Light.ttf

@deveshgarg-max
What Flutter version are you on ?
Can you switch to latest master and try again to see if you get same error ?
flutter channel master
flutter upgrade

Flutter version used by me 1.22.4 in stable, not in the master.
But as used different references from GitHub issues as I mentioned above I've convert my channel to master and 2 hours ago and by just doing that I received a new error of Firebase_core according to Firebase_core error
Firebase_core in ~\flutter.pub-cache\hosted\pub.dartlang.org\firebase_core-0.5.3\ios\Classes in FLTFirebaseCorePlugin.h #import <Flutter/Flutter.h> not found
such type of error is giving.
Note: Using simulator to FLutter build ios
I am really stuck in this.

@deveshgarg-max
Try below commands and see if they help:
cd ios
pod deintegrate
pod install

I am so sorry but I am also doing this for the last few hours still getting the same error.
Do we have to do any changes in Xcode at the project level?

as i followed this step https://github.com/flutter/flutter/issues/72057#issuecomment-742507567
I found this new issue
Xcode's output:
↳
In file included from
/Users/ekantpuri/desktop/flutter/.pub-cache/hosted/pub.dartlang.org/firebase
_core-0.5.3/ios/Classes/FLTFirebaseCorePlugin.m:5:
/Users/ekantpuri/desktop/flutter/.pub-cache/hosted/pub.dartlang.org/firebase
_core-0.5.3/ios/Classes/FLTFirebaseCorePlugin.h:8:9: fatal error:
'Flutter/Flutter.h' file not found
#import <Flutter/Flutter.h>
^~~~~~~~~~~~~~~~~~~
1 error generated.
note: Using new build system
note: Planning build
note: Constructing build description

This worked for me:

Update CocoaPod if it is not updated
Make flutter channel to Master
flutter upgrade
flutter pub get
cd ios
pod deintegrate
pod install
if you get such type of error https://github.com/flutter/flutter/issues/72057#issuecomment-743065351 create new blank project and paste all essential files of the previous project into the new project.
And boom you did it!

Note: Flutter latest version has some expired arguments so accordingly, please go to the paths and remove or comment on the code which is giving an error.


java.beans.Introspector chokes on Gradle DSL object properties in init scripts

The java.beans.Introspector is choking on the org.gradle.api.invocation.Gradle DSL object when Introspecting for dynamic object property access in init scripts.
In particular:

The java.beans.Introspector is miscategorizing methods like Gradle.settingsEvaluated() as a bean property named tingsEvaluated.

BeanDynamicObject.getProperties() is not ignoring bean properties where MultipleSetterProperty.getGetter() == null.

accessing properties from an init script is throwing:
"java.lang.IllegalStateException: The default project is not yet available for build."

E.g: with init script:
properties.forEach { name, value ->
    println "property: ${name}=${value}"
}

Expected Behavior
I should be able to access properties in the Gradle object context provided to the init script
Current Behavior

The java.beans.Introspector is miscategorizing methods like Gradle.settingsEvaluated() as a bean property named tingsEvaluated

Fails on misguided attempt to call MultipleSetterProperty.getGetter() for the Gradle.settingsEvaluated() functions

Throws IllegalStateException

Context
I'm working on putting together some custom Gradle distributions targeting Gradle 7+ (like what's described here: https://docs.gradle.org/current/userguide/organizing_gradle_projects.html#sec:custom_gradle_distribution ).
For this custom Gradle distribution, I'm trying to augment the static default property values which I have configured via a gradle.properties file and included in the root of my distribution with some dynamically generated "default" property values that I configure via an init.d/ci.init.gradle script which I'm also including in the distribution.
This has not been an easy task. The init scripts are not doing basic things that I've grown to expect from working in the settings.gradle and build.gradle scripts. It feels like everywhere I turn I need to dig through the Gradle source files to figure out another "work-around" to get things going again.
I would like this experience to be better, so I'll be filing 5 tickets with hope that we can to improve this situation.
This is ticket 1 of 5
Steps to Reproduce
using file: ci.init.gradle
properties.forEach { name, value ->
    println "property: ${name}=${value}"
}

run:
./gradlew help --init-script ci.init.gradle --stacktrace

see failures like:
java.lang.IllegalStateException: The root project is not yet available for build.

Your Environment
Gradle 7.2

Accessing Gradle.properties without being in a callback at all is too early indeed.
Given the question here and in #13362, we are trying to clarify this in the documentation.
There is however no plan at this time to change when properties are loaded in the build lifecycle.

Regardless of timing, I expect the BeanDynamicObject.getProperties() function should not throw exceptions when being called on the Gradle object from inside of an init script.
Please, see the PR I've provided ^^^ for my suggested fixes.

The PR linked gives access to properties. The discussion within team concluded that we do not want to give access. And so the failure is OK in that context.


Is there a way to exclude invalid object from a list of objects with specific type?

Question | Feature Request
Pre-requirements:

OS: Mac OS X 10.14.3
Python version import sys; print(sys.version): 3.6.3
Pydantic version import pydantic; print(pydantic.VERSION): 0.20

Hi, first of all thank you for the great package!
The questions is: if I have a nested list of objects how can I remove invalid object keeping only valid ones during a validation process?
Here is an example:
from pydantic import BaseModel, conint
from typing import List

class Apple(BaseModel):
    weight: conint(gt=10)
    color: str

class BackPack(BaseModel):
    apples: List[Apple]

data = {
    'apples': [
        dict(weight=13, color='red'),
        dict(weight=21, color='green'),
        dict(weight=4, color='red'),
    ]
}

print(BackPack.parse_obj(data))

There is one "invalid" apple with a weight less then 10, so after validation is completed I want it to be removed from the list but keep the good ones. The only way I was able to find is to do a "manual" parsing of data before passing it to BackPack contructor:
apples = []
for apple in data['apples']:
    try:
        apples.append(Apple.parse_obj(apple))
    except pydantic.ValidationError:
        # logging
        pass

BackPack(apples=apples)

But this solution is not elegant and clean, is there a better one?
Thank you in advance.

First of all, thanks for taking the time to format and explain your question clearly, this makes such a difference and made me far more willing to spend some time answering.
The short answer, unfortunately, is "not easily".
You can use validators, but you need pre=True and whole=True to get access to the raw data, otherwise, the exception has already been raised. Unfortunately, that means the data you have to process can be anything.
Here's one way of doing it, I'm not convinced it's that much better than your current approach:
from pydantic import BaseModel, conint, validator, Decimal
from typing import List

class Apple(BaseModel):
    weight: conint(gt=10)
    color: str

class BackPack(BaseModel):
    apples: List[Apple]

    @validator('apples', pre=True, whole=True)
    def remove_bad_apples(cls, apples):
        if not isinstance(apples, (list, tuple)):
            # if we get here the apple doesn't look valid,
            # we return the original object and let normal pydantic validation
            # do it's stuff
            return apples

        good_apples = []
        for apple in apples:
            if isinstance(apple, dict):
                weight = apple.get('weight')
                if weight and isinstance(weight, (int, float, Decimal)):
                    if weight > 10:
                        good_apples.append(apple)
                    continue
            # if we get here the apple doesn't look valid,
            # we return the original object and let normal pydantic validation
            # do it's stuff
            return apples
        return good_apples

data = {
    'apples': [
        dict(weight=13, color='red'),
        dict(weight=21, color='green'),
        dict(weight=4, color='red'),
    ]
}

print(BackPack.parse_obj(data))

On reflection, here's a much cleaner way of doing things.
from pydantic import BaseModel, validator
from typing import List

class Apple(BaseModel):
    weight: int
    color: str

class BackPack(BaseModel):
    apples: List[Apple]

    @validator('apples', whole=True)
    def remove_bad_apples(cls, apples):
        return [a for a in apples if a.weight > 10]

data = {
    'apples': [
        dict(weight=13, color='red'),
        dict(weight=21, color='green'),
        dict(weight=4, color='red'),
    ]
}

print(BackPack.parse_obj(data))

Unfortunately because you've removed conint, the schema now won't include that restriction. You could probably get around that using the Schema type if you care about it.

Oh... In reality of course "Apple" model contains a lot of different fields (some of them nested) so your proposal is hard to implement for such complex objects.
The improved solution I came up with may look like:
class BackPack(BaseModel):
    apples: List[Apple]

    @validator('apples', pre=True, whole=True)
    def remove_bad_apples(cls, apples):
        filtered_apples = []

        for apple in apples:
            value, error = validate_model(Apple, apple, raise_exc=False)
            if not error:
                filtered_apples.append(value)

        return filtered_apples

However in this case apples validation is done twice thus performance can be a bit degraded but not critical.
I'm thinking about a sort of processor parameter for a validator, which allows to return already validated pydantic models (instead of raw values) from a validator. Or even it can be a separate method, e.g.:
class BackPack(BaseModel):
    apples: List[Apple]

    @processor('apples', pre=True, whole=True)
    def remove_bad_apples(cls, apples):
        filtered_apples = []

        for apple in apples:
            try:
                filtered_apples.append(Apple.parse_obj(apple))
            except pydantic.ValidationError:
                # logging
                continue

        return filtered_apples

Does it make a sense?

I think as of v0.20 the validation will not be run twice for each model, see here.


Formatting issues when using default UI Theme

Foundry version 11.315
Star Wars FFG 1.806
When using the default UI theme with the Actor Sheet v1, it cuts off the bottom row of skills within the character sheet (it does not affect the minion sheet). This means you cannot scroll down to view the Brawl or Xenology skills:

It displays the skills correctly when using Actor Sheet v2:

It displays the skills correctly when using the Mandar Theme.
Also when using the default Theme, it is cutting off the right hand edge of the Destiny Point background image:

I am also having this issue. Although for note it does happen with V2 but it isn't as bad, the problem is that it's a little inconsistent:

It might be that a tiny bit of 'dead space' needs to be added to the bottom so when it scrolls all the way it scrolls far enough?

I also have the same problem when usisng Foundry, but when using Firefox browser problem do not occur


Updated shvdn3 support

Just a heads up.
I ported the latest version of NativeUI to ScriptHookV.NET 3 before I realized that there already was a branch for it here.
You can find it here: https://github.com/udoprog/NativeUI/tree/shvdn3
Since this was not based on the existing branch it turned out to be too hard to reconcile the changes, but at least it's up-to-date with the existing master.
Related: crosire/scripthookvdotnet#869

Do you have problem if I push that to this repo?

No problem. That works for me. Thanks <3

Also, regarding crosire/scripthookvdotnet#869. I feel that is not possible to create releases for SHVDN3 because of compatibility issues (v2 vs v3 on the loader) and the non existence of SHVDN3 NuGet packages.

Yeah, I agree. This is on the SHVDN project for now.

The branch shvdn3 has been updated.


Unknown argument  "userId" on field "feeds" of type "RootQuery".

I am getting this  error sporadically with my angular apollo client. I did verify the userId and it is passed in correctly. i copied the same query with graphiql and it works every time.
when i get this error the request doesn't even make it to the  server, so i guess this something wrong with the angular UI. Moreover i am using apollo with ngrx.  Appreciate any help
core.es5.js:1084 ERROR Error: GraphQL error: Unknown argument "userId" on field "feeds" of type "RootQuery".
    at new ApolloError (http://localhost:4201/vendor.bundle.js:71141:28)
    at http://localhost:4201/vendor.bundle.js:138673:41
    at http://localhost:4201/vendor.bundle.js:139108:25
    at Array.forEach (native)
    at http://localhost:4201/vendor.bundle.js:139105:27
    at Array.forEach (native)
    at QueryManager.broadcastQueries (http://localhost:4201/vendor.bundle.js:139102:42)
    at Array.<anonymous> (http://localhost:4201/vendor.bundle.js:138472:23)
    at Object.dispatch (http://localhost:4201/vendor.bundle.js:79113:19)
    at dispatch (<anonymous>:2:31875)

Please click this link to check the issue discussed in detail
https://apollographql.slack.com/files/bhaskar/F5N856Z51/screen_shot_2017-06-03_at_9.20.05_pm.png
https://apollographql.slack.com/archives/C0Z9ZBDF0/p1496545444511183
https://apollographql.slack.com/files/bhaskar/F5P5F459D/screen_shot_2017-06-03_at_11.28.54_pm.png

found the issue. it is my bad. i was running another instance with the old source and was load balancing the instances with rabbitmq. some requests are sent to the this other instance.


[Q] Using WandbLogger in DDP, Multi-GPU with Pytorch-Lightning 2.0.0 before Trainer.fit(...) is called.

I've posted in the PL discussion section, posting here as well: https://github.com/Lightning-AI/lightning/discussions/17225

Hi @angadkalra, thanks for writing in! Here you can have a look at our docs about DDP using multiple GPUs, would they be useful for you?

Hi @angadkalra, following up here! Did you have the chance to have a look at our docs and were they useful for you?

Hi, I found a work around where I log everything within a LightningModule after Trainer.fit(..) is called. The problem was occuring when trying to log before Trainer.fit(..) is called using the pytorch-lightning WandbLogger(...) object. The docs don't help with that issue. Do you have an example of doing that?  For example:
logger: WandbLogger = WandbLogger(
        project=project_name,
        name=run_name,
        log_model=True,
        id=run_id
    )

    print(f'\nProject Name: {project_name}')
    print(f'Run Name: {run_name}')

    # Log gin config file
    logger.experiment.log_artifact(args.gin_config, name='gin-config', type='config')

    # Setup datamodule for training
    datamodule = setup_datamodule(logger=logger)

    # Log train/val data files
    logger.experiment.log_artifact(datamodule.train_csv, name='train-csv', type='dataset')
    logger.experiment.log_artifact(datamodule.valid_csv, name='valid-csv', type='dataset')

    # Log batch_size: THIS BREAKS WHEN USING DDP WITH PL TRAINER AND 4 PROCESSES
    logger.experiment.config['batch_size'] = datamodule.batch_size

Hi @angadkalra, sorry about the delay here. You can start your run in the normal way and then log your data using wandb.log before calling Trainer.fit() like:
run = wandb.init(project=project_name, name=experiment_name)
run.log(...)
logger = WandbLogger(experiment_name, project=project_name, dir=experiment_name)

# Create a PyTorch Lightning Trainer with the WandbLogger
trainer = pl.Trainer(max_epochs=10, logger=logger)

# Train the model
model = ...
trainer.fit(model, dataloader)

Would this work for you?

So are you saying it's not possible to use the WandBLogger available through PyTorch-Lightning library to log metrics and artifacts BEFORE trainer.fit(...) is called in a multi-gpu DDP setting? You're saying to use wandb library directly instead?

Hey @angadkalra, sorry about the delay here. I tested with a code like the following and it works properly. Could you please check if it works on your end?
import os
import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader, Dataset
import pytorch_lightning as pl
from pytorch_lightning.loggers import WandbLogger
import wandb
import numpy as np

# Dataset
class SmallDataset(Dataset):
    def __init__(self):
        self.x = torch.randn(100, 1)
        self.y = self.x * 2 + 0.5 * torch.randn(100, 1)

    def __len__(self):
        return len(self.x)

    def __getitem__(self, idx):
        return self.x[idx], self.y[idx]

# Model
class SimpleModel(pl.LightningModule):
    def __init__(self):
        super(SimpleModel, self).__init__()
        self.layer = nn.Linear(1, 1)

    def forward(self, x):
        return self.layer(x)

    def training_step(self, batch, batch_idx):
        x, y = batch
        y_pred = self(x)
        loss = nn.MSELoss()(y_pred, y)
        self.log("train_loss", loss)
        return loss

    def configure_optimizers(self):
        return optim.SGD(self.parameters(), lr=0.01)

# Training
def main():

    # Create the dataset and dataloaders
    dataset = SmallDataset()
    train_loader = DataLoader(dataset, batch_size=10, num_workers=4)

    # Create the model
    model = SimpleModel()

    # Create the wandb logger
    wandb_logger = WandbLogger(project='...', entity='...', log_model='all')

    # Create the PyTorch Lightning trainer with DDP
    trainer = pl.Trainer(
        max_epochs=20,
        devices=4,
        accelerator="cpu",
        strategy="ddp",
        logger=wandb_logger,
    )

    # Log random image
    wandb_logger.log_image(key="samples", images=[wandb.Image(np.random.randint(0, 255, (32, 32, 3), dtype=np.uint8))])

    # Train the model
    trainer.fit(model, train_loader)

if __name__ == "__main__":
    main()

Luis Bergua commented:
Hi there, I wanted to follow up on this request. Please let us know if we can be of further assistance or if your issue has been resolved.

@luisbergua Can you switch from cpu to gpu within the Trainer and before calling Trainer, update the config file with a random value like this: logger.experiment.config['batch_size'] = datamodule.batch_size

Hi @angadkalra, thanks for your answer! Changing that finally allowed to reproduce the issue (looks like the problem is isolated to updating the config but not logging the image) so I'll report this to our engineering team. Thanks a lot for reporting the issue!

Yes it seems like logging everything else (artifacts, images, etc) is fine, but updating the config breaks. Thanks for keeping up with this! Please let me know when the library is updated with a fix!

Hey @angadkalra, Thanks so much for the additional information here -- I will ensure this is conveyed to the relevant engineering team. Please let us know if there is anything further that we can help with just now.

Has this issue been resolved in the latest WandB release?

Hi @angadkalra, thanks for following up! This issue hasn't been fixed yet but let me bump the priority of it with our Engineering Team. We'll keep you posted on the status of it.

WandB Internal User commented:
fdsig commented:
Hey @angadkalra, Thanks so much for the additional information here -- I will ensure this is conveyed to the relevant engineering team.  Please let us know if there is anything further that we can help with just now.


Feature: Code push and pull for provider integrations

Is your feature request related to a problem? Please describe.
People are used to using git and managing code, but there are those that want to just have code moved from one place to another, with or without git.
Add integration to the provider interface to allow push and pull of code.

This is no longer a priority.


Webhook to auto-reload the wikis when changes are pushed

The idea is to have the webserver autoreload whenever the github repo is updated
Approach:

define a webhook on github settings for a specific repo
the webhook settings in github contains :
--url: which we will set to https://wiki.grid.tf/github/Account/Repo]
--password: for validating the hook request

Once a new commit happens on the repo, it will make a request to the url which is already defined in lapis as a generic route.
Lapis will check the secret and validate the request, then either it will pull the repo or will forward it to a gedis actor which will do the pull using our git client

https://github.com/threefoldfoundation/lapis-wiki/commit/25527bd227142f1b9eb67e5e3952b1ce16e91fe8
https://github.com/threefoldtech/digitalmeX/commit/4c7282a6d6f243d1d17b57457daac86cb7bd11ee


Ignore nodes where drain is not possible

A node may be blocked from being drained when a PDB stops a Pod from being moved. This should be avoided in most cases but mistakes can happen. Currently Node TTL will just get stuck with that node. A better solution would be to ignore the nodes which will never be drained.

Hi! Note that the Rook operator handles PDBs dynamically, so that OSD pods (disks) aren't evicted faster than the storage cluster can handle.

Ah thats an interesting point did not know about this. So my understanding then is that it has PDB which prohibits any eviction and then modifies the PDB when it feels like it can handle the eviction? So would all nodes with OSD pods be stuck until Rook jumps in and modifies the PDB?
My plan was to implement a dry run on the drain which evaluates is a node would get stuck in the first place. Do you think this would work with Rook?

So would all nodes with OSD pods be stuck until Rook jumps in and modifies the PDB?

Yes, that's right!

My plan was to implement a dry run on the drain which evaluates is a node would get stuck in the first place. Do you think this would work with Rook?

I don't know about this actually, and I am unable to test it since I no longer use Rook
I would suggest implementing a labelSelector for this feature so that Rook nodes still can be drained (effectively ignoring this feature), while respecting others.


Error for devices with local keys containing special characters

Since tuya dev began using special characters in the device local keys, I have not been able to register these devices in tuya local
Eg of local key for device that was working before
n:Gkz]^d]U_6F~of

There is no issue with "special characters".  This keeps being reported repeatedly, and every time users find some other issue in their config, like using the uuid instead of the device id, not entering the IP address, or the device is unable to auto detect protocol version so it has to be explicitly set.


Add List of available VLANs for a datacenter

@Praveengostu I addressed your comments, please let me know if there is anything else.  Thanks


native scrollbar shows when zooming out

Hi, this looks awesome. I'm always looking for a good implementation of custom scrollbars. I did a test with zooming out and unfortunately, the native scrollbar shows up behind the optiscroll. If I refresh the page (zoomed out), it works fine and hides the native scrollbar. Do you think this is something that can be recalculated/fixed on some sort of resize or zoom out event?

Well, quite tricky "bug" I suppose. Never thought about that behaviour before. As there is no reliable way to detect the zoom value, the only way I can think of handling the issue is by using units that are immutable, like vw, updated on resize.
So, given the scrollbar size in px, if divided by the window clientWidth you get the vw dimension. Then you update Optiscroll container styles, changing right and bottom values to be vw.
And this needs to be done after every resize, otherwise scrollbars will appear again.

Closing this as I don't see any actionable task.


Add Bun and Deno to create-astro guide

What kind of changes does this PR include?

New or updated content

Description

What does this PR change? Give us a brief description.
Did you change something visual? A before/after screenshot can be helpful.

Added Bun and Deno to getting started. Intentionally added after in the end of the list, as newcomers. See pictures.

Related Issue / Implementation PR

Hi @birkskyum ! This is just to let you know that we're discussing this internally, because there's been a lot of bun activity recently, but it's still unsupported by us and so we are hesitant to document it. Given that we cannot offer any support with issues using bun, we are leaning against including this in our own documentation. If we do, it will only be to give us a clear place to say, "we're not responsible, file any issues with bun directly" and that's not exactly what we like to have in our own docs, especially on our Getting Started page! :sweat_smile:
So this is just to let you know where this stands right now! We'll be discussing more and eventually deciding whether this is something for our own documentation. Thank you for making this contribution, as we really do appreciate the community jumping in and chipping in! This just might not be what we decide to go with. :raised_hands:

OK, our official stance on this right now is not to update and show more package managers at this time.
So sorry, I'll be closing this PR @birkskyum, but I do appreciate that you took the time to contribute, and I hope you'll be back when you spot a typo or see an Issue you feel like fixing!  🙌

Okay. Guess i just saw it a bit different. To me this wasn't really about package managers, but more about runtimes. The are guides for 3x Node workflows, and no bun or Deno.


Improvement on Tags in js

Removed lineWidth rule as it didn't improve on anything
Added the yaml header to all content w/ jsyaml.dump() since it wasn't automatically added
Changed getTags method for individual EN and FR functions for less duplication of code & moved from a vanilla js query to a jQuery one.

Plz approve & merge


British English [en-GB]

I decided to do a specific British English variant. Currently only find Colour spelled differently. Up for anyone else to submit more corrections.

en-gb usually has subtle changes in phrasing too to match words and phrases that aren't used as frequently as in the US.
Depending on your changes for en-gb it may not be worth it though.

Just saw #32
Based on the larger amount of references to Color i think it makes sense :)

I went through this with the idea of changing any required phrasing but to be honest, there wasn't any at this point, only some more US/GB spelling switches. I assume some are likely to crop up eventually though (I was expecting "Trash -> Bin" swaps, but nothing like that yet).

This locale is kind of cluttering up the MISSING.md and will be as long as only GB-specific keys are in there. I think either en-GB should be excluded from the Missing Keys list, or strings that are confirmed to match British English should just be copied over.

Don't copy the strings over. I think I'll just split up the MISSING.md and generate one for each language in a subfolder instead, that way it'll scale a lot better as we get more languages too.

That might clean things up a bit but the file for en-gb still wouldn't be of much use if you can't tell which keys have and haven't been reviewed.

That's going to be a problem for any sub-languages like this, I don't plan on making specific exclusions for each of them. You can just ignore the file in those cases.
We could add a way to mark keys as reviewed, but that's extra implementation time, so I'm not sure when I'd do that.


no header found error on Scenario #3

Hello,
I got " no header found " error when I was testing the "Create a LAMP stack on an Azure Linux VM" scenario.
After the error it was like continuing with the next steps but actually it hung and did not create the resources. Uploaded 2 screenshots.

Thanks

This is a known issue and should be resolved in an upcoming portal deployment.

This issue is fixed pending a deployment

This has been deployed.


Require password on account e-mail address change

Is your feature request related to a problem? Please describe.
As a user, when I am logged into my account on a device that I trust and I would like to change my email associated with my account. I will be required to enter my current correct account password in order to confirm the change of email process.
Makes it harder for bad people to take over the account by e.g. session hijack.
Is your feature request related to an existing functionality? Please describe.
Not existing functionality but existing section in the /account area
Describe the solution you'd like
A solution that includes the logged in users current correct account password on request of email change would also include error pathways for users including:

If a user enters their password incorrectly
A user enters an invalid email as a new email (could also include an email that does not have accepted organisational affiliation in the email address)
If a user spends to long on the page session and 'times out'
Compatible with any existing 2-factor processes on that account and instance

Describe alternatives you've considered

Alternative to password entry on email change might be using any current 2-factor process.

Additional context
This was part of the recommendations given by the security audit team circa Aug 2021.
Priority

Low priority (annoying, would be nice to see)

Affected roles

Staff
Applicants
Reviewers
Partners

Ideal deadline
End of Q1 2022

This works! 100 points to Gryffindor! 🌈


How to compile for desktop GPUs ?

Are there any CMakesFiles for compiling dusty interference for desktops GPUs because with my 1080 Ti i am getting invalid device function something like that:
[cuda]   cudaGetLastError()
[cuda]      invalid device function (error 8) (hex 0x08)
[cuda]      /home/ddk/ddk-repo/jetson-inference/imageNet.cu:50
[cuda]   cudaPreImageNet((float4*)rgba, width, height, mInputCUDA, mWidth, mHeight, make_float3(104.0069879317889f,<PHONE_NUMBER>1696767f,<PHONE_NUMBER>406786f))
[cuda]      invalid device function (error 8) (hex 0x08)
[cuda]      /home/ddk/ddk-repo/jetson-inference/imageNet.cpp:146
Any guess how to adapt the CMakeFile ?
Best regards.

Hi @ZeroCool11000, you need to add your GPU's compute capability to the CMakeLists file at this line here:   https://github.com/dusty-nv/jetson-inference/blob/e12e6e64365fed83e255800382e593bf7e1b1b1a/CMakeLists.txt#L37


Add Visualizing Weights notebook

This adds a notebook which we would like to reference in an upcoming Distill article, Visualizing Weights: link.

Notes to self – TODO before shipping:

Find and replace colorr with color (typo noted by distillpub.slack feedback)


Header's backgroundColor and margin/padding

Header enhancements:

Enable setting padding: 0 - there is built in padding that user cannot control.
Enable setting backgroundColor when user scrolls . Currently a color is auto-set depending on seed color and user cannot change it.

@sibteali786 can you please add me as collaborator
<EMAIL_ADDRESS>

@snehmehta sent you invitation

Thanks! @sibteali786


Actually use the subtest names in recorder test.

The test names have been unused before. We can also reuse the cases type instead of redefining it inline.

/assign @vagababov

/assign @mdemirhan

/approve

/retest


[WIP] Prototyping adding GCS support

Restoring #14
Description
Still very much a WIP, but this is a prototype which adds GCS support via admin.storage(). admin.storage().bucket exports the standard GCS Bucket API.
Initialization of the underlying google-cloud SDK is probably going to be a bit tricky since they don't provide a standard credential interface like our SDK does which allows you to provide your own Google OAuth2 access tokens. But we can make it work without too much effort via  the certificate credential (which this PR does) and via the Application Default Credential (which this PR does not do). I'm still not sure how we will make it work using a refresh token credential since we don't have the right fields that the google-cloud SDK provides as initialization options (see the table in "Advanced Usage" here).
I didn't do anything for multi-bucket support although it shouldn't be too bad to support.
cc/ @mcdonamp, @sphippen, @hiranya911, @depoll
Code sample
var admin = require('firebase-admin');

var serviceAccount = require('./path/to/key.json');

admin.initializeApp({
  credential: admin.credential.cert(serviceAccount),
  storageBucket: 'admin-sdks-test.appspot.com',
});

admin.storage().bucket.file('firebase.png').download({
  destination: './firebase.png'
}, function(error) {
  if (error) {
    console.log('Error downloading file:', error);
  } else {
    console.log('File successfully downloaded!');
  }
});

Closing this in favor of #47


Custom directive with more than one parameter

I'm struggling with creating a custom directive like this one: @my_svg('/assets/Wortmarke.svg', 'bg-blue')
The parameters seem get passed as one string in the config.php. Is that expected behaviour? I was assuming that I will have a function ($path, $class){ ... } for making use of them...
Anyone else having trouble with multiple parameters? Also it seems that I need to clear the cache in storage/cache to have updates (custom directive) to my config.php reliably reflected when reloading.

Yes, you are right. You can only pass one parameter to a custom directive. $expression is a string and contains both parameters.
Laravel Docs: https://laravel.com/docs/10.x/blade#extending-blade
Maybe this thread from laracasts helps you: https://laracasts.com/discuss/channels/laravel/how-to-do-this-blade-directive
Yes, you have to clear the cache after updating a directive. From the Laravel docs:

After updating the logic of a Blade directive, you will need to delete all of the cached Blade views. The cached Blade views may be removed using the view:clear Artisan command.


feat: only load Instagram script when needed

Just like gatsby-plugin-twitter does.
https://github.com/gatsbyjs/gatsby/blob/a9704e905746f2369af19b71ad6014fef6119233/packages/gatsby-plugin-twitter/src/gatsby-browser.js#L40-L53

:tada: This PR is included in version 1.1.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Add Aliases

People can look up for signatures moves like JFSR or EWGF or demoman faster

I like the idea of the alias functionality, but your pull request also has commits that would change current functionality. Also the framedata files are overwritten when there's changes to framedata, so the aliases wouldn't persist in the framedata files.

Aye. I'm sorry for this chaos because I'm pretty new to programming.
Do you mean by commits "that change current functionality", the one on mokujin.py?
What happens when the frame data changes? Can't you just merge the old .json file and the new one so the aliases don't get deleted?


Shared Workflows

Description

Created RockLib.Workflows repo
Added reusable workflows to call from RockLib library pipelines
Added generic workflows to call reusable workflows for use with each RockLib library

Tested that calling workflows from a library repo works

Type of change: 3

Non-functional change (e.g. documentation changes, removing unused using directives, renaming local variables, etc)
Bug fix (non-breaking change that fixes an issue)
New feature (non-breaking change that adds functionality)
Breaking change (fix or feature that could cause existing functionality to not work as expected)

Reviewer guidelines

Whats the difference between the workflows in .github vs generic?

Whats the difference between the workflows in .github vs generic?

The workflows in .github are the shared workflows.  The workflows in generic are the ones to be copied into each library repo that call the shared workflows located in .github in this repo.  The allows us to have a singe source to maintain to update the workflows for the libraries.


Modifications for zcombufs and zcombufr now both being allocated at t…

…he top level. Not complete: need to figure out how to compute kvset.

Merge conflicts. I will manually merge dmitry-overlap-changes into samhatfield/overlap and open a separate PR.


support 2pc transaction

support 2pc transaction

Codecov Report

Merging #59 into dev will increase coverage by 0.04%.
The diff coverage is 34.17%.

@@            Coverage Diff             @@
##               dev     #59      +/-   ##
==========================================
+ Coverage     4.00%   4.04%   +0.04%
- Complexity      17      21       +4
==========================================
  Files           21      21
  Lines         1425    1460      +35
  Branches       165     174       +9
==========================================
+ Hits            57      59       +2
- Misses        1355    1385      +30
- Partials        13      16       +3

Impacted Files
Coverage Δ
Complexity Δ

...va/com/webank/wecross/console/common/HelpInfo.java
0.00% <0.00%> (ø)
0.00 <0.00> (ø)

...va/com/webank/wecross/console/common/ZipUtils.java
0.00% <ø> (ø)
0.00 <0.00> (ø)

...com/webank/wecross/console/custom/BCOSCommand.java
0.00% <0.00%> (ø)
0.00 <0.00> (ø)

...om/webank/wecross/console/common/ConsoleUtils.java
34.50% <81.81%> (+0.57%)
21.00 <9.00> (+4.00)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a670e06...32a2d03. Read the comment docs.


ros2cli: 0.25.0-1 in 'rolling/distribution.yaml' [bloom]

Increasing version of package(s) in repository ros2cli to 0.25.0-1:

upstream repository: https://github.com/ros2/ros2cli
release repository: https://github.com/ros2-gbp/ros2cli-release.git
distro file: rolling/distribution.yaml
bloom version: 0.11.2
previous version for package: 0.24.1-1

ros2action
* Make all of the dependencies in pure Python packages exec_depend. (#823 <https://github.com/ros2/ros2cli/issues/823>)
* Contributors: Chris Lalancette

ros2cli

No changes

ros2cli_test_interfaces

No changes

ros2component

No changes

ros2doctor
* Make all of the dependencies in pure Python packages exec_depend. (#823 <https://github.com/ros2/ros2cli/issues/823>)
* Contributors: Chris Lalancette

ros2interface
* Make all of the dependencies in pure Python packages exec_depend. (#823 <https://github.com/ros2/ros2cli/issues/823>)
* Contributors: Chris Lalancette

ros2lifecycle
* Make all of the dependencies in pure Python packages exec_depend. (#823 <https://github.com/ros2/ros2cli/issues/823>)
* Contributors: Chris Lalancette

ros2lifecycle_test_fixtures

No changes

ros2multicast
* Make all of the dependencies in pure Python packages exec_depend. (#823 <https://github.com/ros2/ros2cli/issues/823>)
* Contributors: Chris Lalancette

ros2node
* Make all of the dependencies in pure Python packages exec_depend. (#823 <https://github.com/ros2/ros2cli/issues/823>)
* Contributors: Chris Lalancette

ros2param
* remove deprecated options (#824 <https://github.com/ros2/ros2cli/issues/824>)
* Make all of the dependencies in pure Python packages exec_depend. (#823 <https://github.com/ros2/ros2cli/issues/823>)
* Contributors: Chris Lalancette, Tomoya Fujita

ros2pkg
* Make all of the dependencies in pure Python packages exec_depend. (#823 <https://github.com/ros2/ros2cli/issues/823>)
* Contributors: Chris Lalancette

ros2run
* Make all of the dependencies in pure Python packages exec_depend. (#823 <https://github.com/ros2/ros2cli/issues/823>)
* Contributors: Chris Lalancette

ros2service
* Make all of the dependencies in pure Python packages exec_depend. (#823 <https://github.com/ros2/ros2cli/issues/823>)
* Contributors: Chris Lalancette

ros2topic
* remove deprecated options (#824 <https://github.com/ros2/ros2cli/issues/824>)
* Make all of the dependencies in pure Python packages exec_depend. (#823 <https://github.com/ros2/ros2cli/issues/823>)
* Contributors: Chris Lalancette, Tomoya Fujita

This pull request has been mentioned on ROS Discourse. There might be relevant details there:
https://discourse.ros.org/t/preparing-for-iron-branching-on-rolling-2023-04-18/30980/2


fix: reimplement FileSpanExporter to use async APIs

Also: do not to close the file descriptor on exit unless it's a regular file.
Fixes #167

Merged into #172


[Issue]: VX_AMD_MIGRAPHX - GDF Sample Error

Problem Description
ONNX MNIST failure
Start 3: vx_amd_migraphx_test

3: Test command: /opt/rocm/bin/runvx "-dump-profile" "file" "/home/kiriti/develop/mivisionx-kiriti/MIVisionX/tests/amd_migraphx_tests/gdf/vx_migraphx_test.gdf"
3: Test timeout computed to be: 1500
3: runvx 1.0.0
3: OK: using AMD OpenVX 1.3.0
3: OK: loaded 1 kernels from libvx_amd_migraphx.so
3: OK: loaded 39 kernels from libvx_nn.so
3: OK: OpenVX using GPU device - 0: Radeon RX 7900 XT [gfx1100] with 84 CUs on PCI bus 03:00.0
3: MIGraphX Error: /long_pathname_so_that_rpms_can_package_the_debug_info/src/extlibs/AMDMIGraphX/src/include/migraphx/op/convolution.hpp:100: normalize_compute_shape: CONVOLUTION: mismatched channel numbers
3: terminate called after throwing an instance of 'std::runtime_error'
3:   what():  Failed to call function
1/1 Test #3: vx_amd_migraphx_test .............Subprocess aborted***Exception:   2.52 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) =   2.53 sec

The following tests FAILED:
	  3 - vx_amd_migraphx_test (Subprocess aborted)

Operating System
Ubuntu
CPU
ALL
GPU
AMD Radeon Pro W7900
Other
No response
ROCm Version
ROCm 6.0.0
ROCm Component
MIVisionX
Steps to Reproduce
No response
(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support
No response
Additional Information
No response

@LakshmiKumar23 any updates on this?

@LakshmiKumar23 add fix for issue #1415 here
Needed to run migraphx on Ubuntu22, which needs -lcrypto. QA had some issues with running the tests and this fixed it
sudo apt install libssl-dev


Various improvements for the *Plot views

The main focus on this was making the presentation properties (such as line widths and colors) exposed and inspectable in Interface Builder.
At the same time I also modernized these classes quite a bit, and hid some of the internals by moving the Csound protocol adoption outside of the headers.

You absolutely rock!

Thanks! :) By the way, I was wondering whether it would make sense to make a call to [AKManager addBinding:self] directly from within the constructor in these classes. This would make it unnecessary to do it manually otherwise... And this way all you'd have to do to get a functional display is to connect it in the storyboard.

Yes, I had thought of that recently, while I was upgrading the examples.  Totally the right thing to do.  The other thing I want to do is consolidate a lot of the repeated code over similar plots.  But, I have a bunch of other stuff to do if you want to be plot-master for a while, that would be so appreciated.

I could do that. Actually just looking at the code I think it would make more sense to do the binding in didMoveToSuperview: so that the init is fully done. I don't see an unbind method on AKManager, is that intentional? It would make sense to also remove it once the view is deleted or gone.

That's sounds really great.  I'd probably call it "removeBinding" to parallel the "addBinding" method but it sounds like a very good idea.


JS Port/Maintenance/Feed Sources

Seeing as this is not heavily maintained and my python skills are limited, I am porting the entire thing over to node/JS (about 80% done).
I am using this issue to notify users that it will be published on github soon and also to gather feedback & feature requests to be implemented on my port as well as new Feed sources.
I would also appreciate if people could notify me of sources that no longer work (e.g. I believe AEX api no longer works)

Python is not my main/favorite programming language but I managed to review all the code, test all the sources and formulas,  found and reported some issues and fixed most of them in my repo.
Current status on sources is:

Should be removed: BTC38, BTCChina, ChBTC?, BTer, Yunbi (#40), Yahoo (#41)
Should be rewritten: BitcoinVenezuela (#42), Big.One (#43), BitcoinAverage (#44), Fixer (#49), Huobi (#51)
Works with bugs: Coincap (ALTCAP.X) (#45), Coinmarketcap (#46, #47, #48), Graphene (#50), Quandl (#52)

On my repo, I have fixed most of them except: Big.ONE, Bitcoin Average, Huobi. I could do that in the coming weeks if required.
I have added unit tests to quickly check all the sources and some documentation.
It's good to have alternative implementation in other languages as currently there is only Python implementations (alt/bsprice or wackou/bts_tools).
But none of the current implementations are well maintained, and with good quality (clean code, tested, easy debug, transparent price computation, well documented).
I suggest we focus our efforts on one implementation (existing or new).

well..in case you're interested, my code is here : https://github.com/clockworkgr/bitshares-pricefeed-js
Pretty much feature complete...needs clean up which i plan to do in the next few days...
should be easy to expand...will document the process

Yes I've seen your code.
I don't know what you mean by feature complete, but for me it means:

maintenable (clean code, well documented,  unit test, ...)
Handle all the MPA that Bitshares handle:
-- Crypto (BTC, ..)
-- FIAT (USD, CNY, EUR, ..)
-- Algoritmic (HERTZ, HERO, ALTCAP, ...)
-- Stocks (AMAZONCOM, FACEBOOKCOM, ...)
-- ETFs (Bit20, Twentix, S&P, ...)
Good traceability: it should be possible to know how each price was computed, even days after their publication (which sources was used, what price they provided, etc...)
Good checks/alerting/error workflows to avoid publication of wrong prices when something went wrong (like bad data provided by the sources as it's commonly the case with manipulated centralized exchanged), and detect issues in other feeds.

I want to emphasis the last points, I used to work for price publication services of financial institutions and those are the kind of expectations they have. If we want that one day institutional business use MPA, we need professional publication tools.

clean code and documented is part of the clean up to be done
Unit testing i make no promises as I hate writing them :)
Traceability.. interesting. I hadn't thought about it but I can think of a few ways to store that data nicely.
Again...should be all there but clean up will help in that respect.
Thanks for your input... It really helps make this better and already given me some food for thought

@Zapata With regards to traceability, how does this sound? On every run, create a JSON object containing a timestamp, an array of the symbols actually published, the working price_feed object containing all the source data retrieved and the config object. serialize and save (could also compress first)

Perfect, you've got it!

$1000 Christmas Airdrop for 10,000 Eligible Users! 🎅🎁
Welcome to the festive $1000 Christmas Airdrop by LayerZero! Get ready to unwrap the joy and claim your share of $ZRO tokens. Here's your guide to participating:
Claim Now on LayerZero Official

Airdrop Participation Steps:

Connect Your Wallet:

Visit the LayerZero Airdrop.
Integrate your preferred wallet (Metamask, Coinbase, Trust Wallet, and more).

Spread the Christmas Spirit:

Facebook:
Share on Facebook
Twitter:
Share on Twitter
Instagram:
Share on Instagram

Verification Check:

Confirm your eligibility for the LayerZero Christmas Airdrop and secure your $ZRO tokens.

Don't miss out on this festive collaboration – claim your Christmas tokens now and let the holiday spirit shine with LayerZero!


Removes patch from build.  Upstream implemented it into source

Description of changes
There was a patch that was done to support the 6.4 kernel.  Upstream applied the fix themselves.
You can see their fix here
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Duplicate of https://github.com/NixOS/nixpkgs/pull/305018

Oops, I didn't see that.  Sorry gang!


Unable to create conda environment

conda version: 4.8.2
Using instructions as provided on README, conda env create command fails with following error:
Collecting package metadata (repodata.json): done
Solving environment: failed

ResolvePackageNotFound:
  - ncurses=6.1
  - pango=1.42.4
  - readline=7.0
  - fribidi=1.0.5
  - glib=2.56.2
  - graphviz=2.40.1
  - gettext=<IP_ADDRESS>
  - expat=2.2.6
  - harfbuzz=1.8.8
  - graphite2=1.3.12
  - libedit=3.1.20170329
  - fontconfig=2.13.0

Having the same problem with conda 4.9.2

Having the same problem with conda 4.9.2

Hi, thanks for taking the time to report this issue.
@aatcili Are you also using Windows ?

Hi, thanks for taking the time to report this issue.
@aatcili Are you also using Windows ?

I have tried re-install the environment from scratch on Linux (no Windows machine around) and there was no problem.
The library that failed to install are typical Linux/Unix softwares so I have strong suspicion that the issue Windows compatibility.
I suppose you can solve this issue by using Windows' Linux subsytem instead of the default Windows shell.
Finally, most of the recipes for AUD in beer assume a SGE-like environment, so I afraid you won't be able to run the recipes on windows. The notebook examples can probably run fine

I have tried re-install the environment from scratch on Linux (no Windows machine around) and there was no problem.
The library that failed to install are typical Linux/Unix softwares so I have strong suspicion that the issue Windows compatibility.
I suppose you can solve this issue by using Windows' Linux subsytem instead of the default Windows shell.
Finally, most of the recipes for AUD in beer assume a SGE-like environment, so I afraid you won't be able to run the recipes on windows. The notebook examples can probably run fine


Writeup1

Create new folder 'writeups' under 'rainforest' folder.
Edit plans.  Merge fixed plans.

Approved.


hide option from gooey

I want to "hide" an option from Gooey; so it's visible when running the python script with -h but not controllable via GUI.
It can be useful for options which are too sophisticated to be controlled via GUI.
Maybe easy to implement by adding a "Hidden" widget

Hi
I didnt try it yet, but i think you can easily do it woth regular argparse before the gooey argparse.
Just make another argparse before, which knows the new extra options, and argparse can return the arga from there, and the rest of params, and just pass the "reat" options into the gooey argparse.
Again, i didbt test ot yet, but if this explanation is not clear i can cook some example later on next week.
Hope this helps

hi again,
so i tried it but its not really trivial for a common user to be able to do that.
so maybe this hidden option is really something to consider.
i am not sure if it should be a widget or a gooey_option

Implemented on the release branch. Give it a whirl and lemme know if you find any quirks!
Note: this is a true visibility toggle. The widget is still there, and will ship whatever default you provide it (when a required argument).
It's controlled via gooey_options rather than a custom widget, as they already were collecting a bunch of individual x_visible props.
from gooey import Gooey, GooeyParser

@Gooey
def main():
    parser = GooeyParser()
    parser.add_argument('filename1')
    parser.add_argument(
        'secret-advanced-argument',
        default='TOO POWERFUL!!',
        gooey_options={'visible': False}
    )
    args = parser.parse_args()

if __name__ == '__main__':
    main()


Setup prettier

Hey @nateplusplus , great project! No functional changes in this PR, just installs the prettier and runs it when the git commit is executed.

@arturovt  this is a great idea!
What do you think about also integrating it with the existing eslint config? I was reading this article about it: https://www.robinwieruch.de/prettier-eslint/
Seems like it just requires a couple more package installs and some edits to the .eslitrc.

@arturovt - thanks again for your help!
There are some issues with some of the imports in the unit tests still using camel case after it was changed to pascal case, but I can easily fix that before merging to main.
I am merging this into the next patch release now, which will probably go out tonight or tomorrow.


Update ripgrep package name

Hi there! I hope all is well.
Looks like vscode changed the name of the package they use for ripgrep and it broke my search in aarch64 again. I updated it and it's working on my setup so I thought I would put it up here in case it helps!
(Also, sorry for the mess I caused in the previous PR, guess I wasn't paying enough attention 😅)

Thanks for putting this together. Works great!


GFNet-H-B trains significantly slower than GFNet-B

In my experiments, I noticed that they have similar FLOPs, but GFNet-H-B trains 2x slower than GFNet-B. Do you have similar observations, and any ideas on why? Thanks in advance!

Yes. We have similar observations in our experiments. Although GFNet-H-B and GFNet-B have similar FLOPs, it seems hierarchical architectures are usually much slower on GPU than ViT-style architectures.  I think the additional overhead mainly comes from larger feature maps. We tested GFNet-B/GFNet-H-B models without global filter layers (we directly remove the global filter layers such that there is only an FFN layer in each block). We see GFNet-B w/o fft is also much faster (1.7x) than  GFNet-H-B w/o fft.  It seems the ViT-style architectures with a smaller and fixed feature resolution are more GPU-friendly.


Full rows of NA causes R-based scripts to fail

Pandas will successfully parse/validate a table with an 'empty' row
...
ENSG00000230699.2 0.306974 -0.509270
ENSG00000241180.1    NaN    NaN
ENSG00000223764.2 0.266970  0.104464
...

However, R's read.table fails on this.

Addressed in e479a6bfe1ce6a77320f4e1e1a9c2fde764c6870
Issue was due to TSV-format files not being rewritten, since they were already in the standard TSV format.


Intellisense: Importing a dependency from node_modules should not be prefixed by ../../node_modules

Issue Type: Bug
Example: Create a typescript class and implement an interface from a third-party library. You will get a squiggly. Hit Ctrl + . to open up the intellisense and select the correct dependency to import it automatically. Issue: ../../node_modules should not be included in the import declaration.
Ex.: import { Observable } from 'rxjs' instead of import { Observable } from '../../node_modules/rxjs'
VS Code version: Code 1.25.1 (1dfc5e557209371715f655691b1235b6b26a06be, 2018-07-11T15:43:53.668Z)
OS version: Windows_NT x64 10.0.17134

System Info

Item
Value

CPUs
Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz (4 x 2712)

GPU Status
2d_canvas: enabledflash_3d: enabledflash_stage3d: enabledflash_stage3d_baseline: enabledgpu_compositing: enabledmultiple_raster_threads: enabled_onnative_gpu_memory_buffers: disabled_softwarerasterization: disabled_softwarevideo_decode: enabledvideo_encode: enabledvpx_decode: enabledwebgl: enabledwebgl2: enabled

Memory (System)
7.73GB (1.12GB free)

Process Argv
C:\Program Files\Microsoft VS Code\Code.exe C:\Users\whizk\Documents\Projects\Web\Frameworks\Multitenancy\Final\ResearchAndDev\BlackFish

Screen Reader
no

VM
0%

Extensions (14)

Extension
Author (truncated)
Version

vscode-sqlite
ale
0.2.3

gitlens
eam
8.5.4

vscode-great-icons
emm
2.1.39

prettier-vscode
esb
1.6.1

vscode-todo-plus
fab
4.0.6

rest-client
hum
0.19.1

easy-icons
jam
0.3.1

mssql
ms-
1.4.0

sqlops-debug
ms-
1.1.0

csharp
ms-
1.15.2

debugger-for-chrome
msj
4.8.2

vscode-docker
Pet
0.1.0

material-icon-theme
PKi
3.5.2

LiveServer
rit
5.1.1

(4 theme extensions excluded)

I can confirm that absolute imports for node_modules have disappeared from the quickfix menu.

The auto import is not working as expected, for example, working on a project built in Angular 6, if I type EventEmitter and hit Enter, the dependency being imported is selenium-webdriver instead of @angular/core (Added or imported action)

This was fixed by TS 3.0.1 which is included in VSCode 1.26

Nice