rpc+docs: update set of external auto-generated documentation to also support sub-servers

This really belongs in this repo, but lnd is more active so we'll be tracking it here instead.
We have our current dev docs which can be found at api.lightning.community that we all know and love. However, these docs haven't yet been updated to also cover the set of RPC sub-severs. As these sub-servers are starting to become more and more fully featured, we make available the same level of documentation we have for the rest of the RPC interface.

This is now addressed. See https://api.lightning.community/#lnd-grpc-api-reference.


add some spells

Mizushima: family name
Hayato:       first name
bibtex:        formatting tool for references of papers http://tug.org/bibtex/
XYZRPY:   X, Y, Z, Roll, Pitch, Yaw
VeloView:   The Velodyne Lidar Viewer based on Paraview Lidar https://www.paraview.org/veloview/

Thanks!


PL freezes in  logger_connector - result - horovod

🐛 Bug
PL sometimes freezes deep inside logger_connector - result - horovod.
The but is actually hard to reproduce  - happens about once in 100 times and might depend on timing etc, so I'm not sure it will be easy to catch on to model. I'm preparing a public version of code on which it is happening to be released really soon.
It happens even if there's a single worker.
From simple debug prints I see that freezing happens after train_epoch_end callback is called but before the next train_epoch_start.
Bellow is the python call stack extracted by gdb connected to the frozen process:
Backtrace
(gdb) py-bt
Traceback (most recent call first):
  <built-in method horovod_torch_wait_and_clear of PyCapsule object at remote 0x7fd9b7cf4ed0>
  File "/home/blackbird/.local/lib/python3.9/site-packages/horovod/torch/mpi_ops.py", line 944, in synchronize
    mpi_lib.horovod_torch_wait_and_clear(handle)
  File "/home/blackbird/.local/lib/python3.9/site-packages/horovod/torch/mpi_ops.py", line 972, in join
    return synchronize(handle).item()
  File "/home/blackbird/.local/lib/python3.9/site-packages/pytorch_lightning/plugins/training_type/horovod.py", line 147, in join
    hvd.join(self.local_rank)
  File "/home/blackbird/.local/lib/python3.9/site-packages/pytorch_lightning/plugins/training_type/horovod.py", line 174, in reduce
    self.join()
  File "/home/blackbird/.local/lib/python3.9/site-packages/pytorch_lightning/trainer/connectors/logger_connector/result.py", line 248, in compute
    value = self.meta.sync(self.value)
  File "/home/blackbird/.local/lib/python3.9/site-packages/pytorch_lightning/trainer/connectors/logger_connector/result.py", line 541, in wrapped_func

  File "/home/blackbird/.local/lib/python3.9/site-packages/pytorch_lightning/trainer/connectors/logger_connector/result.py", line 516, in _get_cache
    result_metric.compute()
  File "/home/blackbird/.local/lib/python3.9/site-packages/pytorch_lightning/utilities/apply_func.py", line 92, in apply_to_collection
    return function(data, *args, **kwargs)
  File "/home/blackbird/.local/lib/python3.9/site-packages/pytorch_lightning/trainer/connectors/logger_connector/result.py", line 543, in metrics
    value = apply_to_collection(result_metric, ResultMetric, self._get_cache, on_step, include_none=False)
  File "/home/blackbird/.local/lib/python3.9/site-packages/pytorch_lightning/trainer/connectors/logger_connector/logger_connector.py", line 323, in metrics
    return self.trainer._results.metrics(on_step)
  File "/home/blackbird/.local/lib/python3.9/site-packages/pytorch_lightning/trainer/connectors/logger_connector/logger_connector.py", line 279, in on_epoch_end
    metrics = self.metrics
  File "/home/blackbird/.local/lib/python3.9/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 812, in on_run_end
  File "/home/blackbird/.local/lib/python3.9/site-packages/pytorch_lightning/loops/base.py", line 151, in run
    output = self.on_run_end()
  File "/home/blackbird/.local/lib/python3.9/site-packages/pytorch_lightning/loops/fit_loop.py", line 234, in advance
    self.epoch_loop.run(data_fetcher)
  File "/home/blackbird/.local/lib/python3.9/site-packages/pytorch_lightning/loops/base.py", line 145, in run
--Type <RET> for more, q to quit, c to continue without paging--c
    self.advance(*args, **kwargs)
  File "/home/blackbird/.local/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1314, in _run_train
    self.fit_loop.run()
  File "/home/blackbird/.local/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1284, in run_stage
    return self._run_train()
  File "/home/blackbird/.local/lib/python3.9/site-packages/pytorch_lightning/plugins/training_type/horovod.py", line 111, in start_training
    self._results = trainer.run_stage()
  File "/home/blackbird/.local/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1274, in _dispatch
    self.training_type_plugin.start_training(self)
  File "/home/blackbird/.local/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 4267, in _run
  File "/home/blackbird/.local/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1284, in _fit_impl
    return self._run_train()
  File "/home/blackbird/.local/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 682, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/blackbird/.local/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1249, in fit
    elif self.lightning_module._log_hyperparams:
  File "/home/blackbird/Projects_heavy/NLP/langmo/langmo/benchmarks/base.py", line 117, in run
    self.trainer.fit(self.model, self.data_module)
  File "/home/blackbird/Projects_heavy/NLP/langmo/langmo/benchmarks/SST/__main__.py", line 42, in main
    finetuner.run()
  File "/home/blackbird/Projects_heavy/NLP/langmo/langmo/benchmarks/SST/__main__.py", line 46, in <module>
    main()
  <built-in method exec of module object at remote 0x7fdaad6608b0>
  File "/usr/lib/python3.9/runpy.py", line 343, in _run_code
  File "/usr/lib/python3.9/runpy.py", line 453, in _run_module_as_main

Environment
it seems to persist over couple of recent updates
but the current setup is like this
* CUDA:
	- GPU:
		- NVIDIA GeForce GTX 1060 6GB
	- available:         True
	- version:           10.2
* Packages:
	- numpy:             1.21.2
	- pyTorch_debug:     False
	- pyTorch_version:   1.10.0+cu102
	- pytorch-lightning: 1.5.3
	- tqdm:              4.62.2
* System:
	- OS:                Linux
	- architecture:
		- 64bit
		- ELF
	- processor:         x86_64
	- python:            3.9.7
	- version:           #22-Ubuntu SMP Fri Nov 5 13:21:36 UTC 2021

cc @tchaton @rohitgr7 @awaelchli

okey, I'm not 100% sure if that's the same issue or different one, but looks very similar to what I could trace on a single node (my desktop, first message)
when I'm using over a hundred GPUs, PL hands pretty much every single time after one epoch. I removed logging, removed validation, and from just debug prints I see this is happening after on_training_epoch_end callback

@undertherain Is there any chance you could verify whether this is related to horovod or not, by switching to regular ddp and try to reproduce?

I'm preparing a public version of code on which it is happening to be released really soon.

❤

so far could not reproduce with DDP, but didn't try at large scale yet

Hi everybody
finally could port my codes to DDP  (was waiting to upgrade of incompatible infiniband drivers on our cluster) - and it seems to work just fine.
Hotovod is still freezing, but personally I don't see it as an issue anymore, since  DDP enables lot's of extra features that horovod doesn't, like optimizer sharding etc.


I need to get the progress of file upload on server side.

I'm making an app that transfers files on Android and I need to show in app the progress of file upload from the client to the server (the phone in this case).
NanoHTTPD.parseBody does help with this.

Try this library which supports showing upload progress.
https://github.com/tejmagar/tinyweb


HDFS-15875. Check whether file is being truncated before truncate

NOTICE
Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.)
For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute

@ayushtkn Could you please help to  review this? Thanks

@ayushtkn Thanks for review ! Will commit the fix soon!


Wrapping CLASS or CAMB

After a little bit of digging, it would technically be possible to fully wrap either code in jax-cosmo,  using finite difference to get an approximation of gradients. Just for future reference, or if someone wants to do it, here is the procedure:

[ ] The easy part, adding a new JAX primitive,  essentially, just telling JAX how to manipulate your function, see here: https://jax.readthedocs.io/en/latest/notebooks/How_JAX_primitives_work.html#Defining-new-JAX-primitives .  You can define a custom JVP based on finite differences, so that you can get approximate gradients within the rest of the computation.

[ ] The not so easy part: Adding an XLA custom op to perform the computation.  So the problem is that doing the above will work, but you can't JIT  your program. For that, you would need to define a custom XLA op in C++ that would call CLASS/CAMB. It's technically possible, but painful >.< .... Here is the XLA reference for custom CPU ops: https://www.tensorflow.org/xla/custom_call

I'm labeling this as won't fix, because although in principle possible, may turn out to be a major pain in the long run. So, I"m gonna wait until someone write their own Boltzmann code in JAX, and failing that, we'll probably use an emulator strategy!

Oh wow, forget what I said:
https://github.com/google/jax/blob/master/jax/experimental/host_callback.py
With this API it looks like it will be possible to define host callbacks from compiled code!


Serial zmq relay

Added project built C executable which wraps a serial port and channels packets to zmq

What's the difference between this and #22? They look pretty similar to me? If they have similar changes lets get #22 merged and then see what's left in this PR.

closing #15


HTTPS on Marketing site is broken

Working with Heroku support.

looks like a dns issue on our end:
$ dig +noall +answer NS bootstrapcdn.combootstrapcdn.com.   2140    IN  NS  dns1.p07.nsone.net.
bootstrapcdn.com.   2140    IN  NS  dns2.p07.nsone.net.
bootstrapcdn.com.   2140    IN  NS  dns3.p07.nsone.net.
bootstrapcdn.com.   2140    IN  NS  dns4.p07.nsone.net.
bootstrapcdn.com.   2140    IN  NS  ns-1474.awsdns-56.org.
bootstrapcdn.com.   2140    IN  NS  ns-1689.awsdns-19.co.uk.
bootstrapcdn.com.   2140    IN  NS  ns-265.awsdns-33.com.
bootstrapcdn.com.   2140    IN  NS  ns-570.awsdns-07.net.
$ dig +noall +answer @dns1.p07.nsone.net www.bootstrapcdn.com
www.bootstrapcdn.com.   300 IN  CNAME   kanagawa-8115.herokussl.com.
$ dig +noall +answer @ns-1474.awsdns-56.org www.bootstrapcdn.com
www.bootstrapcdn.com.   300 IN  CNAME   bootstrapcdn-prod.herokuapp.com.

Updating CNAME, should propagate in 1 hour or less.

propogated.


chore: removed unsupported samples [talent solution]

Devsite update : cl/366134953
go/cloud-ai-2021-snippet-cleanup

PTAL @dwsupplee


Utils.find_min() performance issue and general poor performance remarks

While casually profiling this plugin with DotTrace with a large part count vessel, I noticed the Utils.find_min() method is taking 8% of the frame time :

The implementation is indeed extremely wasteful :
https://github.com/cmac994/KerbalWeatherProject/blob/748bd3c23ec83d96a69819a2c4ec5cf0933fdefa/KerbalWeatherProject/Util.cs#L469-L483
There is no reason to allocate a temporary array, and finding the index of the minimum difference can be done directly in the loop :
internal static int find_min_optimized(float[] arr1, double vval)
{
    int minIdx = 0;
    double minDiff = double.MaxValue;
    for (int i = 0; i < arr1.Length; i++)
    {
        double diff = Math.Abs(arr1[i] - vval);
        if (diff < minDiff)
        {
            minIdx = i;
            minDiff = diff;
        }
    }

    if ((arr1[minIdx] > vval) && (minIdx > 0))
        minIdx = minIdx - 1;

    return minIdx;
}

Additionally, I noticed that the values in the float[] passed to that method are always sorted in ascending order, in which case the method can be simplified to :
internal static int find_min_sorted(float[] arr1, double vval)
{
    for (int i = arr1.Length; i-- > 0;)
        if (vval > arr1[i])
            return i;

    return 0;
}

Here are benchmarks for the 3 variants, for an array size of 6 and 180 (which are the minimum and maximum sizes I observed being passed to the method) :
BenchmarkDotNet v0.13.12, Windows 11 (10.0.22631.4169/23H2/2023Update/SunValley3)
Mono x64 : Mono 5.18.0 (Visual Studio), X64 VectorSize=128
Job=Mono x64  Runtime=Mono x64

Method
SetSize
Mean
Error
StdDev
Ratio

Original
6
91.582 us
0.2676 us
0.2503 us
1.00

Optimized
6
9.153 us
0.1505 us
0.1408 us
0.10

OnSortedArray
6
4.332 us
0.0394 us
0.0369 us
0.05

Original
180
1,783.211 us
1.1990 us
1.0012 us
1.00

Optimized
180
215.198 us
1.4136 us
1.3223 us
0.12

OnSortedArray
180
79.295 us
0.5653 us
0.5011 us
0.04

On a side note, the FlightIntegrator overrides as a whole could really use a general optimization pass. Even excluding that specific issue, the plugin adds a significant overhead over a stock install. In my profiling setup (which is a high part count, worst case scenario), I get the following results for the FlightIntegrator.FixedUpdate() method :

Install
Normalized call time
Frame time

Stock
3 537 ms
9.1%

KWP
13 685 ms
31.7%

Even ignoring the overhead of the Utils.find_min() method (around 3300 ms of normalized call time), and after skimming through the codebase and profiler results, I'm relatively confident that most of the KWP overhead is due to suboptimal code in hot paths and could be eliminated.
One abomination I noticed is this :
https://github.com/cmac994/KerbalWeatherProject/blob/748bd3c23ec83d96a69819a2c4ec5cf0933fdefa/KerbalWeatherProject/climate_data.cs#L129-L135
Whatever you're doing with this 4 key dictionary, it doesn't need to be a dictionary, the keys shouldn't be strings, there shouldn't even be keys. The return value should be a struct of 4 fields, not a dictionary. Overall, I'm noticing a lot of wrong and utterly inefficient data types used all over.
For reference, here is the flattened call tree of the main KWP FI override :

And the expanded branch for GetDragLiftForce

I'm guessing the fact that you posted this 2 years after any activity on the repo was to show everyone how smart you are? With an absentee mod author and language that nearly mocks the code and authors attempt to bring something amazing to the game.
I do not really know a whole lot about coding but have tried to bring some things to a few of the games i play, its always nice when someone appreciates the effort and helps you improve it without insulting the efforts.
I know you got 50-100 hours doing this completely free thing for everyone. You know i never thought about something like this but could definitely do it better.
Perhaps real developers have thicker skin.

The repo might and author might be inactive, but people are still using this mod, and it was user reports and discussions that got me here.
I don't mean to disrespect the mod author in any way, and sure, I could have been a bit more gentle in my language, but on the other hand I don't see why I would sugarcoat the issues, which are affecting the users experience quite badly.
And finally, I actually took the time to provide guidance and research solutions, which is more productive than your useless personal attack.


net: dns: Add DNS cache for improved performance

In order to reduce dns resolve requests when using the dns resolver i added a caching mechanism. I would appreciate some feedback as to if a cache is even something we should add as well as opinions regarding implementation. I have described the current implementation as well as known problems here.
Implementation:
The current implementation is as follows:

One cache for all DNS resolver contexts
The cache returns as many results as possible for each query
Multiple queries per entry are supported
Entries are retained based on their RR TTL and removed as soon as it runs out
If the cache is full the oldest entry is replaced with the new addition

Known limitations:

When the resolver supports multiple addrinfo per query (CONFIG_DNS_RESOLVER_AI_MAX_ENTRIES) and those entries have different TTLs the cache will at some point no longer return all entries due to TTL expiry. This could be the expected behaviour but also might not be what the user wants.
If multiple DNS resolver contexts run the same query in parallel duplicate entries could be introduced into the cache since the same address will be added twice.

@jukkar I addressed your newest comments (still investigating regarding CI).
Regarding negative caching should we implement this now or later? I mentioned in my https://github.com/zephyrproject-rtos/zephyr/pull/68825#discussion_r1485928466 that the SOA ist still unclear.

Regarding negative caching should we implement this now or later?

We can do it later too.

@jukkar Build error has been fixed. Let me know if something still needs to be done.


Adding to_dask/from_dask

Was thinking the other day that it might be nice to have some convenience methods on Zarr's Array for converting it to a Dask Array and for storing a Dask Array to Zarr. May also make sense to have such methods on Zarr Groups (thinking of cases where the array has not been created yet). For the most part these are pretty straightforward to do outside of Zarr. That said, they would be convenient and maybe cut some boilerplate for end users.

Should add this is not urgent. Just something that might be interesting to think about later.

Yes this would be useful, although I reckon these should go into dask as
da.from_zarr and da.to_zarr - I think of dask as sitting one level above
zarr in the dependency stack.
On Fri, Nov 24, 2017 at 12:49 AM, jakirkham<EMAIL_ADDRESS>wrote:

Should add this is not urgent. Just something that might be interesting to
think about later.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/alimanfoo/zarr/issues/198#issuecomment-346714016, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAq8QtIGVFRMLILNeJ1wo5lpfuPe5yr2ks5s5hKHgaJpZM4QpRK_
.

--
Alistair Miles
Head of Epidemiological Informatics
Centre for Genomics and Global Health http://cggh.org
Big Data Institute Building
Old Road Campus
Roosevelt Drive
Oxford
OX3 7LF
United Kingdom
Phone: +44 (0)1865 743596
Email<EMAIL_ADDRESS>Web: http://a http://purl.org/net/alimanlimanfoo.github.io/
Twitter: https://twitter.com/alimanfoo

That sounds like a reasonable alternative. Thoughts @mrocklin?

What would the da.from_zarr or da.to_zarr functions take as inputs?
In the from_zarr case if it's just a Zarr array then this would presumably be identical to da.from_array, no?

Well it would be able to extract the dtype and chunks from the Zarr Array as well. This crosses over with issue ( https://github.com/dask/dask/issues/1983 ) a bit.

Yeah, I agree that this is the same as with HDF5 and other array objects.  The .dtype, .chunks, .shape attributes effectively form a protocol.  The issue here is that it's not always clear that dask.array should use the chunksizes in the storage format.

Fwiw I always do d = da.from_array(z, chunks=z.chunks). I'd appreciate a
convenience, e.g., da.from_zarr(z).
It would also be useful to have da.to_zarr(d, **create_kws) which is
basically a convenient way to materialise the result of a computation into
a new zarr array. Happy to provide further details if you think this is
worth considering.
On Fri, 24 Nov 2017 at 17:42, Matthew Rocklin<EMAIL_ADDRESS><javascript:_e(%7B%7D,'cvml','notifications@github.com');>> wrote:

Yeah, I agree that this is the same as with HDF5 and other array objects.
The .dtype, .chunks, .shape attributes effectively form a protocol. The
issue here is that it's not always clear that dask.array should use the
chunksizes in the storage format.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/alimanfoo/zarr/issues/198#issuecomment-346875102, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAq8Qr6Ov5KrkpPTBjDuH1uuGZ5HEMQpks5s5wAIgaJpZM4QpRK_
.

--
Alistair Miles
Head of Epidemiological Informatics
Centre for Genomics and Global Health http://cggh.org
Big Data Institute Building
Old Road Campus
Roosevelt Drive
Oxford
OX3 7LF
United Kingdom
Phone: +44 (0)1865 743596
Email<EMAIL_ADDRESS>Web: http://a http://purl.org/net/alimanlimanfoo.github.io/
Twitter: https://twitter.com/alimanfoo

One path is that if we want to support the convenience of matching the dask.array's chunks value to the zarr array's chunks value then I would suggest that we just bake this into da.from_array instead of creating a new from_zarr function.  The counterargument here is that often the chunksize of the underlying storage format is too small or too large for dask.array, and so you do want the user to explicitly specify the chunksize.

I'm ok with that. IOW just go with the solution proposed in issue ( https://github.com/dask/dask/issues/1983 )?

Separately would add that part of the reason I had been thinking about having to_dask in Zarr is we could bypass some of the indexing logic if the chunks in the Dask Array are the same as those in the Zarr Array. Namely we could pull directly from the underlying store and decompress the contents instead of worrying about how to handle slicings that overlap multiple chunks. If we want to get even more clever for this case, we don't even have to do the decompression initially at all. Instead we just return an in memory Zarr Array for the chunk. Thus allowing it to be decompressed when used.

FWIW the overhead of the indexing logic is very minimal compared to the
time taken in decompression, even with the fastest compressor, so I don't
think you would gain much performance-wise by providing a more direct route
to access a chunk.
Out of interest, what is the motivation for thinking to return an in-memory
zarr array for each chunk?
On Friday, November 24, 2017, jakirkham<EMAIL_ADDRESS>wrote:

Separately would add that part of the reason I had been thinking about
having to_dask in Zarr is we could bypass some of the indexing logic if
the chunks in the Dask Array are the same as those in the Zarr Array.
Namely we could pull directly from the underlying store and decompress the
contents instead of worrying about how to handle slicings that overlap
multiple chunks. If we want to get even more clever for this case, we don't
even have to do the decompression initially at all. Instead we just return
an in memory Zarr Array for the chunk. Thus allowing it to be decompressed
when used.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/alimanfoo/zarr/issues/198#issuecomment-346887073, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAq8QjK43HthFJ5D409DPxYngI1ALCgWks5s5xm2gaJpZM4QpRK_
.

--
Alistair Miles
Head of Epidemiological Informatics
Centre for Genomics and Global Health http://cggh.org
Big Data Institute Building
Old Road Campus
Roosevelt Drive
Oxford
OX3 7LF
United Kingdom
Phone: +44 (0)1865 743596
Email<EMAIL_ADDRESS>Web: http://a http://purl.org/net/alimanlimanfoo.github.io/
Twitter: https://twitter.com/alimanfoo

The main benefit is that we delay decompression and keep the memory footprint small in Dask. Depending on the operations performed, it may not be necessary to decompress at all. The tail end of this comment provides one such use case.

Should add discussion about adding to_zarr/from_zarr is in issue ( https://github.com/dask/dask/issues/3457 ) and an implementation is being pursued in PR ( https://github.com/dask/dask/pull/3460 ).

This is irrelevant as Dask can now convert to/from Zarr thanks to @martindurant's PR ( https://github.com/dask/dask/pull/3460 ).


RestForm should clear values when datatype changes in "add" mode

When adding a new record via RestForm, you can still change the data type, causing the fields to re-render.  As this can cause unexpected values to attempt to be set on fields of different types in unusual ways - and as we are adding a new record and don't have any data we really need to maintain - it seems simpler and more robust to simply null out any already-entered values if the user goes and changes the type in-flight.

Fixed this here, with addReaction().  Very useful method in this context.
https://github.com/exhi/hoist-react/commit/74497e9cffb2
(cr: Don)


Statement on testing Rancher on Kubernetes seems inaccurate

On https://rancher.com/docs/rancher/v2.x/en/cluster-admin/editing-clusters/#upgrading-kubernetes it states: "Before a new version of Rancher is released, it’s tested with the latest versions of Kubernetes to ensure compatibility."
However, Rancher 2.2.6 was released on July 23rd and supported up to Kubernetes 1.15.0. Latest version of Kubernetes at that time was v1.15.1.
Semantically, it's also vague what "latest versions" mean. There's really only one latest version. However, you could claim "latest versions" means the latest version of each 1.x line (example, 1.14.9, 1.15.2)

Closing this issue because https://github.com/rancher/docs/pull/1860 was merged


Clean up white-spaces

Remove unnecessary white-spaces and replaced tabs with 4 spaces for consistency.

@veniware I think you need to raise the PR against test branch rather than main branch.


feat: port Analysis.Calculus.Deriv.Pow

[ ] depends on: #4434
[ ] depends on: #4436

This PR/issue depends on:

leanprover-community/mathlib4#4434
leanprover-community/mathlib4#4436
By Dependent Issues (🤖). Happy coding!

This PR/issue depends on:

leanprover-community/mathlib4#4434
leanprover-community/mathlib4#4436
By Dependent Issues (🤖). Happy coding!


Large number of metadata objects: what can be a reason?

Performing memory profiling, found quite a number of objects related to EF metadata, mostly TypeUsage and MetadataProperty .
What could be a reason and how to diagnose this issue?
I assume that it depends on number of tables/columns/relations, but looks like there is no linear dependency.
Here is the test for just two tables - about 800 TypeUsage objects.

Here is the test for 50 tables - about 10K TypeUsages:

Production mode with several apps in worker process:

EF version: 6.1.3.

@nchaly in case it helps: EF6 canonical functions and providers specific functions are described using the same metadata objects you are counting here and those get loaded into memory even before a specific model is loaded. In other words, there will be a minimum of metadata objects loaded into memory even for an empty model.
Also, AFAIR, there are TypeUsage objects for each column and each property and each parameter in the model.

@rowanmiller I don't think the question has the proper answer.

@nchaly can you share a project or code listing that we can use to reproduce the issue?

Hi again. The code in question was a part of IP and I had to create something from scratch. It took a while, but now it's ready - https://github.com/nchaly/ef6-metadata-test
My goal was to make something simple and "all-inclusive": I've taken a Northwind model as a sample; using ClrMD I was able to count objects without need to get an external profiler.
Here is the chart built on sample data - https://raw.githubusercontent.com/nchaly/ef6-metadata-test/master/Chart.PNG. Growth rate for TypeUsage is quite high, as well as for entire System.Data.Entity.Metadata.Edm namespace.

@divega Can you investigate this?


Installing LibSassBuilder hides SCSS files from the solution explorer in Rider

As the title says my SCSS files disappear in the solution explorer of Jetbrains Rider when I install this library.
I can still see the files in Windows Explorer and they also get compiled when building the project.
Uninstalling the library makes the files re-appear.
Do you have any idea why this is happening?

I might depend on the file location, as certain folders are hidden by default like bin and logs - what is your file path?

The file is located in a path like this: "[project root]/Shared/Test.razor.scss".
And the file is visible without LibSassBuilder installed.
I can even have two projects in the same solution - one with the library installed and one without and the SCSS files are hidden only in the one where it is installed.

I can confirm this - all scss files invisible in Rider (macOS).

Think I have a fix, just want to do some final testing...
For now you can add the below to your csproj to show the files:
    <ItemGroup>
        <None Include="**/*.scss" />
    </ItemGroup>

Awesome, thanks! I'll check it out first then after the summer vacation :)

This should be resolved in v1.6.4


Scrolling to hash on IE doesn't seem to work

For example, when using footnotes, the generated link [N] does not scroll to the bottom of the page.

Well I have a book where it only fails for a certain footnote. I can provide it on demand...


SCB-622 Compatible with old version

Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Make sure there is a JIRA issue filed for the change (usually before you start working on it).  Trivial changes like typos do not require a JIRA issue.  Your pull request should address just this issue, without pulling in other changes.
[ ] Each commit in the pull request should have a meaningful subject line and body.
[ ] Format the pull request title like [SCB-XXX] Fixes bug in ApproximateQuantiles, where you replace SCB-XXX with the appropriate JIRA issue.
[ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[ ] Run go build go test go fmt go vet to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Coverage increased (+0.2%) to 73.058% when pulling 43045b741a1cb8e87f5697cd99404c28e1b5c1c6 on little-cui:del into eaa45804c97f05450f5ea066daaec01978244896 on apache:master.

Codecov Report

Merging #360 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #360   +/-   ##
=======================================
  Coverage   69.33%   69.33%
=======================================
  Files          18       18
  Lines        3437     3437
=======================================
  Hits         2383     2383
  Misses        885      885
  Partials      169      169

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update eaa4580...d0b8d26. Read the comment docs.


Support asynchronous filters

My issue is that I'm trying to use mathjax-node to create a showdown extension, but mathjax only exposes an asynchronous API. Ex:
var mjAPI = require("mathjax-node/lib/mj-single.js");
mjAPI.start();

var yourMath = 'E = mc^2';

mjAPI.typeset({
  math: yourMath,
  format: "TeX", // "inline-TeX", "MathML"
  svg: true,
}, data => {
  if (data.errors) return data.errors;

  console.log(data.svg);
});

Would it be difficult to allow for the filter function in an extension to accept a promise that resolves to a string?

It looks like it would be decently difficult, seeing as how all calls to makehtml assume it synchronously returns the text. Something like this would probably affect a lot of files.

Due to the nature of markdown itself, each block, to be parsed correctly, requires context.
For instance,    - bar can either mean:

a sub list- foo
    - bar

or a block of code

This means the parsing, to catch all possible edge cases, must be done sequentially.
Turning showdown in async would prove extremely dificult.
However, I think you can pre parse the text with math-jax and then parse it through showdown. Since html is valid in markdown, showdown would leave those blocks alone.

This is the solution that I ended up on: https://github.com/Awk34/awk34_project_euler/blob/9bd20689c98be289d66112ac48d0d81404ce93ea/js/util/markdownTools.js
I'm not saying all of the parsing has to be asynchronous, and to be done non-sequentially. It would just be nice if a subParser could handle an asynchronous response. That way I could use the callback response of MathJax in an extension. The full converter could still wait for the subParser to finish, but it would be done with an asynchronous callback/promise instead of assuming that the subParser returns text in a synchronous matter. Like this:
let text = '...';

text = syncParser1(text);

asyncParser.then(newText => {
  text = newText;

  text = syncParser2(text);

  return text;
});

@Awk34 I think I might be missunderstanding you (or failing to grasp the concept), but I don't see how it can be accomplished properly.
I mean, sure, I can wrap every subparser in a promise, since you can "inject" an extension before or after any subparser, but...

I think this is a better example:
// settable extensions
let extensions = [ext1, ext2, ...];

async function convert(text) {
  // Showdown converters
  text = syncParser1(text);

  // User extensions
  for(let i = 0; i < extensions.length; i++) {
    let result = extensions[i](text);

    // if extension returns a promise, wait for it
    // if not, assume synchronous extension returning mutated text
    if(result instanceof Promise) {
      text = await result;
    } else {
      text = result;
    }
  }

  // more Showdown converters
  text = syncParser2(text);

  return text;
}

Assume extensions is an array of extension functions, and assume convert is the main Showdown converter function. Instead of assuming that all extensions synchronously return text, it allows for them to also return a Promise, in which it waits for the promise to be fulfilled, and then moves onto the next extension. I hope this gets my point across better.

Since parser order is mandatory, and they have to run sequentially, in your example the text = syncParser2(text); can only run after the extension completes execution or the extension might not work.
So, I can only think of 2 ways of allowing async extensions properly:

block the execution of the parser and wait for said extensions to finish processing (effectively turning async into sync)
Rewriting the library, making each sub-parser async,

Scenario 1
async / await browser support is currently very limited and showdown should be reasonable cross browser compatible (ES5).
With ECMAScript 5, I don't know how to block execution and wait for promises to resolve.
@Awk34 Maybe there is a way?
Scenario 2
This is possible, obviously,
But it would require a complete refactoring of the library and respective tests, something I would ratehr not do.
@Awk34 I was hoping I've missed something, and there was a 3rd option.

I wrote my example with async/await just for clarity's sake. You can see the ES5 Babel would convert this to here. The Promise check would probably have to be changed to something like if(tyepof result.then === 'function')

I will put this on pending. Although I think I found a way to implement this, this requires a bit of rewriting in the library and an overhaul of the testing framework too.


[Task] - Add file exists

Check file exist prior to open it. Should be lighter than std::filesystem::exists [Link].

Attempt with

#1163


CS0056 & CS0057 code-fix

Fixes CS0056 and CS0057 in #381.
Implementation with tests for CS0056 and CS0057 compiler error numbers.
@giggio, please take a look because it's been a while since I made a pull request to code-cracker. Let me know if anything.
Thanks :+1: !

@baks these 2 looked real nice!
The only thing I would ask is for you to please update your branches. I had to manually merge your work.
Checkout the vnext branch and run:
git pull https://github.com/code-cracker/code-cracker.git vnext

@giggio, I'm sorry about that, github showed me that these branches can be automatically merged :cry:
I'll do what you say and hope next time there will be any problem with merging my PR :smiley:
Thanks!


Correction for raw affiliation Chaire UNESCO "formation professionnelle, construction personnelle, transformations sociales"

Correction needed for raw affiliation Chaire UNESCO "formation professionnelle, construction personnelle, transformations sociales"
raw_affiliation_name: Chaire UNESCO "formation professionnelle, construction personnelle, transformations sociales"
new_rors:
previous_rors: 03q99v794
works_examples: W4389489384;W4392120026;W4392181886;W4392380689
contact: 609d56d07fed037ef0f9cfe696f7dbd0:fef29077d9175b6fc2a7ad78749e @ univ-lille.fr

This issue was accepted and ingested by the OpenAlex team on 2024-12-19. The new affiliations should be visible within the next 7 days.


Update cache_domains.json

What CDN does this PR relate to
Crossfire
Capture method
Netlimiter
Testing Scenario
i haven't tested it yet because i dont have the knowledge to manually add a zone but i notice it in my limiter that give me this remote name.
can you teach me how to manually add zone to the container? thank you.
PS: I am from Philippines.

i will edit the cdn after running wireshark. thank you. i will leave it blank first. it seems netlimiter and wireshark give diffrent host.

Excuse my ignorance but what is crossfire?

crossfire is a FPS game here in the Philippines. it is well known here and lots of players.
here is now their official website since they upgrade from previous one
http://forums.gameclub.ph/index.php?board=237.0

Thanks for the input but I'm afraid i'm going to have to reject this due to invalid configuration, crossfire.txt in json but not included in commit. Please feel free to reopen/resubmit with these changes resolved.


added arraytype to the readme file

Sorry, created two pull requests for related changes

Closed by #8


Reorganize tests folder

We have quite many tests for the Task class. We can create a tests/task folder and split them into multiple files.

We've decided to implement the functionalities originally planned for the annubes package within the NeuroGym package instead, which represents the current state-of-the-art.


Ignore CheckReturnValue in assertThrows / expectThrows

If the invocation occurs in the body of a lambda inside assertThrows / expectThrows, ignoring the return value is probably intentional.
see also #576, #446

er, this happened: f9ea34b52ace79c5f3e1d39fd0f1ade2eba9b06a


Exception when alpacka-backtrader-api client is connected

2022-11-03 19:22:15,758 root Alpaca Proxy Agent v1.0.1
2022-11-03 19:22:15,758 root Using the Alpaca Websocket
2022-11-03 19:22:15,758 root Connecting to the paper account endpoint
2022-11-03 19:22:15,758 root Using the free subscription plan(iex)
2022-11-03 19:22:42,767 root Subscribing to: {'quotes': ['META']}
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "main.py", line 50, in consumer_thread
    subscribe(channels)
  File "main.py", line 56, in subscribe
    conn.subscribe_trades(on_message, *channels['trades'])
KeyError: 'trades'

Any idea?

+1


HttpRequestException StartAsync in .NET Client v.1.0.2

Hi
In my .NET Core 2.1 Azure Function I use the SignalR .Net Core Client v.1.0.2. Very often I get an exception in the function on the StartAsync call.
System.Net.Http.HttpRequestException:
at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode (System.Net.Http, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at Microsoft.AspNetCore.Http.Connections.Client.Internal.SendUtils+d__0.MoveNext (Microsoft.AspNetCore.Http.Connections.Client, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.IO.Pipelines.PipeCompletion.ThrowLatchedException (System.IO.Pipelines, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
at System.IO.Pipelines.Pipe.GetReadResult (System.IO.Pipelines, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
at System.IO.Pipelines.Pipe.GetReadAsyncResult (System.IO.Pipelines, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
at System.IO.Pipelines.Pipe+DefaultPipeReader.GetResult (System.IO.Pipelines, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
at Microsoft.AspNetCore.SignalR.Client.HubConnection+d__45.MoveNext (Microsoft.AspNetCore.SignalR.Client.Core, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.SignalR.Client.HubConnection+d__31.MoveNext (Microsoft.AspNetCore.SignalR.Client.Core, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.SignalR.Client.HubConnection+d__31.MoveNext (Microsoft.AspNetCore.SignalR.Client.Core, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.AspNetCore.SignalR.Client.HubConnection+d__24.MoveNext (Microsoft.AspNetCore.SignalR.Client.Core, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=adb9793829ddae60)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at TruckerWebApp.Functions.NotifySignalrHubUserFunction+d__3.MoveNext (TruckerWebApp.Functions, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=nullTruckerWebApp.Functions, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=null: D:\a\1\s\TruckerWebApp.Functions\TruckerWebApp.Functions\NotifySignalrHubUserFunction.csTruckerWebApp.Functions, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=null: 94)
Exception Message
Response status code does not indicate success: 404 (Not Found).
Thanks in advance
Eric

How are you hosting the site? Is it a single server or do you have multiple instances of your app running? This often occurs when the initial request to create the connection goes to one server but another request for the same connection goes to a different server. All requests for a single connection need to go to the same server, so for a service like Azure App Server you need to enable ARR Affinity.

On a Single Instance on App Service Plan in Azure.

So you're hosting your server inside an Azure Function? Can you share sample code as to how you're doing this? It's not something I'm familiar with.

I use Azure SignalR Service, but I can post the Hub Code

Can you provide client-side logs as described in the Diagnostics Guide? Network traces would also help. It should be possible to get logs from Azure Functions, though I don't have direct experience with it. Does the issue reproduce with a standard client outside of Azure Functions?
If the issue is with the client, we can handle it here, but if it ends up being a server problem it would actually be something for the Azure SignalR Service repo to track. Let's see what we can figure out about the client behavior though.

I improved my function and added the diagnostic code setup. But only my own log messages get logged to the azure function log, but not yours from the signalr .net client.
Please take a look on my setup in gist:
https://gist.github.com/cnlogistics/f631c20d8feb4daa0f1f69edc8a17644

Azure Function NotifyUserSignalrFunction Logfile:
2018-08-12T10-42-11Z-41ce5e8c7e (1).log

@anurse  maybe helpful. I exported the Application Insights logs from the modified Azure Functions code that invokes StartAsync
application_insights_query_with_404_startasync_signalr.xlsx

The problem is that Azure Functions don't seem to log their console output, and you've only configured the Console logger in ConfigureLogging so you're not seeing any SignalR logging. Does the issue reproduce at all if you run the same code in a simple Console app? That might make it easier to gather logging information.
Alternatively, you can look at solutions to get logs out of the Functions app. One way is to pipe the logs from SignalR to the TraceWriter you receive as a parameter. I sketched up a way to plug the ASP.NET Core Logging system that SignalR uses into that: https://gist.github.com/anurse/ed4eb12851498c87e259fd123f8f37e5 . Alternatively you can investigate using Application Insights or some other similar tool to ship logs out of your application.
I notice you also have an issue filed in azure-signalr, please let us know if you're able to track down the solution in that issue so we can close this issue out. Feel free to ping me over there if you have questions (we don't track that repo as closely but I'm very happy to chime in if you have questions!)

Hi @anurse
I applied your Azure Function logger code tovthe HubConnection.  It's even harder to export data in a readable format from Application Insights. I attached the CSV format from that period in time. To help you I made screenshots from the Analytics Query around that expection in StartAsync() The red arrow marks that timestamp where the error starts taking place.

From the logs, it looks like you're getting the 404 when talking to the service:
warn: Microsoft.AspNetCore.Http.Connections.Client.Internal.LoggingHttpMessageHandler[2]
      Unsuccessful HTTP response 404 return from POST 'https://truckerwebapp-dev.service.signalr.net:5001/client/?hub=truckerwebapphub&id=xGkiZp4NCvl_oFU8oMy49A'.

I think this is an issue someone on the Azure SignalR Service team is going to need to diagnose, since the 404 is coming from there. Can you file an issue over at https://github.com/azure/azure-signalr to see if they have an insight into thise? Also, I'm not seeing much of the "debug"-level logging from SignalR either, perhaps try changing from SetMinimumLevel(LogLevel.Information) to SetMinimumLevel(LogLevel.Debug) and see if any more tracing comes from SignalR? (I realize that the Diagnostics Guide suggests exactly the code you wrote, it's possible the guide is incorrect here)

Sure. I will file an issue there tomorrow. Regarding the logs.

What am I missing?

@anurse You can close that issue now. I can't.

@anurse One last thing to mention:
I noticed that this SDK (.NET Signal Client) creates an new HttpClient on each StartAsync() and disposes that HttpClient on StopAsync().
That is considered as an anti-pattern:
https://aspnetmonsters.com/2016/08/2016-08-27-httpclientwrong/
Since Asp.Net Core 2.1 there is a HttpClientFactory:
http://www.talkingdotnet.com/3-ways-to-use-httpclientfactory-in-asp-net-core-2-1/
If the SDK really disposed the HttpClient then you should really change the code. In my Use-Case in an Azure Function I Start the SignalR communcation, send some message and stop the communication again. So open -> send -> close. If I , and I will , do that 100000 times a day ... I will run in that HttpClient issue.
I close the SignalR socket only because there is currently no reliable property on the HubConnection to verify if the SignalR communication is open.

I noticed that this SDK (.NET Signal Client) creates an new HttpClienton eachStartAsync() and disposes that HttpClienton StopAsync().
That is considered as an anti-pattern:

Yes, we're very aware of that pattern. SignalR reconnections are much rarer and so this wasn't considered an immediate concern in our initial capacity testing. Are you seeing a problem? We can certainly consider improving upon this in the future. If you are seeing a capacity problem here, please post a new issue with the data so we can use that for our planning.

In my Use-Case in an Azure Function I start the SignalR communcation, send some message and stop the communication again. So open -> send -> close

You can share the HubConnection across multiple invocations, in fact you already are in the code you posted because the fields are static. In fact that could cause some race conditions in the code you've got because one invocation could be stopping the connection while another tries to send. If you apply proper locking around starting and stopping, it is safe to have multiple threads calling SendAsync/InvokeAsync/etc. The HubConnection object will perform the appropriate locking to ensure only one thread is writing to the connection at a time.

I close the SignalR socket only because there is currently no reliable property on the HubConnectionto verify if the SignalR communication is open.

In the next version, the connection state will be exposed on the HubConnection object itself. See https://github.com/aspnet/SignalR/issues/2127

What I saw in the logfile is the instantiation of a HttpClient on each hubconnection.StartAsync() and the disposal of the same on hubconnection.StopAsync(). Maybe I haven't understand properly. But can I leave connection simply started?

You are correct. A new client is created every time the connection is stopped and restarted, but you can leave the connection running for as long as necessary. In the case of a Functions App, it's possible the Azure functions infrastructure will shut it down if you don't receive any invocations for a while but that's fine, since it will kill the whole process, thus terminating the TCP connections left outstanding. If you experience network issues and have to reconnect, then you might leave TCP connections open for a short period as described in the article, but this should have a minor impact.

@anurse I now know why only Information level trace logs got written to Application Insights. I had to turn the loglevel to Trace for Function in host.json. Here my one
Well some hidden gems.
Here the Azure Function Monitoring reference


PYNQ-Z1 output JPEG by HDMI

I want to output a JPEG  by HDMI,
but I get a 'JPEG' object has no attribute 'flush' error,
img_folder = './yoloimages/'
img_file = os.path.join(img_folder, random.choice(os.listdir(img_folder)))
im = Image.open(img_file)
hdmi_out.writeframe(im)

How to transform a JPEG to a HDMI frame?

make na new output frame and then assign jpeg image to that frame . and pass that output frame to hdmi_out.writeframe().

Thank you. @aniqurrehman
Now I can output  my own JPEG by convert it to np.array. But I can directly assign outframe = np.array
import cv2
import numpy as np
from PIL import Image

img = Image.open("/home/xilinx/jupyter_notebooks/base/video/data/opencv_filters.jpg")
inframe = np.array(img)

grayscale = np.ndarray(shape=(hdmi_in.mode.height,
                              hdmi_in.mode.width), dtype=np.uint8)
result = np.ndarray(shape=(hdmi_in.mode.height,
                           hdmi_in.mode.width), dtype=np.uint8)
cv2.cvtColor(inframe,cv2.COLOR_RGB2GRAY,dst=grayscale)
cv2.Laplacian(grayscale, cv2.CV_8U, dst=result)
outframe = hdmi_out.newframe()
cv2.cvtColor(result, cv2.COLOR_GRAY2RGB,dst=outframe)
hdmi_out.writeframe(outframe)

I get the error by directly assign outframe = np.array
outframe = hdmi_out.newframe()
outframe = inframe
hdmi_out.writeframe(outframe)

AttributeError                            Traceback (most recent call last)
<ipython-input-87-536a109be150> in <module>()
      1 outframe = hdmi_out.newframe()
      2 outframe = inframe
----> 3 hdmi_out.writeframe(outframe)

/opt/python3.6/lib/python3.6/site-packages/pynq/lib/video.py in writeframe(self, frame)
   1290
   1291         """
-> 1292         self._vdma.writechannel.writeframe(frame)
   1293
   1294     async def writeframe_async(self, frame):

/opt/python3.6/lib/python3.6/site-packages/pynq/lib/video.py in writeframe(self, frame)
    632                     asyncio.ensure_future(self._interrupt.wait()))
    633             self._mmio.write(0x04, 0x1000)
--> 634             self._writeframe_internal(frame)
    635
    636         async def writeframe_async(self, frame):

/opt/python3.6/lib/python3.6/site-packages/pynq/lib/video.py in _writeframe_internal(self, frame)
    614                 self.sourcechannel.tie(None)
    615
--> 616             frame.flush()
    617             next_frame = (self.desiredframe + 1) % len(self._frames)
    618             self._frames[next_frame] = frame

AttributeError: 'numpy.ndarray' object has no attribute 'flush'

You need to use the outframe[:] = inframe syntax. This will copy the data into the outframe array which is known to the video subsystem. By using outframe = inframe you are only setting the outframe reference to be the same as inframe. As inframe isn't know to the video subsystem you get the error.

Thank you so much, I will close this issues. @PeterOgden @aniqurrehman


Improve and clarify README and gh-pages

Maybe even tackle #14 as well

The donate tag is broken and needs to be fixed as well


Number of layers have to match

I believe both transformers and LSTM layers require that the number of encoder and decoder layers match. However, they're specified as separate arguments and flags. This makes it hard to do hyperparameter search including them; if one does something like
parameters:
  encoder_layers:
    values: [1, 2]
  decoder_layers:
    values: [1, 2]

in a hyperparameter sweep, half the draws will fail at initialization because you'll draw a different number of the two.
I propose that we just standardize this as --layers. Thoughts?

number of Encoder and Decoder layers do not need to match. IIRC this constraint only applies to pointer generator (though I do not think that it has to? I think they just do in the implementation I followed because the decoder is initialized with he last encoder state or something like that.).
I believe that we initialize most models with a random, learned tensor, and that during decoding, we only attend to the last layer of the encoder.

Oh, this is pointer-generator only? I didn't realize. Then nevermind: maybe I can document.

Yes! Although I am not sure for Transducer, but would assume it also considers the last encoder layer only.


Is it possible to cover with asterisk partial of the user's input?

First of all, I want to say thank you for this great plugin! I have more of a question or feature request than a bug report.
Is it possible to cover with asterisk partial of the user's input?
For example, the US social security number is a nine digit number that fits the 999-99-9999 mask. This mask works great. However, since social security number is a sensitive data, I don't want to display the full number on the screen. Instead, I want to show only the last 4 digits.
If the social security number is 123-45-6789 then I want it to show as ***-**-6789. If the user is still typing, we would only hide the first characters. So if the user typed 7 out of 9 characters and still typing then we show ***-**67__ >> next key up ***-**678_ and finally ***-**6789
How can I create this behavior?

Hi, did you find any solution?

@CrestApps ,
That would indeed a nice extra feature.  I will add such functionality.

Could you please tell me how to use inputMask in react 18.
Requirement is as follows,
If the input number is 6765 6666 567, then I want to show it as **** **6 567.
When user is typing, the digit should convert as asterisk() up to 13 digits and rest the numbers.
Few examples:
6765 6666 567   -> **** ***6 567
6765 6666       -> **** ***6
67              -> **
6               -> *
Also, is it possible to get the unmasked number using any function as I need to show the actual number when used click on the eye icon.
How can we achieve this behavior using your library. Please advise


Consider adding the bar at the bottom

Is your feature request related to a problem? Please describe.
I am used to this bottom bar so I am missing it in your app.
Describe the solution you'd like
It would be really nice if you would consider to adding it in files UWP.
Describe alternatives you've considered
currently no alternative idea how these features could be implemented otherwise.
Additional context
I am talking about this orange marked bar

Especially showing the contained files/folders inside the current folder and how many files you have currently selected and the two buttons to switch between list view and grid view on the right are really useful imo. (I use them a lot and never the ribbons to switch the view)

I think this is something that should be optional as many users want a really simple UI. In any case I see how this could is useful so I will approve the request.

If you were to add this bar, could you also add a label that would show the total size of the folder or selected documents/items?

@Phil4102 The status bar is in its early stages, we will explore the behavior you suggested when we spend more time on this control.

A very basic implementation will be included with the v0.7.3 release. We don't have a timeline for when this control will be fully functional but we will work on improving it over the next few releases.

Thank you for considering my feedback. Its' a feature that doesn't exist in the native File Explorer, and it would be really easy to view the size of selected items or the current file path.

@szaimen The status bar has been implemented and will roll out with the next release.


[BUG] Wrong calculation in event sender

This is an issue related to this bug : itJustWorksTM/EiffelVis#115
We will take care of implementing a fix for this

Fixed with https://github.com/ItJustWorksTM/EiffelVis/pull/121 using cherry picking from https://github.com/ItJustWorksTM/EiffelVis/pull/121/commits/45ff7f0bd25b441b668200f57929613abc0e887b


fix broken table in tooltip-documentation.md

This markdown is automatically generated from the code comments, we can't change it that way


Gradle reformatting

Reformats the gradle docs to match the output of gradle init --type java-application

Hi @mwkroening,
Can you please the conflict?
IMO, refactoring of arguments into javaArgs() is not a very good idea since we want to explicitly show users, arguments required to compile and run a program.

I updated the PR. Also I undid the reordering.
Although the order of the configurations does not match 'gradle init --type java-application' anymore, it is fitting with the explanatory HTML file.


[FR] Maximize the only window on screen

Is your feature request related to a problem? Please describe.
I like to have a small gap between windows just to have a little more visual separation without having to up the size of window decorations. That is not necessary when there is only a single window on screen. It also means you cannot just slam your cursor into one side to click on things on window edges but need to actually aim for the buttons, e.g. firefox tabs at the top.
Describe the solution you'd like
(Option to) Maximize the only window so that it fills the entire maximized area.
Additional context
https://github.com/zeroxoneafour/polonium/compare/master...Daxtorim:polonium:maximize-lone-window
I already tried the naive thing and added client.setMaximize(true, true) to rebuildLayout() in src/main.ts but that doesn't work. I mean it kind of does. The window changes state—the maximize button on the window decoration changes—but it wont actually change geometry, it still remains in its tile with gaps around it.
I also tried to put client.setMaximize(true, true) after the for (... of ...) loop (keeping track of the last client of the loop and checking that it is the only one), and that works, but that will untile the window and basically break tiling entirely.
Hopefully you have a better idea of where to put it and can find a solution. Thanks :+1:

KWin tiling overpowers basically everything, so the maximize function would have to be similar to the fullscreen function in that it would have to disable all other tiling on the window. I definitely think I can cook something up, though.

So it was quite a bit more complicated than just using setMaximize, but I think that I have a decent solution commited. See if it works for you. It has quite a few random bugs, but if you switch desktops to allow the layout to rebuild, it goes back to normal (making me think KWin wasn't really designed for these sorts of API-spammy scripts)

When starting from an empty screen and opening a window it works, but when opening a second window the first one gets placed in a random geometry and any action that causes the layout to be rebuild fixes it. But from then on out it works fine as long as you don't end up with an empty screen at any point.
Kinda janky but it works I guess. I'll see over a few days if it's fine or gets annoying.

The latest build has a shortcut to rebuild the layout. I don't know if I can get it any less janky with maximizing (theoretically it should work, as the layout rebuilding correctly tells me everything is working internally in the engine), but if you only need the window to fill the space and not to be "maximized", I may be able to set the root tile padding to 0 when there is only one window to fill the space.

but if you only need the window to fill the space and not to be "maximized"

No, I do want the window to be maximized so clicking on the border wont try to resize the window but actually click a button there.
I've been playing around with it myself in an attempt to maybe find a solution but all I managed was a workaround that is basically just calling rebuildLayout() again after a timer runs out ('cause no sleep for kwin scripts).
https://github.com/zeroxoneafour/polonium/compare/master...Daxtorim:polonium:workaround-for-single-window-unmaximization
It's a a hacky workaround but it works—and is better than using a shortcut to do the same thing manually. I can make a PR if you want or you can cherry-pick the commit yourself.

Dang it, and I was just in the middle of writing release notes! :)
This is actually incredibly useful, as I didn't know you could use QTimer in KWin scripts (when doing things like making new QRectF() it always said it wasn't defined). I'll also use a QTimer to time layout rebuilding (as KWin never gets all the tiles down in time before calling layoutChanged callbacks).
Needless to say, this is an instant merge. Thanks for your help with this issue! I'll probably close it as complete once I test the changes.


Go not found in macOS 13 runner

Description
The macOS 13 runner README states that it has the "Go command" installed.
However, if you run go version you get an error:
go: command not found

Platforms affected

[ ] Azure DevOps
[X] GitHub Actions - Standard Runners
[ ] GitHub Actions - Larger Runners

Runner images affected

[ ] Ubuntu 20.04
[ ] Ubuntu 22.04
[ ] macOS 11
[ ] macOS 12
[X] macOS 13
[ ] Windows Server 2019
[ ] Windows Server 2022

Image version and build link
macOS 13 version 20230509.4 ❌
Build: https://github.com/paulofierro/GoCommandTester/actions/runs/4950872824/jobs/8855065200
macOS 12 version 20230505.1 ✅
Build: https://github.com/paulofierro/GoCommandTester/actions/runs/4950874346
Is it regression?
yes
Expected behavior
Go is installed and available as it was in macOS 12.
Actual behavior
The go command is not found.
Repro steps
Create a workflow with a single step:
name: Go command test
on:
  push:

jobs:
  test:
    runs-on: macOS-13

    steps:
      - name: Print go version
        run: go version

Hello! We only provide go that can be installed via the actions/setup-go, i.e. these are the cached go versions, not the go version from homebrew or something else. So please use the setup task to setup your environment.
Feel free to reach us again if you have questions left!

@mikhailkoliada thanks, but I'm confused.
go version works fine in the macOS-12 runner, but not in the macOS-13 runner. In the same repo, without using homebrew or something else to install it.
To me that seems like a regression, but I guess I don't understand what I'm supposed to be doing differently here.

@paulofierro macOS-12 has go installed both from tool cache (the one used in the actions/setup-go) and go from homebred, these are different go distributions as per idea. macOS-13 does not have go installed from homebred yet and only has the tool cached one.


doc: requirements: update Sphinx and rtd

1.2.x is required when using Sphinx >= 6.0, otherwise certain features
like search are broken. Also update Sphinx to latest 6.x release, 6.2
as it contains some fixes.

@gmarull not on my computer, but pretty sure you need this, as I remember breaking search myself when migrating to Sphinx 6 locally https://github.com/kartben/zephyr/commit/b46c8d9b6a59fb51efaed999741e7e59d60db6ec#diff-e170e9a7d787c21095c6c11bb25f0f1ff0294a42a46d45ba6fb5ed794e457624R87

@gmarull not on my computer, but pretty sure you need this, as I remember breaking search myself when migrating to Sphinx 6 locally -- https://github.com/kartben/zephyr/blob/b46c8d9b6a59fb51efaed999741e7e59d60db6ec/doc/conf.py#L87

that was it, thanks.
ref. https://blog.readthedocs.com/sphinx6-upgrade/
Unbelievable that they still require jQuery...

Unbelievable that they still require jQuery...

😱


Stacks

Submission Title
Stacks
Submission Type
DS
Submission Description
A stack is an abstract data structure that contains a collection of elements.
Contributors

[x]  C++ -
[ ]  C -
[x]  Python -
[x] Java -
[ ] JavaScript -
[ ] TypeScript -
[ ] Golang -
[ ] Ruby -
[ ] Scala -

I would like to contribute implement stack in Python, let assign this task to me.

I would like to work on this, pls assign. I'll use python.

Kindly assign me this. I want to implement it in c++.

Created a PR for stacks implementation using python.
#132

Hi @ahampriyanshu , Please assign me this issue in C

thanks @GhostUser and @Mr-Vaibhav-Shukla :v:

Happy to contribute!

Thank you for the opportunity!


[BUG] Graphics Slider

Acknowledgement of preliminary instructions

[X] I have read the preliminary instructions, and I am certain that my problem has not already been addressed.
[X] I have thoroughly looked through the available Wiki articles and could not find a solution to my problem.
[X] I am using the latest version of Bloxstrap.
[ ] I did not answer truthfully to all the above checkboxes.

What problem did you encounter?
Im currently using the advance graphics quality selector and now its not working

Bloxstrap Log
N/A

UPDATE. https://github.com/pizzaboxer/bloxstrap/issues/2825

https://github.com/pizzaboxer/bloxstrap/issues/2825 someone else already reported on the topic of this


changed ports as per issue #3

Hi @coderbyheart request you to please review the changes for issue #3 and please comment on any further modifications. Thank you.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Thanks for replying @coderbyheart ; learnt this new thing today that we indeed need to keep the service alive on old ports for not breaking end user experience. Will work on it and get back to you for more learning and guidance. Thanks again.

Hi @coderbyheart , please review . Besides the changes made, i did not add old ports in the client.go file as , i guess, we want new clients to access the server on new ports, although, clients with old ports may still continue to access the service.
I have already learnt a thing or two from this work. Looking forward to your further guidance and mentoring on this. Thanks again !

Tests needed that demonstrate that old ports are still used. Client.go can be modified since it's only used for testing.

Hi @coderbyheart , thanks for being with me all along. Have added the tests for both old and new ports, please check and review and let me know for modifications / improvements.
Thanks !

Tests pass: https://github.com/NordicPlayground/nordic-developer-academy-coap/actions/runs/9125300745/job/25091193741


Allow for udev rules without devnode

Is your feature request related to a problem? Please describe.
I'm trying to get Akri to detect if a display is connected or not then schedule an X11 container to it. I found the correct rule but Akri doesn't seem to like it.
Is your feature request related to a way you would like Akri extended? Please describe.
Currently, when I use the rule SUBSYSTEM=="drm", ATTR{status}=="connected" I see the below in the logs:
find_devices - ignoring device with devpath "/devices/pci0000:00/0000:00:02.0/drm/card0/card0-DP-1" due to having no devnode
I would imagine this is because the drm subsystem is at /sys/class/drm/?
Describe the solution you'd like
I would like for udev rules without devnodes to be allowed in the udev discovery handler
Describe alternatives you've considered
I don't think there are alternatives?

I'm looking at the traceback from https://github.com/project-akri/akri/blob/4cad8e767a7d67e0338cdf5da8d8cee4edb16d92/discovery-handlers/udev/src/discovery_impl.rs#L157
I think there might be 2 options:

I build a custom discovery handler based on the udev handler that uses something other than devnode
If get_devnode(&device) returns None, you could optionally return a predictable name based on the KERNEL or SUBSYSTEM and the node name

Hi @agracey. I think option 1 is a good route and one that we have discussed before in Slack (https://kubernetes.slack.com/archives/C01D9L7QE8Z/p1662486545856829?thread_ts=1662022333.544559&cid=C01D9L7QE8Z). It seems that using syspath rather than devnode would be more ideal which would keep device ids more deterministic too.

I went with option 1: https://github.com/agracey/display-discovery and it works!
An example of it's use is here: https://gist.github.com/agracey/96cfa8c150687ec43b0ca18d42c05332
Should I close this? Or rename to reflect the previously discussed syspath change that you linked (for some reason slack isn't deep-linking for me so I can't actually see  the conversation ☹)?

@agracey looks like you are using devpath if devnode does not exist. Should we just always use devpath/syspath? Then rather than a custom DH we could just update the current udev one? Great that you were able to create a DH to get things going in the meantime.

🎉  Thank you!


Fix restart of containers that use a dockerfile

Description
Adjust timeouts and retry policy for request made from app host to DCP.
Fixes https://github.com/dotnet/aspire/issues/6392
Checklist

Is this feature complete?

[x] Yes. Ready to ship.
[ ] No. Follow-up changes expected.

Are you including unit tests for the changes and scenario tests if relevant?

[ ] Yes
[x] No (N/A)

Did you add public API?

[ ] Yes

If yes, did you have an API Review for it?

[ ] Yes
[ ] No

Did you add <remarks /> and <code /> elements on your triple slash comments?

[ ] Yes
[ ] No

[x] No

Does the change make any security assumptions or guarantees?

[ ] Yes

If yes, have you done a threat model and had a security review?

[ ] Yes
[ ] No

[x] No

Does the change require an update in our Aspire docs?

[ ] Yes

Link to aspire-docs issue:

[x] No

Microsoft Reviewers: Open in CodeFlow

This needs a backport

/backport to release/9.0


Feat: Enhance the navlinks hover animation for all navbars

Describe the feature
I want to enhance the hover for navlinks of navbar present on each page by adding new hover animations as the video given below
Add ScreenShots
https://github.com/user-attachments/assets/22ceaa50-9361-469b-b70e-81be481a2345
Record

[X] I agree to follow this project's Code of Conduct
[X] I'm a GSSOC'24 contributor
[X] I want to work on this issue

@Its-Aman-Yadav @Kalivarapubindusree please assign me this issue

Now it is fine with the hover effect and please make sure the alignment of the website  is good in mobile view also. @Asymtode712

@Kalivarapubindusree So I should implement the hover effect like the above video?

No it's already there u can improve alignments in the website so that UX will be good in the mobile view

See this pic if u want to contribute please make this in good manner


Unable to run DownloadBuildArtifacts task when Pipeline is set to run in container (self-hosted agent)

Required Information
Entering this information will route you directly to the right team and expedite traction.
Question, Bug, or Feature?
Type: Bug
Enter Task Name:  DownloadBuildArtifact
https://github.com/microsoft/azure-pipelines-tasks/blob/master/Tasks/DownloadBuildArtifactsV1/
Environment

Server - Azure Pipelines
Agent - Private (self-hosted)

OS: CentOS Linux release 7.9.2009 (Core)
Agent version: 2.200.2

Issue Description
I'm trying to execute a package build from inside a container running on my self-hosted Pipelines Agent. See my azure-pipelines.yml file:
trigger:
- master

pool:
  name: <agent-name>

container:
  image: <image>

steps:
- script: |
    pwd
    ls -lah
    whoami
    groups
  displayName: Some pre-checks to make sure I'm inside teh container

- task: DownloadBuildArtifacts@1
  inputs:
    buildType: 'specific'
    project: '<project>'
    pipeline: '<pipeline>
    specificBuildWithTriggering: true
    buildVersionToDownload: 'latest'
    downloadType: 'single'
    artifactName: '<artifact-name>'
    downloadPath: '$(System.ArtifactsDirectory)'
    #downloadPath: '/opt/<package>/'
  displayName: Download dependency

- script: |
    make
  displayName: 'Compile package with dependency'

As far as I've been able to identify, both of the tasks under script run just fine INSIDE the container, but DownloadBuildArtifact is running OUTSIDE of the container, on the machine where the pipelines agent is installed. This means the dependency is not available at compile-time.
Not sending any logs as they contain sensitive data, but as far as I've been able to test DownloadBuildArtifacts is downloading directly to the machine running the azure agent rather than the container (as was the expected behavior given I've specified the container tag for this pipeline run).

Hi @matman26, I currently investigate this issue, for now as a workaround you can use the CopyFiles task after the DownloadBuildArtifact task to pass the necessary files into the container.
Example of using the workaround:
container:
  image: <image>

steps:
...
- task: DownloadBuildArtifacts@1
  inputs:
    buildType: 'specific'
    project: '<project>'
    pipeline: '<pipeline>
    specificBuildWithTriggering: true
    buildVersionToDownload: 'latest'
    downloadType: 'single'
    artifactName: '<artifact-name>'
    downloadPath: '$(System.ArtifactsDirectory)'

- task: CopyFiles@2
  inputs:
    sourceFolder: '$(System.ArtifactsDirectory)/<artifact-name>'
    contents: '**'
    targetFolder: '$(Build.SourcesDirectory)'
...

Hi @matman26, as far as I see the DownloadBuildArtifactsV1 task works as expected. The script task runs in sources directory by default, if you want to use artifacts in the script task, you would put the artifact to $(Build.SourcesDirectory) or specify the custom working directory parameter of the script task. Also, for clarification, artifact are placed in a folder with the same name as the artifact.
Example:
steps:
- script: |
    pwd
    ls -lah
    whoami
    groups
  displayName: Some pre-checks to make sure I'm inside teh container

- task: DownloadBuildArtifacts@1
  inputs:
    buildType: 'specific'
    project: '<project>'
    pipeline: '<pipeline>
    specificBuildWithTriggering: true
    buildVersionToDownload: 'latest'
    downloadType: 'single'
    artifactName: '<artifact-name>'
    downloadPath: '$(Build.SourcesDirectory)'
  displayName: Download dependency

- script: |
    make
  displayName: 'Compile package with dependency'
  #workingDirectory: '$(System.ArtifactsDirectory)' # Instead of changing the dowloadPath


kanagawa.nvim is not working properly for Rust code after last update

Here is kanagawa.nvim at commit fc2e30820c73b0cf56bb0fb9e4a83dd8b90f9bf4 with nvim-treesitter at commit 4cccb6f494eb255b32a290d37c35ca12584c74d0 (before the removal of obsolete TS* highlight groups)

Here is a screenshot after the update, self and TODO: has no color . kanagawa.nvim at commit a6f8ea10900e8d891f9c93e0ed258f118010fb24 and nvim-treesitter at commit commit b9bcbf8d73b5a6c3e04922936b5fc500b436d4f5

I'm not sure if this is a problem with the kanagawa or with treesitter.

I am pretty sure this is due to rust-tools implementation of LSP semantic highlighting. They probably link tokens to tree sitter highlights, which are now missing.

The problem appears to be because I was using an old build of version 0.8, after I update neovim the issue was fixed.


NPE fix for step "these rows are present in the query result"

due to yesterday's github outage, the travis didn't got a notification, could you force push the commit (but with some change, so the commit will be different)


BACK-1600: Refactored logger newline concatenation…

…for increased clarity. Changed eslint "prefer-template" setting to result in a warning (instead of a failure)

@galniv ^^ ready for another look. It's not exactly related to 'style' refactoring but I also removed the unnecessary logging metadata in this branch.

LGTM


Verify WFI does not cause instruction fetches

Need to add functionality to TB to verify that upon encountering a WFI instruction, ibex does not perform any instruction fetches before any stimulus is driven by the testbench.

udi,
can i also ask that we check the core finishes all pending transactions
before asserting an idle indication to the outside world?
On Wed, Aug 28, 2019 at 11:57 AM udinator<EMAIL_ADDRESS>wrote:

Need to add functionality to TB to verify that upon encountering a WFI
instruction, ibex does not perform any instruction fetches before any
stimulus is driven by the testbench.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/lowRISC/ibex/issues/269?email_source=notifications&email_token=AAH2RSVSKMCEV376ZALX4B3QG2N6NA5CNFSM4IRLGH2KYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HH66FTQ,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAH2RSU6ITM5C7NGZKZUBXDQG2N6NANCNFSM4IRLGH2A
.

@tjaychen We don't signal "idle" to the outside world yet, that's #258

This feature has been tested, and core_sleep_o successfully indicates to the outside world that the core is sleeping and that all pending transactions have been finished and no more transactions are being fetched. This issue will be closed.


fix!: stop unbounded memory usage from query log

Resolves: #15433
When I converted prometheus to use slog in #14906, I update both the
QueryLogger interface, as well as how the log calls to the
QueryLogger were built up in promql.Engine.exec(). The backing
logger for the QueryLogger in the engine is a
util/logging.JSONFileLogger, and it's implementation of the With()
method updates the logger the logger in place with the new keyvals added
onto the underlying slog.Logger, which means they get inherited onto
everything after. All subsequent calls to With(), even in later
queries, would continue to then append on more and more keyvals for the
various params and fields built up in the logger. In turn, this causes
unbounded growth of the logger, leading to increased memory usage, and
in at least one report was the likely cause of an OOM kill. More
information can be found in the issue and the linked slack thread.
This commit does a few things:

It was referenced in feedback in #14906 that it would've been better
to not change the QueryLogger interface if possible, this PR
proposes changes that bring it closer to alignment with the pre-3.0
QueryLogger interface contract
reverts promql.Engine.exec()'s usage of the query logger to the
pattern of building up an array of args to pass at once to the end log
call. Avoiding the repetitious calls to .With() are what resolve the
issue with the logger growth/memory usage.
updates the scrape failure logger to use the update QueryLogger
methods in the contract.
updates tests accordingly
cleans up unused methods

Builds and passes tests successfully. Tested locally and confirmed I
could no longer reproduce the issue/it resolved the issue.
Signed-off-by: TJ Hoplock<EMAIL_ADDRESS>

cc: @SuperQ

Been running this for about an hour now, and its going significantly better than yesterday in #15433

Most recent pprof:
Time: Nov 21, 2024 at 4:54pm (UTC)
Showing nodes accounting for 4373.55MB, 92.00% of 4754.06MB total
Dropped 519 nodes (cum <= 23.77MB)
      flat  flat%   sum%        cum   cum%
 1397.27MB 29.39% 29.39%  1397.27MB 29.39%  github.com/prometheus/prometheus/model/labels.(*Builder).Labels
 1168.86MB 24.59% 53.98%  2508.12MB 52.76%  github.com/prometheus/prometheus/storage/remote.(*QueueManager).StoreSeries
  304.31MB  6.40% 60.38%   304.31MB  6.40%  github.com/klauspost/compress/flate.(*dictDecoder).init (inline)
     272MB  5.72% 66.10%      272MB  5.72%  github.com/klauspost/compress/zstd.(*fastBase).ensureHist (inline)
  227.05MB  4.78% 70.88%   227.05MB  4.78%  github.com/prometheus/prometheus/tsdb.(*seriesHashmap).set
  187.16MB  3.94% 74.81%   482.73MB 10.15%  github.com/prometheus/prometheus/tsdb.(*stripeSeries).getOrSet
  170.36MB  3.58% 78.40%   176.22MB  3.71%  github.com/prometheus/prometheus/tsdb.(*Head).gc
   76.52MB  1.61% 80.01%    76.52MB  1.61%  github.com/prometheus/prometheus/model/labels.(*ScratchBuilder).Labels
   73.47MB  1.55% 81.55%   152.06MB  3.20%  github.com/prometheus/prometheus/tsdb.(*headAppender).Append
   60.21MB  1.27% 82.82%    60.21MB  1.27%  github.com/klauspost/compress/zstd.encoderOptions.encoder
   58.01MB  1.22% 84.04%    68.51MB  1.44%  github.com/prometheus/prometheus/tsdb.newMemSeries (inline)
   46.50MB  0.98% 85.02%    46.50MB  0.98%  github.com/prometheus/prometheus/tsdb/chunkenc.NewXORChunk (inline)
   44.25MB  0.93% 85.95%    44.25MB  0.93%  github.com/prometheus/prometheus/scrape.(*scrapeCache).addRef (inline)
   41.24MB  0.87% 86.81%    43.24MB  0.91%  golang.org/x/net/trace.NewEventLog

Compare with the results in the aforementioned issue, but I think this fixes the issue. Great work TJ! Thanks for helping and getting a fix out so quickly.

More data for illustrative purposes while running @tjhop's branch, compared to upstream 3.0.0 (RAM spikes, disk usage increase)

Just working on understanding the whole scope of this bug:
Is this a similar (albeith smaller) problem?

Also scrapePool.restartLoops() calls setScrapeFailureLogger() calls l.With(), so that will grow every time config is reloaded?

Just working on understanding the whole scope of this bug: Is this a similar (albeit smaller) problem?

No, because that discovery config is using a proper slog.Logger, as opposed to a JSONFileLogger. The issue stems from this method:
https://github.com/prometheus/prometheus/blob/main/util/logging/file.go#L58

Also scrapePool.restartLoops() calls setScrapeFailureLogger() calls l.With(), so that will grow every time config is reloaded?

The scrape failure logger is indeed susceptible to the same issue, but thankfully it's current usage is fine (more info in slack here)
I'm working on making the JSONFileLogger safer to use in general as well, I'll open a separate PR when that's ready :+1:

This needs to go into a 3.0 patch release, I think.
Which is easiest if you change the base branch of the pr to release-3.0 and rebase.

Base branch changed, I'll clean up the change set soon

@bboreham rebased and repointed PR, lemme know if you see anything else :+1:


youtubemacropad

Ready to submit your project? Nice job! Hopefully you've read the submission guidelines. If not, make sure to read them first.
If you're unsure about the format, make sure to check out the Orpheuspad example and follow the format there.
If this PR isn't a submission, that's okay too! If there's anything about this PR template that seems confusing, let me know in #hackpad on the slack!

I approve
https://github.com/hackclub/hackpad/pull/80

Looks good to me! Get your hackpad here: https://forms.hackclub.com/hackpad
Thanks to @MrDomocle as well :)


Add types for EsriJSON*

npx tsc reveals things like this:
src/ol/format/EsriJSON.js(304,12): error TS2304: Cannot find name 'EsriJSONGeometry'.
src/ol/format/EsriJSON.js(404,27): error TS2304: Cannot find name 'EsriJSONPoint'.
src/ol/format/EsriJSON.js(452,13): error TS2304: Cannot find name 'EsriJSONPolyline'.
src/ol/format/EsriJSON.js(506,13): error TS2304: Cannot find name 'EsriJSONMultipoint'.
src/ol/format/EsriJSON.js(523,13): error TS2304: Cannot find name 'EsriJSONPolygon'.

We got rid of these types in #8273.  Similar to #8596, we need to add in @typedefs for those that we use.  In addition, I think we'll want to quote property access on these objects - but this should probably be confirmed.

@tschaub Would it be preferred to add TS type declarations for these in types, similar to the extensions I added for DOM API's? That should also allow dot property access, but we'll still want to confirm.


Add new meltano config settings

Migrated from GitLab: https://gitlab.com/meltano/meltano/-/issues/2646
Originally created by @tayloramurphy on 2021-03-15 15:14:37

add new meltano config <plugin> add <key> <value> and meltano config <plugin> remove <key> <value> commands for array settings like this _select
based on https://gitlab.com/meltano/meltano/-/issues/2638#note_528335663

View 1 previous comment from the original issue on GitLab

Still relevant but likely needs to be reworked

I don't have a strong opinion about this one, but I'll bring it up in the next OH.


Avoid possible nil pointer dereferences in consumergroup scheduling

Maybe fixes the panic seen in https://storage.googleapis.com/knative-prow/pr-logs/pull/knative-extensions_eventing-kafka-broker/3558/upgrade-tests_eventing-kafka-broker_main/1769808258517700608/artifacts/new-logs/kafka-controller-8565b5bfd7-ffw6g-controller-2024-03-18 19:58:12.178049569 +0000 UTC m=+537.848531106.log

Proposed Changes

Double check that variables which could be nil are not nil before accessing properties on them

/cc @pierDipi @Leo6Leo @creydr

/cherry-pick release-1.13

/cherry-pick release-1.12

/hold

/unhold
/cc @pierDipi

/retest-required

/retest

/retest-required


fix: increase right padding

Alaska Airlines Pull Request
Summary:
Per @sryckman, increase the right padding of the alert component to 1rem.
Before:

After:

Type of change:
Please delete options that are not relevant.

[x] Revision of an existing capability

Checklist:

[x] My update follows the CONTRIBUTING guidelines of this project
[x] I have performed a self-review of my own update

By submitting this Pull Request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Pull Requests will be evaluated by their quality of update and whether it is consistent with the goals and values of this project. Any submission is to be considered a conversation between the submitter and the maintainers of this project and may require changes to your submission.
Thank you for your submission!
-- Auro Design System Team

Commit has been reworded.

:tada: This PR is included in version 1.1.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Failed to build on MSYS2 UCRT64

$ make
[  0%] Building C object external/dwSampleFramework/external/glfw/src/CMakeFiles/glfw.dir/context.c.obj
[  1%] Building C object external/dwSampleFramework/external/glfw/src/CMakeFiles/glfw.dir/init.c.obj
[  1%] Building C object external/dwSampleFramework/external/glfw/src/CMakeFiles/glfw.dir/input.c.obj
[  1%] Building C object external/dwSampleFramework/external/glfw/src/CMakeFiles/glfw.dir/monitor.c.obj
[  2%] Building C object external/dwSampleFramework/external/glfw/src/CMakeFiles/glfw.dir/vulkan.c.obj
[  2%] Building C object external/dwSampleFramework/external/glfw/src/CMakeFiles/glfw.dir/window.c.obj
[  2%] Building C object external/dwSampleFramework/external/glfw/src/CMakeFiles/glfw.dir/win32_init.c.obj
[  3%] Building C object external/dwSampleFramework/external/glfw/src/CMakeFiles/glfw.dir/win32_joystick.c.obj
[  3%] Building C object external/dwSampleFramework/external/glfw/src/CMakeFiles/glfw.dir/win32_monitor.c.obj
[  3%] Building C object external/dwSampleFramework/external/glfw/src/CMakeFiles/glfw.dir/win32_time.c.obj
[  4%] Building C object external/dwSampleFramework/external/glfw/src/CMakeFiles/glfw.dir/win32_thread.c.obj
[  4%] Building C object external/dwSampleFramework/external/glfw/src/CMakeFiles/glfw.dir/win32_window.c.obj
[  4%] Building C object external/dwSampleFramework/external/glfw/src/CMakeFiles/glfw.dir/wgl_context.c.obj
[  5%] Building C object external/dwSampleFramework/external/glfw/src/CMakeFiles/glfw.dir/egl_context.c.obj
[  5%] Building C object external/dwSampleFramework/external/glfw/src/CMakeFiles/glfw.dir/osmesa_context.c.obj
[  6%] Linking C static library /C/msys64/home/Administrator/hybrid-rendering/lib/libglfw3.a
[  6%] Built target glfw
[  7%] Building CXX object external/dwSampleFramework/external/assimp/code/CMakeFiles/assimp.dir/Common/Assimp.cpp.obj
[  7%] Building CXX object external/dwSampleFramework/external/assimp/code/CMakeFiles/assimp.dir/CApi/CInterfaceIOWrapper.cpp.obj
[  7%] Building CXX object external/dwSampleFramework/external/assimp/code/CMakeFiles/assimp.dir/Common/BaseImporter.cpp.obj
[  8%] Building CXX object external/dwSampleFramework/external/assimp/code/CMakeFiles/assimp.dir/Common/BaseProcess.cpp.obj
In file included from C:/msys64/home/Administrator/hybrid-rendering/external/dwSampleFramework/external/assimp/include/assimp/GenericProperty.h:52,
                 from C:/msys64/home/Administrator/hybrid-rendering/external/dwSampleFramework/external/assimp/code/Common/BaseProcess.h:46,
                 from C:/msys64/home/Administrator/hybrid-rendering/external/dwSampleFramework/external/assimp/code/Common/BaseProcess.cpp:44:
C:/msys64/home/Administrator/hybrid-rendering/external/dwSampleFramework/external/assimp/include/assimp/Importer.hpp:505:16: error: 'exception_ptr' in namespace 'std' does not name a type
  505 |     const std::exception_ptr& GetException() const;
      |                ^~~~~~~~~~~~~
C:/msys64/home/Administrator/hybrid-rendering/external/dwSampleFramework/external/assimp/include/assimp/Importer.hpp:61:1: note: 'std::exception_ptr' is defined in header '<exception>'; did you forget to '#include <exception>'?
   60 | #include <assimp/types.h>
  +++ |+#include <exception>
   61 |
In file included from C:/msys64/home/Administrator/hybrid-rendering/external/dwSampleFramework/external/assimp/code/Common/BaseProcess.cpp:45:
C:/msys64/home/Administrator/hybrid-rendering/external/dwSampleFramework/external/assimp/code/Common/Importer.h:104:10: error: 'exception_ptr' in namespace 'std' does not name a type
  104 |     std::exception_ptr mException;
      |          ^~~~~~~~~~~~~
C:/msys64/home/Administrator/hybrid-rendering/external/dwSampleFramework/external/assimp/code/Common/Importer.h:49:1: note: 'std::exception_ptr' is defined in header '<exception>'; did you forget to '#include <exception>'?
   48 | #include <vector>
  +++ |+#include <exception>
   49 | #include <string>
C:/msys64/home/Administrator/hybrid-rendering/external/dwSampleFramework/external/assimp/code/Common/Importer.h: In constructor 'Assimp::ImporterPimpl::ImporterPimpl()':
C:/msys64/home/Administrator/hybrid-rendering/external/dwSampleFramework/external/assimp/code/Common/Importer.h:139:9: error: class 'Assimp::ImporterPimpl' does not have any field named 'mException'
  139 |         mException(),
      |         ^~~~~~~~~~
make[2]: *** [external/dwSampleFramework/external/assimp/code/CMakeFiles/assimp.dir/build.make:118: external/dwSampleFramework/external/assimp/code/CMakeFiles/assimp.dir/Common/BaseProcess.cpp.obj] Error 1
make[1]: *** [CMakeFiles/Makefile2:224: external/dwSampleFramework/external/assimp/code/CMakeFiles/assimp.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

I guess the external submodules are too old to build with modern C++ compilers?

Hi, only Visual Studio 2022 is tested at the moment. I have no plans to support anything else at the moment. Please feel free to submit an MR if you fix the issue.