Support for the ESP-IDF framework (Xtensa and RiscV arch)

Dear all,
This PR is implementing support for the ESP-IDF newlib-based framework, which is the open source SDK provided by Espressif for their MCU family (esp32, esp32s2, esp32c3 and all other forthcoming ones).
Note that this is the second PR on that topic. Approx. an year ago, @reitermarkus contributed an initial set of changes which are merged already.
Note also that this PR has a sibling PR against Rust's libStd which enables full STD support for the Espressif chipsets (that is, modulo process support as we are obviously talking about non-kernel bare metal platform here).
A short overview of the changes:

The original contribution of @reitermarkus is renamed from xtensa.rs to espidf.rs and all branching through this patch is no longer done based on target_arch = "xtensa", but based on target_os = "espidf" (this target_os value is to be used by the upcoming Rust targets for the ESP-IDF framework). The primary reason for this change is that branching for ESP-IDF based only on the architecture is no longer valid: the newer Espressif chips (esp32c3 and other upcoming ones) are based on the RISCV32IM(A)C architecture, so this patch now supports both Xtensa and RiscV32. Moreover, I would expect that - given the popularity of the riscv ISA - there will be other ports of newlib to riscv which will surely have the layout and sizes of the structures etc. different from the ESP-IDF framework.
The pthread structures had sizes which did not match what is used in the ESP-IDF. Ditto for the various *_INITIALIZER constants, which do not use 0x00, but 0xff sequences there
The BSD socket API on ESP-IDF is prefixed with lwip_. Rather than doing heavyweight proxying in Rust libStd, it is best to just address this here with a custom link_name, just like it had been done for MacOS and other systems.
Similar to the l4re case, the libc crate should NOT issue a link command  to the CRT lib for the ESP-IDF, due to the way the SDK is linked
Various other small fixes: primarily - declaration of standard APIs that are available in the ESP-IDF SDK

I should also admit that the patch has one little ESP-IDF-specific cheat:

Since the pthread support in ESP-IDF is still lacking RW-locks, I've implemented it as a temporary workaround by simply remapping (with "link_name") the pthread_rwlock_* symbols to their equivalent pthread_mutex_* symbols. While this implementation is suboptimal, it reduces significantly the PR changeset surface of the other PR which is against Rust libStd.

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @Amanieu (or someone else) soon.
Please see the contribution instructions for more information.

Since the pthread support in ESP-IDF is still lacking RW-locks, I've implemented it as a temporary workaround by simply remapping (with "link_name") the pthread_rwlock_* symbols to their equivalent pthread_mutex_* symbols. While this implementation is suboptimal, it reduces significantly the PR changeset surface of the other PR which is against Rust libStd.

This is not OK. pthread_rwlock is very different from pthread_mutex and libc should not expose one as the other.
Note that rwlocks can be implemented using mutexes and condition variables, so perhaps ESP-IDF could add pthread_rwlock support in the future.
In the meantime, pthread_rwlock should not be exposed by libc.
The rest of the changes look fine.

Note that rwlocks can be implemented using mutexes and condition variables, so perhaps ESP-IDF could add pthread_rwlock support in the future.

Indeed. That's what I was doing inside the libStd patch itself, by re-using and modifying a variant of this implementation. However it resulted in way too may changes all over the place (related to const fn and static rwlocks), so the cleanest approach is to just have native phtread_rwlock support - somehow.
Out of pure curiosity: are there any observable differences between rwlock and a mutex (that is, putting performance concerns with many readers aside, as that won't be the case on this tiny MCU which can only spin a handful of threads anyway)?
Condvars in rust seem to be deeply tied to mutexes, so that's one of the reasons I though assembling this cheat is not so un-shameful.

In the meantime, pthread_rwlock should not be exposed by libc.
The rest of the changes look fine.

Great. I'll push-force the removal of the rwlock cheat shortly.

Out of pure curiosity: are there any observable differences between rwlock and a mutex (that is, putting performance concerns with many readers aside, as that won't be the case on this tiny MCU which can only spin a handful of threads anyway)?

Actually there is, but it's quite subtle and specific to POSIX rwlocks. pthread_rwlock_rdlock has the requirement that multiple read locks from the same thread must not deadlock (at least with the default mutex type).
But even without this, I can imagine some contrived situations where forward progress can only happen after multiple threads have acquired a read lock and are waiting for some other condition, which would not work if rwlock was implemented as a mutex.

Indeed. That's what I was doing inside the libStd patch itself, by re-using and modifying a variant of this implementation. However it resulted in way too may changes all over the place (related to const fn and static rwlocks), so the cleanest approach is to just have native phtread_rwlock support - somehow.

Actually I would suggest implementing pthread_rwlock support in C in ESP-IDF itself using the algorithm I linked to. This would allow both C and Rust code to use rwlocks.
Alternatively, it might be acceptable to stub out rwlocks in the standard library as unimplemented for now.

Indeed. That's what I was doing inside the libStd patch itself, by re-using and modifying a variant of this implementation. However it resulted in way too may changes all over the place (related to const fn and static rwlocks), so the cleanest approach is to just have native phtread_rwlock support - somehow.

Actually I would suggest implementing pthread_rwlock support in C in ESP-IDF itself using the algorithm I linked to. This would allow both C and Rust code to use rwlocks.

We are aligned - that's what I'm saying too. Otherwise, the libStd changes become way too unwieldly.

Alternatively, it might be acceptable to stub out rwlocks in the standard library as unimplemented for now.

They are used within the sys\unix libStd implementation itself unfortunately (at multiple places, and growing).
So we do need them. As you say - ideally as a simple but pthread- compatible implementation.

I've force-pushed the removal of the pthread-rwlock hack.
When / if this MR gets merged, shall we raise the version of libc?
We need this to be done so that when / if the libStd MR gets merged, it gets these definitions (via an updated dependency to libc).

OK I'm happy with rwlock as it is, with the understanding that it will soon be implemented in ESP-IDF.
@bors r+

:pushpin: Commit 85210e6b2e226216431f80f5a0b3ebb329f98adb has been approved by Amanieu

:hourglass: Testing commit 85210e6b2e226216431f80f5a0b3ebb329f98adb with merge ca93b20b8435d238ceeb75fd09eaddee133c4147...

:sunny: Test successful - checks-actions, checks-cirrus-freebsd-11, checks-cirrus-freebsd-12, checks-cirrus-freebsd-13
Approved by: Amanieu
Pushing ca93b20b8435d238ceeb75fd09eaddee133c4147 to master...


ui: unable to input text on automation condition template type

Running
Home Assistant 0.76.0
docker, ubuntu 16.04
Attempting to use UI to to create new automations, with condition to look at mqtt (json) payload value using value template.
Whenever I select "template" for "condition", the "value template" text area box does not show any text when typed in. So I'm unable to enter any text into "value template" field. (See red circled area of screenshot.)
(I can create other types of automations OK, and with other types of conditions, text boxes are working ok.)

have tested with safari and chrome, both exhibit this behavior.

this also happens with "numeric state" condition type in the "value template" entry box:

I'm unable to enter any text in "value template"

/move to home-assistant/home-assistant-polymer


Forward roadmap.maplibre.org to maplibre.org/roadmap

Solved with a cname for roadmap.maplibre.org and a page rule in cloudflare.


Meet-up: Paris, France, dotJS 2018 conference, Friday 9th November

Hello to all the sharp users and contributors who will be in Paris attending dotJS 2018, the largest JavaScript conference in Europe.
I'll be there (with the LVMH Digital team) so if you would like to say hello in-person, ask a question or learn more about how you can contribute then please +1 this issue and subscribe for updates.
(Please also see #1416 for details of an NYC meet-up.)

https://twitter.com/lovell/status/1060826461916069888?s=19

Merci beaucoup to everyone that said hello. I learnt of one company that is currently processing around a billion images a week using sharp!


Unpacker please


Fix typo/wording for transitioning warning regarding supported apps

@MicrosoftGuyJFlo
Can you review the proposed changes?
Important: When the changes are ready for publication, adding a #sign-off comment is the best way to signal that the PR is ready for the review team to merge.
#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

#sign-off


Crashes at runtime.  Are the files up to date?

When I build and run the app, the tools and swatches don't load completely.  The action bar works, as does the logotool, but the pentools and swatches are broken.  No icons are visible, a small, white pentool is active by default and can be drawn with, but when I try to change the tool or swatch color the app crashes.  Are all the files here complete, and up-to-date?

There's a bunch of stuff that's in the "wear" branch that I never merged to master. Try that first.

Ok, I will try the "wear" branch. I should point out that the app version on the Play store works fine, so whatever files/branch that were used to build that is known to be good.

I tried loading in the changed files but I still get several errors on build.
In MarkersActivity
import org.dsandler.apps.markers.R; cannot be resolved
(SwatchButton) mLastColor).getColor() cannot be resolved
MediaClient  cannot be resolved
MarkersUtils  cannot be resolved
In Slate
import org.dsandler.apps.markers.R; cannot be resolved
 I thought this would build and run from the start since the app from the Play Store is working. Are there other files/libs missing from org.dsandler.apps.markers, or other merge problem?


Experiments planning

Research the most insightful ways to apply topic modeling to source code.

[x] Review existing literature: https://www.overleaf.com/7163137971dqhdpqmkdhyv
[x] Create the feature extraction and model training pipeline:

[x] Parse repositories at several points in history and extract features for each revision, following either the Diff or Hall model defined by Thomas (cf the literature review)
[x] Define a pipeline to compute a topic model over the complete features
[x] Post-process the obtained model to obtain topics for each file in each considered point in history
[ ] Explore the interaction of the diff model with document sparsity regularization and general topic model training by changing the corpus we train on: instead of keeping diff documents (both added and deleted) to train on, try to concatenate all add-documents of a given document into a consolidated document for training, and use the diff-documents only to compute topics for each revision in post-processing

[ ] Make sure the models trained with the Python ARTM client are equivalent to the ones trained with the ARTM CLI.
[ ] Experiment on https://github.com/pytorch/pytorch/ (20+ releases):

[ ] Find the best way to automatically label topics:

[x] Implement Mei's approach
[ ] Test different ways to define contexts for labeling: https://github.com/src-d/tm-experiments/pull/31#issuecomment-519007096, with the added option of using the max/mean/median count across revisions for each identifier as a context
[ ] Test monoword labels to mitigate the bigram mining/ordering problem
[ ] Test filtering label names to use only some type of identifiers (class names, function names, …), potentially boosting them compared to others when we score them using the trained topic model

[x] Research how to circumvent the need for a manually set number of topics
[ ] Measure topic evolution cohesion with existing changelogs
[x] Define the best evaluation schemes for production setups (which metrics, at which stages)

[ ] Scale up the experiments on http://github.com/tensorflow/tensorflow/ (80+ releases), and to commit granularity. Currently blocked by:

bblfsh python client OOMs
gitbase UAST cache issues/OOMs when parsing UASTs
diffs are not parsable in gitbase

[ ] Compute the topics at the user level, not document level, using either blame to determine ownership (easy to try, not so precise) or identifiers in diffs (a bit harder, very precise), and apply all the above points to the obtained topics (automatic labels, evolution, …)

@r0mainK I've added a few points on feature extraction/model building, feel free to edit/enhance/…!

Please note: we should use BigARTM for everything around topic modeling for multiple reasons:

Much more handles and parameters to explore
We are in good control of the code, the maintainers are quite nice and we do not have issues with fixing bugs
Related to (2), when something does not work, the maintainers are always helpful
The code is better than gensim

This is a research exploration, I don't think we should pinpoint the exact software we use, it might limit the amount of insight we get into the problem. Also, there are 2 big problems related to BigARTM for this experiment:

there is no non-parametric method implemented
the code is mainly documented in Russian, which is a problem for more than half of the team
But indeed it's a good and efficient project and we already planned to use it it was at suitable.

As Romain demonstrated, the non-parametric method failed miserably, so this is no longer a problem :smile:
I am mostly thinking about future use in production and trying to reduce the number of artificial rewrites. From what I saw in the leading R&D labs in the past, it is much beneficial to enforce such software constraints earlier than later. The alternatives should be comparable, however.
If we rely on a unique property of some specific library, and we require another one in production due to some research-irrelevant reasons, this may be a huge problem on a later stage. The difference between the research team at source{d} and a typical university research group is that our sole research purpose is to help the downstream folks commercialize.
Returning back from the metaphysical discussions: we should use a non-production-friendly library for a really, really serious reason. E.g. we are exercising PyTorch because GGNNs are an infinite pain to experiment with in Tensorflow.

As I mentioned in the previous comment we did plan to use BigARTM if it was feasible. But let's take the result that @r0mainK obtained: it was not obtainable quickly with BigARTM.
I agree with the conclusion of your second message :+1: (with the note that there are often serious reasons to depart from a lib in experiments).

from #18 by @r0mainK:

I actually rethought about it, I think it would probably make more sense to do the following:

take a much lower sparsity tolerance for doctopic, like 1%, which will reflect better reality,
distinguish that tolerance from the prob used to get non-superfluous topic, which should be higher

I think we should simply throw topics which are not at say 30 or even 50% dominance in at least one file. Also, it doesn't necessarily have to be one file, depending on the corpus, so we should add that as a parameter. By the way, using this config (over 50% in at least one file), I get only 17 topics with the perplexity metric. Going to do the changes in a new commit, and test for both metrics, tell me what you think of this idea ?

So in the end we talked about this on Slack, here is the jist:

agreed that 10% is a bit high when printing sparsity, settled on 5%
agreed on separating value used to compute score and value use to remove/select topics
given that the current strategy amount to remove topics that don't have at least probability of 0.1 in at least one document and apparent success of experiments, next PR will generalize this with harder threshold (0.5) and add arguments regarding min number of documents (either absolute or relative to the number of documents/delta-documents)
in the future we will add selection based on entropy, which requires some hyperparameter opt - probably a threshold which should be chosen from the entropies distribution after using the selectin reg

@r0mainK I updated the first comment according to https://github.com/src-d/minutes/pull/685/files, feel free to add/modify/delete any point :upside_down_face:

Update
Just a small update on the progress:

see issue #33 , I identified some more bugs (from the code, but also BigARTM), and implemented fixes/workarounds. Unfortunately, this meant removing perplexity score. As it is not completely straightforward to implement, I will not do it right now (especially since distinctness seems to be a better metric)
I added the consolidated corpus logic in create-bow and train-$, we can now start experimenting on this
I implemented the different context creation methods for labeling, we can now start experimenting on this
I started discussing with LA the need for diffs, it seems there already exists an implementation (that is not exposed) however it is not really suited for our needs. I proposed an alternative way of doing it which I expect is unfeasible given what I learnt from Babelfish in that conversation, however hopefully the API  design proposed will not.

Next steps

Finish multi-repo refactoring:

postprocess: 2 outputs currently, the doc membership (immediate for HALL, not so much for DIFF) and total word count, which could be move to create-bow (not much overhead, but duplicate), add typing
metrics: add multi-repo, maybe typing
visualize: add multi-repo, maybe typing

Once the new babelfish python-client is out, preprocess pytorch once more and start experimenting seriously - and also hopefully remove some of the restart logic.
Once the new Gitbase is out (with hard limit on cache), experiment to see if it can be used to make preprocessing way faster, possibly doing most of the work in it through SQL queries (less I/O client side)
A PR in Gitbase is currently under way to expose blame function. Once it is merged, we can start implementing user ownership evolution.


Feat(eos_designs): deprecate port_channel.short_esi under connected endpoints

Change Summary

Related Issue(s)
Fixes #
Component(s) name
arista.avd.<role-name>
Proposed changes

How to test

Checklist
User Checklist

N/A

Repository Checklist

[ ] My code has been rebased from devel before I start
[ ] I have read the CONTRIBUTING document.
[ ] My change requires a change to the documentation and documentation have been updated accordingly.
[ ] I have updated molecule CI testing accordingly. (check the box if not applicable)

is it the same as https://github.com/aristanetworks/ansible-avd/pull/2443 ?

is it the same as #2443 ?

Hi, PR #2443 is used for eos_cli_config_gen adding deprecation warning and #2526 is for removing port_channel.short_esi in eos_designs. https://github.com/aristanetworks/avd-internal/issues/86

is it the same as #2443 ?

Hi, PR #2443 is used for eos_cli_config_gen adding deprecation warning and #2526 is for removing port_channel.short_esi in eos_designs. aristanetworks/avd-internal#86

Indeed! apologies read too fast the PR title!

Collaborator

no problem...

This should only be a deprecation and not a removal. It is probably simpler to start fresh since we have done a lot of changes in eos_designs.


New command: m365 file copy

Usage
m365 file copy
Description
Copies a file to another location using the Microsoft Graph
Options

Option
Description

-u, --webUrl <webUrl>
The URL of the site where the file is located.

-s, --sourceUrl <sourceUrl>
Server-relative or absolute URL of the file.

-t, --targetUrl <targetUrl>
Server-relative or absolute URL of the location.

--newName [newName]
New name of the destination file.

--nameConflictBehavior [nameConflictBehavior]
Behavior when a document with the same name is already present at the destination. Possible values: fail, replace, rename. Default is fail.

Examples
Copies a file using the graph
m365 file copy--webUrl https://contoso.sharepoint.com/sites/project --sourceUrl "/sites/project/Shared Documents/Document.pdf" --targetUrl "/sites/IT/Shared Documents"

Default properties
No response
Additional Info

We currently have a m365 file add command using the Microsoft Graph this should be part of that command structure
Should use the new Drive API
Validate if we already have appropriate permissions for this command
Related to #4607

similar comment like in the move command. Do you think this could replace the m365 spo file copy command with an alias 🤔?
@pnp/cli-for-microsoft-365-maintainers could we double check this one 😊

@appieschot i think the related/similar issue was rechecked and is opened now. Maybe we could open this one as well?

Let's open it up 👍

Validate if we already have appropriate permissions for this command

Is the requirement to check if the user executing the command has the necessary permissions in both the source and target sites and to report if not, or, also grant the permission if missing?

Validate if we already have appropriate permissions for this command

Is the requirement to check if the user executing the command has the necessary permissions in both the source and target sites and to report if not, or, also grant the permission if missing?

I would say just checking without granting.

I am wondering if we really need this additional check or let the command throw 'access denied' error if permissions are missing. Like m365 file add/list and many other commands.

yep. I agree. Not really sure what @appieschot had in mind 🤔. But I say trying and if the permission is not sufficient throwing an error is good enough for me 👍
Any different opinion @pnp/cli-for-microsoft-365-maintainers ?

@Adam-it I added that one as a note to validate if the CLI has enough permissions to do so (if we have the appropriate scope already). We do not need to have that in the command itself, just need to be sure we can call the Graph Endpoint to do so :-). I just checked and we have those permissions already: https://learn.microsoft.com/en-us/graph/api/driveitem-copy?view=graph-rest-1.0&tabs=http

@Saurabh7019 I guess should be no clear 👍
@appieschot thanks for the clarification

Thank you for clarifying. Can I work on it?


Best pratice to modify p.dt inside a kernel ?

Hello all !
Within my custom kernel, I would like to change particle.dt because in some part of the advection, I want to refine or enlarge the sampling rate. I see that I can do particle.update_next_dt(new_dt) but :

does it apply to all the kernels ? (I have kernels = custom_kernel + AdvectionRK4 + periodicBC)
How do I set the file writing ? because my dt is not constant, so what about the ParticleFile outputdt  ? Is it possible to write all the data, when irregularly sampled ?

I tried combining particle.update_next_dt(new_dt) in my kernel, and leaving outputdt empty but I only get the first and last values of the simulation in my output file, butI think it's normal with the default outputdt=np.infty.
I you have any recommandation of example of varying, thanks very much.
Kevin

Dear @quai20. Yes, I think that what you are doing is fine, and that Parcels should be able to indeed adapt dt on the fly. outputs will be respected no matter how you change dt, I think. But there's only one way to find out I guess, by just trying and see what happens?


Fix broken console output and startup detection

Sends a status command to RCON once it has connected.

I have verified this is working.


8.0: Update What's New: SignalR stateful reconnect

Fixes #30675
Updates the SignalR stateful reconnect section of the What's New for 8.0.

SignalR seamless reconnect is renamed to SignalR stateful reconnect.
Code example updated.

Internal previews

📄 File
🔗 Preview link

aspnetcore/release-notes/aspnetcore-8.0.md
What's new in ASP.NET Core 8.0

I left the issue tracking link in for now at the bottom of the stateful reconnect section and will remove it in sync with the next release.


Split function bug

Hello, we would like to raise a bug encountered by one of our users related to the Split functionality. The issue happened while Azure Devops is crashing and then random stories/tasks were moved to a particular sprint. These were either old items or ones from the backlog. Multiple changes were also made automatically which are not initiated by the user, however, the history for the work items indicate that the action was done by the same user. Do you have an idea what may have caused this issue?
Thank you.

If you can reproduce the bug, we can schedule a web session .

If the issue persists, please inform me.


Removing model introspection and adding allow_calendar_dimensions

What is this PR?
This is a:

[x] documentation update
[x] bug fix with no breaking changes
[x] new functionality
[ ] a breaking change

All pull requests from community contributors should target the main branch (default).
Description & motivation
Related to issue #85 , this PR removes the default behavior of running queries against the data warehouse to validate the input. The only part where this introduced issues was in the use of calendar dimensions in the macro call.
To support this use case while improving performance, we added the allow_calendar_dimensions parameter into the macro that can be set to true if the individual wants to add calendar dimensions. The package will only run a filtered column relation query IF this is set to true and will now only run 1 instead of multiple per metric.
Example:
  select *
  from
  {{ metrics.calculate(
      metric('base_sum_metric'),
      grain='day',
      dimensions=['is_weekend'],
      allow_calendar_dimensions=true
  }}

Checklist

[ ] I have verified that these changes work locally on the following warehouses (Note: it's okay if you do not have access to all warehouses, this helps us understand what has been covered)

[ ] BigQuery
[X] Postgres
[ ] Redshift
[ ] Snowflake

[X] I have updated the README.md (if applicable)
[x] I have added tests & descriptions to my models (and macros if applicable)

Tenets to keep in mind

A metric value should be consistent everywhere that it is referenced
We prefer generalized metrics with many dimensions over specific metrics with few dimensions
It should be easier to use dbt’s metrics than it is to avoid them
Organization and discoverability are as important as precision
One-off models built to power metrics are an anti-pattern

@callum-mcdata this is groovy! When we last spoke about this, I was under the impression that this would be a project-level variable/config, not a metric-level argument! I don't love the UX here because metrics consumers will probably not know if they need or want to supply this argument -- it feels like much more of a consideration for metric builders, not metric readers!
Is there a path to doing this as a dbt project-level variable, or env var, or something like that? Or does it need to be an argument to the metric call given the way that things are working today?

This is a good point - in our current implementation there is nothing stopping it from being a project variable or environment variable. Because of current dbt_server limitations around environment variables, I'll not go down that route but we can change it in the future.
In my mind this makes most sense to live as part of the metric definition, in which case I'd need to add it to parsed.py, unparsed.py, and schemas.py.
metrics:
  - name: metric_name
    label: metric
    model: ref('model')
    allow_calendar_dimensions: true

All righty, this has now been changed based on feedback from @drewbanin and @joellabes .
Instead of this being a metric level config, we now support the custom_calendar_dimension_list in the project.yml. If this field is present, the dimensions within will be allowed in the calculate or develop query.


Do the embedding layer's layernorm parameters need to be reparameterized accordingly?

hi, According to appendix C in the paper, i am not sure whether the embedding layer's layernorm parameters need to be reparameterized accordingly, if not, it means that the attention_ratio_change in the first transformer layer is not used in the reparameterization process? By the way, does the bias term has been taken into consideration in all self-attention layers? Thank you very much!

The reparameterization aims to convert Admin back to the vanilla Transformer for a better compatibility, otherwise this process is not necessary.
To this end, the first attention_ratio_change needs to be token care of. For some models, a layer norm is added after the embedding, while for others not. If the layer norm is not available, then the embedding weight needs to be re-scaled accordingly (and weight sharing needs to be stopped).
For the bias term, I believe the deviation in Appendix C gives clear explanations on necessary transformations for bias terms.


C38408: Extra hyperlink at line 159

Hello, @DuncanmaMSFT, @BillWagner, @JasonWHowell, @Jemash,
Translator has reported possible source content issue.
Description of the source issue: Example URL should be escaped in order to avoid creating a link
Please review and merge the suggested proposed file change to avoid this error. If you make related fix in another PR, then share your PR number, so we can confirm and close this PR.
Many thanks in advance.

@v-makoud Thanks for the help to improve Microsoft Docs.
We could also use back tick around the url to show that URL string as a code, but its fine like you adjusted it.
#sign-off


Use Collection Resource to list all resources in a namespace

What would you like to be added?
Use Collection Resource to list all resources in a namespace
Why is this needed?
Use Collection Resource to list all resources in a namespace

Hi @27149chen ,  all resources  what is it reference to? Do you mean workloads filtered by namespace?

@ycsk02 The all resources are all types of resources (rbac resources, nodes, workloads etc...), should be a new Collection Resource.
However, custom resources may hold multiple versions, so we may only be able to support kubernetes built-in resources

I think we could add the new Collection Resource in collectionResources map, but maybe we can only support resources filtered by namespace through API query request, rather than kubectl command.
https://github.com/clusterpedia-io/clusterpedia/blob/562f7b78450d58823c67ee2c9f6697176d617fe3/pkg/storage/internalstorage/storage.go#L129
https://github.com/clusterpedia-io/clusterpedia/blob/562f7b78450d58823c67ee2c9f6697176d617fe3/pkg/apiserver/registry/clusterpedia/collectionresources/rest.go#L86-L91

/assign

TODO in 0.3.0: support onlyMetadata in the url query to return only metadata data

Use kubectl get collectionresources [workloads|kuberesources] or specify query (onlyMetadata) in the url to get only the metadata
$ kubectl get collectionresources kuberesources

$ kubectl get --raw "/apis/clusterpedia.io/v1beta1/collectionresources/kuberesources?onlyMetadata=true"

kubectl specifies that fetching yaml or json will return the full resource data, kubectl get collectionresources kuberesources -o yaml

collection resources supports most of the complex search, but for the time being only supports url query does not support search label . https://clusterpedia.io/docs/usage/search/
eg.
specify a single or some namespaces
$ kubectl get --raw "/apis/clusterpedia.io/v1beta1/collectionresources/kuberesources?onlyMetadata=true&namespaces=default"

$ kubectl get --raw "/apis/clusterpedia.io/v1beta1/collectionresources/kuberesources?onlyMetadata=true&namespaces=default,kube-system"

We should provide a client for collecting resources in client-go, what do you think? @calvin0327

I think so.

https://github.com/clusterpedia-io/client-go/issues/37


What/where is the expected output of ./build.sh 8.6.3 linux-x64 ?

I'm running this command with Docker installed and it seems to complete successfully, but I can't for the life of me find any output from this command anywhere.

Hello, when you say "output", are you referring to stdout logging or an output file (or both? or something else?).
The build script maps the current working directory $PWD into the containers so the resultant .tar.gz files end up there.
My best guess would be that an error is occurring.

I'll assume from your question opened against sharp that you got working.

Ah unfortunately I did not, at least on Linux. Works beautifully elsewhere.
I've not had time to go back to this yet and gather enough info to meaningfully move this issue forward. Thanks for all your help always @lovell.

@lovell thanks a lot for your help. I finally got around to looking back into this.
I'm running these commands from a Linux Mint 18.3 x64 machine.
If I run ./build.sh 8.6.3 I get a libvips-8.6.3-win32-x64.tar.gz in my current working directory, but if I run ./build.sh 8.6.3 linux-x64 I get nothing.
This is the full console output I get when running ./build.sh 8.6.3 linux-x64
wheezy: Pulling from library/debian
Digest: sha256:f3dc825cffd4d98f9af923a704f67b790c741a4f706296f0f77000ec580d3a1e
Status: Image is up to date for debian:wheezy
stretch: Pulling from library/debian
Digest: sha256:f1f61086ea01a72b30c7287adee8c929e569853de03b7c462a8ac75e0d0224c4
Status: Image is up to date for debian:stretch
Using default tag: latest
latest: Pulling from socialdefect/raspbian-jessie-core
b4c8471bb3fe: Already exists
Digest: sha256:cc5cc638f11fa93709a44f99b42395e90e2d5c4089e903698404c36401b3497b
Status: Image is up to date for socialdefect/raspbian-jessie-core:latest
Building linux-x64...
Sending build context to Docker daemon   2.56kB
Step 1/4 : FROM debian:wheezy
 ---> 2d79c539d0db
Step 2/4 : MAINTAINER Lovell Fuller<EMAIL_ADDRESS> ---> Using cache
 ---> 23c60565365c
Step 3/4 : RUN   echo "deb http://ftp.debian.org/debian wheezy-backports main" | tee /etc/apt/sources.list.d/wheezy-backports.list &&   apt-get update &&   apt-get install -y build-essential autoconf libtool nasm gtk-doc-tools texinfo gperf advancecomp &&   apt-get -t wheezy-backports install -y jq &&   curl https://sh.rustup.rs -sSf | sh -s -- -y
 ---> Using cache
 ---> 94cffe5f846c
Step 4/4 : ENV   PATH="/root/.cargo/bin:$PATH"   PLATFORM="linux-x64"   FLAGS="-O3"
 ---> Using cache
 ---> 4dbd9388a1b6
Successfully built 4dbd9388a1b6
Successfully tagged vips-dev-linux-x64:latest
glib version 2.55.1 has been superseded by 2.57.1
xml2 version 2.9.7 has been superseded by 2.9.8
gsf version 1.14.42 has been superseded by 1.14.43
jpeg version 1.5.3 has been superseded by 1.5.90
webp version 0.6.1 has been superseded by 1.0.0
gdkpixbuf version 2.36.11 has been superseded by 2.37.0
freetype version 2.9 has been superseded by 2.9.1
harfbuzz version 1.7.4 has been superseded by 1.8.4
pango version 1.41.0 has been superseded by 1.42.1
svg version 2.42.0 has been superseded by 2.43.2

I wasn't sure if using just linux was valid, so I tried it, ./build.sh 8.6.3 linux results in this output:
wheezy: Pulling from library/debian
Digest: sha256:f3dc825cffd4d98f9af923a704f67b790c741a4f706296f0f77000ec580d3a1e
Status: Image is up to date for debian:wheezy
stretch: Pulling from library/debian
Digest: sha256:f1f61086ea01a72b30c7287adee8c929e569853de03b7c462a8ac75e0d0224c4
Status: Image is up to date for debian:stretch
Using default tag: latest
latest: Pulling from socialdefect/raspbian-jessie-core
b4c8471bb3fe: Already exists
Digest: sha256:cc5cc638f11fa93709a44f99b42395e90e2d5c4089e903698404c36401b3497b
Status: Image is up to date for socialdefect/raspbian-jessie-core:latest
6614ccf119f081e86e2429cc0286d850a0659281341896c141a37e6c32df5eef  libvips-8.6.3-win32-x64.tar.gz

Running ./test-linux-x64.sh results in:
Testing debian:jessie...
jessie: Pulling from library/debian
Digest: sha256:a64a7d8ff7ff87edb78004ef0b159661546d2ddbd82772128b344c90cf8422ab
Status: Image is up to date for debian:jessie
./test-linux-x64.sh: 16: ./test-linux-x64.sh: cannot create packaging/debian:jessie.log: Directory nonexistent
debian:jessie fail
cat: 'packaging/debian:jessie.log': No such file or directory
Testing debian:stretch...
Error response from daemon: Get https://registry-1.docker.io/v2/library/debian/manifests/stretch: net/http: TLS handshake timeout
./test-linux-x64.sh: 16: ./test-linux-x64.sh: cannot create packaging/debian:stretch.log: Directory nonexistent
debian:stretch fail
cat: 'packaging/debian:stretch.log': No such file or directory
Testing ubuntu:trusty...
trusty: Pulling from library/ubuntu
Digest: sha256:71529e96591eb36a4100cd0cc5353ff1a2f4ee7a85011e3d3dd07cb5eb524a3e
Status: Image is up to date for ubuntu:trusty
./test-linux-x64.sh: 16: ./test-linux-x64.sh: cannot create packaging/ubuntu:trusty.log: Directory nonexistent
ubuntu:trusty fail
cat: 'packaging/ubuntu:trusty.log': No such file or directory
Testing ubuntu:xenial...
xenial: Pulling from library/ubuntu
Digest: sha256:14066a391d902c386d6164d44ade3460ba044abcdf8df88b0ff79a6f635be8d3
Status: Image is up to date for ubuntu:xenial
./test-linux-x64.sh: 16: ./test-linux-x64.sh: cannot create packaging/ubuntu:xenial.log: Directory nonexistent
ubuntu:xenial fail
cat: 'packaging/ubuntu:xenial.log': No such file or directory
Testing centos7...
7: Pulling from library/centos
Digest: sha256:b67d21dfe609ddacf404589e04631d90a342921e81c40aeaf3391f6717fa5322
Status: Image is up to date for centos:7
./test-linux-x64.sh: 25: ./test-linux-x64.sh: cannot create packaging/centos7.log: Directory nonexistent
centos7 fail
cat: packaging/centos7.log: No such file or directory
Testing archlinux...
latest: Pulling from pritunl/archlinux
Digest: sha256:c111496ffe9a106c6dd4d183bbacbb7136ed729008aa158a92dca37a976b4ec0
Status: Image is up to date for pritunl/archlinux:latest
./test-linux-x64.sh: 33: ./test-linux-x64.sh: cannot create packaging/archlinux.log: Directory nonexistent
archlinux fail
cat: packaging/archlinux.log: No such file or directory

Thanks very much for the tip @lovell! Going to continue to work on this today and hopefully finally crack the case.

@lovell and thoughts on this message: autoreconf: 'configure.ac' or 'configure.in' is required ? Below is the message with some preceding lines.
Making install in testbed
make[2]: Entering directory `/deps/lcms2/testbed'
make[3]: Entering directory `/deps/lcms2/testbed'
make[3]: Nothing to be done for `install-exec-am'.
make[3]: Nothing to be done for `install-data-am'.
make[3]: Leaving directory `/deps/lcms2/testbed'
make[2]: Leaving directory `/deps/lcms2/testbed'
make[2]: Entering directory `/deps/lcms2'
make[3]: Entering directory `/deps/lcms2'
 /bin/mkdir -p '/target/lib/pkgconfig'
 /usr/bin/install -c -m 644 lcms2.pc '/target/lib/pkgconfig'
make[3]: Leaving directory `/deps/lcms2'
make[2]: Leaving directory `/deps/lcms2'
make[1]: Leaving directory `/deps/lcms2'
autoreconf: 'configure.ac' or 'configure.in' is required

I can provide the full output of the command in a gist if needed.

Downloading the lcms2-2.9.tar.gz and extracting and building with the same instructions you used (minus --target and --host) builds without any issue on my local machine.
I also haphazardly threw a autoreconf -fiv into the lin.sh just after  cd ${DEPS}/lcms2 and just before: ./configure --host=${CHOST} --prefix=${TARGET} --enable-shared --disable-static --disable-dependency-tracking, but no luck with that either.

@lovell I know you're already offering a tremendous library for free and I am very appreciative for that and for all the help you've offered. I'm having a lot of trouble with this script and feel bad troubling you with it too given all that you've already put out for free use.
All that being said, is there any chance you'd consider support for prebuilt binaries for Linux which expect zlib 1.2.11? I'd love to be able to have my project https://recollectr.io support Linux and have gotten quite a few requests, but I've been stuck on packaging a version that works. Part of this has been that it seems like many distros skipped straight from zlib 1.2.8 to zlib 1.2.11.
The zlib homepage states, "Due to the bug fixes, any installations of 1.2.9 or 1.2.10 should be immediately replaced with 1.2.11."  1.2.11 has been the most recent version of zlib since January 15, 2017, so it looks to be very stable, and all new major Linux distributions include it with the exception of Debian, which does not include 1.2.9 in any case.

Distribution
** Release date**
Zlib version

Ubuntu 17.10
19 October 2017
1.2.11

openSUSE Leap 15
25 May 2018
1.2.11

Debian Stretch
17 June 2017
1.2.8

Fedora 26
11 July 2017
1.2.11

Mageia 6
16 July 2017
1.2.11

Debian looks like it won't include zlib 1.2.11 until Q2 or Q3 of 2019, but everything else is now using 1.2.11. I'd love to simply build my own, but so far I've been unable to do anything in this regard.
Bugs fixed since 1.2.9

Fix bug in deflate_stored() for zero-length input
Fix bug in gzwrite.c that produced corrupt gzip files
Fix deflate stored bug when pulling last block from window

Any help you can offer either with troubleshooting this script or supporting prebuilt binaries for zlib 1.2.11 would be greatly appreciated by myself and indirectly quite a few Linux users I'm sure. Sorry for the wall of text and thank you again @lovell.

The lin.sh script already compiles and provides zlib 1.2.11. If you want to use the binaries it creates with Electron on a Linux OS that only provides zlib 1.2.8 then you'll need to downgrade VERSION_ZLIB in the script to match that version (and remove the version_latest "zlib" check).

@lovell thanks for your reply. I know it should compile for zlib 1.2.11 and that's what I'd very much like to do, but the script fails for me with:
make[1]: Leaving directory `/deps/lcms2'
autoreconf: 'configure.ac' or 'configure.in' is required

I'll have to try locally to see what's up but might not have time to do this for a week or two.

Thanks @lovell! I appreciate it a lot

I thought perhaps this was a transient issue, so I came back to this to try again. Unfortunately I get a different issue now. I tried running ./build.sh 8.6.1 linux-x64 and I got:
make[3]: Entering directory `/deps/ffi'
 /bin/mkdir -p '/target/lib/../lib'
 /bin/bash ./libtool   --mode=install /usr/bin/install -c -s  libffi.la '/target/lib/../lib'
libtool: install: /usr/bin/install -c .libs/libffi.so.6.0.4 /target/lib/../lib/libffi.so.6.0.4
libtool: install: strip --strip-unneeded /target/lib/../lib/libffi.so.6.0.4
libtool: install: (cd /target/lib/../lib && { ln -s -f libffi.so.6.0.4 libffi.so.6 || { rm -f libffi.so.6 && ln -s libffi.so.6.0.4 libffi.so.6; }; })
libtool: install: (cd /target/lib/../lib && { ln -s -f libffi.so.6.0.4 libffi.so || { rm -f libffi.so && ln -s libffi.so.6.0.4 libffi.so; }; })
libtool: install: /usr/bin/install -c .libs/libffi.lai /target/lib/../lib/libffi.la
libtool: finish: PATH="/root/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/target/bin:/sbin" ldconfig -n /target/lib/../lib
----------------------------------------------------------------------
Libraries have been installed in:
   /target/lib/../lib

If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the `-LLIBDIR'
flag during linking and do at least one of the following:
   - add LIBDIR to the `LD_LIBRARY_PATH' environment variable
     during execution
   - add LIBDIR to the `LD_RUN_PATH' environment variable
     during linking
   - use the `-Wl,-rpath -Wl,LIBDIR' linker flag
   - have your system administrator add LIBDIR to `/etc/ld.so.conf'

See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------
 /bin/mkdir -p '/target/share/info'
 /usr/bin/install -c -m 644 ./doc/libffi.info '/target/share/info'
 install-info --info-dir='/target/share/info' '/target/share/info/libffi.info'
install-info: warning: nothing done since /usr/bin/install-info doesn't exist,
install-info: warning: you might want to install an info-browser package.
 /bin/mkdir -p '/target/lib/pkgconfig'
 /usr/bin/install -c -m 644 libffi.pc '/target/lib/pkgconfig'
make[3]: Leaving directory `/deps/ffi'
make[2]: Leaving directory `/deps/ffi'
make[1]: Leaving directory `/deps/ffi'
/packaging/build/lin.sh: 105: /packaging/build/lin.sh: ./configure: not found

Full console output: https://gist.github.com/Slapbox/062bb2c3fdd105b7e251a4837f491a2e
Any thoughts? Several related package versions were "superseded" when I came back to this, and I had to update their version numbers in the build script.

I've started updating these scripts ahead of the forthcoming libvips v8.7.0. The latest glib requires a newer automake than Debian Wheezy provides so for now you'll need to peg glib at v2.56.1.

Thanks very much for the update and the advice @lovell!

@lovell will the new version you're working on support<EMAIL_ADDRESS>I've revisited this issue several times in recent weeks, but I've not had any success unfortunately.
The closest I've gotten is getting back to the issue I had here after disabling the check for a newer version of glib.
According to this it seems like lcms might be a viable alternative to lcms2?

@lovell any update on the possibility of official support for<EMAIL_ADDRESS>? I know you've got a lot going on with this project, but I figured it wouldn't hurt to check in. I've tried a few more times to make headway with the script, without luck unfortunately.

Both the current v8.6.1 and forthcoming v8.7.0 bundles provide zlib v1.2.11.

"But is there any plan to offer a precompiled version for<EMAIL_ADDRESS>as you do with<EMAIL_ADDRESS>
The precompiled binaries have provided zlib v1.2.11 for at least a year.

Thanks for that @lovell. I have no idea how I've ended up so off the mark. I'll re-investigate to find out where I've been going wrong. Perhaps in the past I was trying to build with 1.2.11 on a system with 1.2.9 and now that I have a system with 1.2.11 I'm mixing up timelines and versions. Not sure, but that's really excellent news (at least it's news to me.)
Thank again very much for all that you do and for setting me right here.

Hey @lovell. I got a chance to give this another shot on Linux 18.3, which uses<EMAIL_ADDRESS>but I'm still running into the same issues as before with the prebuilts. Namely, that it looks for<EMAIL_ADDRESS>Uncaught Error: /lib/x86_64-linux-gnu/libz.so.1: version `ZLIB_1.2.9' not found (required by /home/myUser/projects/myApp/app/node_modules/sharp/build/Release/../../vendor/lib/libpng16.so.16)

I don't understand why it's still looking for 1.2.9. Can you shine any light on this for me?
My Sharp/vendor/versions.json looks like this:
{
  "cairo": "1.14.12",
  "croco": "0.6.12",
  "exif": "0.6.21",
  "expat": "2.2.5",
  "ffi": "3.2.1",
  "fontconfig": "2.12.6",
  "freetype": "2.9",
  "gdkpixbuf": "2.36.11",
  "gif": "5.1.4",
  "glib": "2.55.1",
  "gsf": "1.14.42",
  "harfbuzz": "1.7.4",
  "jpeg": "1.5.3",
  "lcms": "2.9-",
  "orc": "0.4.28",
  "pango": "1.41.0",
  "pixman": "0.34.0",
  "png": "1.6.34",
  "svg": "2.42.0",
  "tiff": "4.0.9-cda4b06",
  "vips": "8.6.1",
  "webp": "0.6.1",
  "xml": "2.9.7",
  "zlib": "1.2.11"
}

@lovell sorry for the lack of clarity. These latest messages have moved off the topic of running the sharp-libvips. My interest in this script was for getting Sharp working on Linux distros using<EMAIL_ADDRESS>but since there are prebuilt binaries which support this I've moved on to trying to get the relevant prebuilt binary working. I've followed up here in the Sharp repo:  https://github.com/lovell/sharp/issues/1095#issuecomment-422551755


Qrcode with logo or Icon

How to add logo or icon to the qr code can anyone share the code to show qrcode with logo into the picturebox
by using this code i can generate and show image into the picturebox but without logo in center.
using (QRCodeGenerator qrGenerator = new QRCodeGenerator()) using (QRCodeData qrCodeData = qrGenerator.CreateQrCode(Qrtext.ToString(), QRCodeGenerator.ECCLevel.Q)) using (PngByteQRCode qrCode = new PngByteQRCode(qrCodeData)) {     byte[] qrCodeImage = qrCode.GetGraphic(20);     using (MemoryStream ms = new MemoryStream(qrCodeImage))     {         pictureBox1.Image = Image.FromStream(ms);     } }

@aklegend I can only give you a solution that uses Bitmap (so it only works in QrCoder up to version 1.4.2 with System Drawing version 7.0.0. - see open issues)

That's not 100% correct. It works also with the latest QRCoder release (v1.6.0). Just just have to target your project either < net6.0 or especially target against net6.0-windows. The reason therefore is that all the icon/logo related parts rely on System.Drawing which was dropped by Microsoft from net6.0 for non-Windows targets. So if you aim your project towards net6.0-windows (or use older target) everything should work as expected. How to target: https://learn.microsoft.com/en-us/visualstudio/ide/visual-studio-multi-targeting-overview?view=vs-2022
Besides the QRCode rendering class there are also other renderers that support icons/logos, e.g. Base64QRCode and SvgQRCode. For more details check the documentation: https://github.com/codebude/QRCoder/wiki/Advanced-usage---QR-Code-renderers (And search for "logo" and "icon".)


Consider supporting githubuser0xFFFF/Qt-Advanced-Docking-System

Hello
The advanced docking system is really a great docking system for Qt. It uses an internal stylesheet to make it look like this:

When trying to use the stylesheet provided here without any changes, this happens:

What happens is that Qt-ADS uses an internal stylesheet to keep things in order.
If the internal stylesheet is disabled, this happens:

Notice how in particular the icons are wrong (like a double icon in one of them), and the tabs aren't clear, etc.

I guess it would require these things:
https://github.com/githubuser0xFFFF/Qt-Advanced-Docking-System/tree/master/src/stylesheets
To be included in the theme.
Anyway, thanks for this work! It's amazing!

The only issue with including these directly into the theme is this library is MIT licensed, which makes it useful for commercial usage, however, the above code is LGPLv2.1 licensed. I'll see if there are any fixes, or provide instructions on how to incorporate the above stylesheet internally, since obviously this is a very useful feature.

I'm not sure using the names of the stylesheet tags would exactly constitute an LGPL violation. You're not, in any way, using the library... But understand the caution, of course.

@ruilvo No, that should be fine, as long as I don't base anything off their source code. Their documentation also seems to publish the tag names, so this should be doable. As an extra precaution, you should be able to just provide the tag names and then I can just play with styles until everything works.
Is there a particular example you want me to base off of (like, a minimal, simple Qt project that I can ensure works before moving onto checking more complicated scenarios work)?

Those are the config tags, these are the stylesheets:
https://github.com/githubuser0xFFFF/Qt-Advanced-Docking-System/tree/master/src/stylesheets
I don't have a base application, but I think their examples are a great place to test.
Btw, they have the "Focus Highlighting feature" that they show in there as a separate file, but it seems it can be easily made into a single stylesheet:
https://github.com/githubuser0xFFFF/Qt-Advanced-Docking-System/blob/master/doc/user-guide.md#focushighlighting

@ruilvo Right, I just want to make sure I'm not looking through source files just as an extra precaution to find the tags. However, someone just telling me what all the tags are should be fine. I'll then just use a stock example.
For example, based on the documentation, I can see ads--CDockWidgetTab[focused="true"]. If there's any others not covered in the docs, that would be great to know.

Looking at the code, there's the problem that some of it has to do with the buttons and etc. And if you want to be clean with licensing, you can't copy that. And you can't also override from a "parent" QSS...
So I guess this can't be fixed from your end... I'll have to write my own.

At the very least, I should be able to provide "clean" install instructions or scripts that can help isolate this, so we can avoid licensing issues in the main code, but also make it easier to opt-in for compatibility with an LGPL project.

I'll see what I can, at the very least, something will be more ergonomic than everyone trying to recreate the same solution on their end.

Ok, this should be doable, and I'll start working on it. As long as I just touch the actual tags they provide, it should be fine. Their official bindings are BSD 3-Clause, and they provide a lot of API documentation, so I can just "play around" without touching the source code.
https://github.com/conda-forge/pyqtads-feedstock

Thank you for the enthusiastic feedback and willingness!! If there's something you need in the future regarding this topic, get in touch.
Thanks again!

Do you have a sample set of widgets for the dock widget? I'm reproducing some of the behavior, but I'm not very familiar with the Advanced Docking System, so these are the examples I get. Obviously, the tab is next to unreadable, but child QPushButtons seem to be correct.

See here for my sample code.

Actually, I have never used it with Python. I don't understand exactly what you mean with a set of widgets of ADS but it pretty much just the docking widget and the docking manager. I think you're nest to achieving the right thing. You already have got the tab buttons pretty ironed out. Sorry, I didn't understand what you want exactly.

I've got a fairly rough initial demo, and it seems to be pretty decent so far. This won't take more than a few days to finish (I've got actual work commitments, so obviously this is somewhat lower priority).
Please note there's a few minor style issues that are due to GTK issues with the conda Python environment (what I use to quickly mock up styles, even if I use C++ for real app development): it's due to GLIBCXX_3.4.29 being missing, causing issues with iris and swrast.
Currently, the floating maximize and minimize buttons look bad, and there's a few issues with the tabs (they don't style properly, and don't show the highlight tab at all), but it's coming a long way.

Whoa! Thanks! I'll test on C++ :D

Here's a few images, @ruilvo. Let me know if there are any issues, and I'll re-open it.

Oh @ruilvo, by default, support for ADS isn't enabled. To enable it, you can pass:
python configure.py --extensions=advanced-docking-system --styles=all

You can then use custom.qrc (and rename it as you wish) to use it with the ADS support.

It seems to be working :D
Thanks!

Let me know if anything doesn't work as expected, I'd be happy to fix any issues.

Have you noticed that when resizing, sometimes there's a funny glitch with the svg? Notice the minimize one:
https://i.gyazo.com/6c30b0bd3a702451ce7799c6e751cda6.mp4

Have you noticed that when resizing, sometimes there's a funny glitch with the svg? Notice the minimize one:
https://i.gyazo.com/6c30b0bd3a702451ce7799c6e751cda6.mp4

No, it doesn't replicate for me on Linux. When I get access to my Windows computer, I can try to fix it. I'm currently out-of-town, so it will be a week before I get a chance to.


Power button does not lock or unlock

I patched the most recent lineage-21.0-20241118-UNOFFICIAL-arm64_bvS.img.gz from https://sourceforge.net/projects/andyyan-gsi/files/, which seemed successful until it boots: the power button does not lock or unlock the screen. When I flashed the original image, without patching, it worked OK. How can I help debug this?
Edit: Tested with v1.4.1 and the power button works as expected, so must be something in v3.5.0, right?

Were you able to even reach the setup screen? I tried patching the same GSI and the screen just went blank after the initial boot screen.

@jaromeyer Yes, flashing and setup went OK. Everything else seemed to work as well, except for the power button


MeshCamera for point-like source

This issue to ask about the possibility to have a customized MeshCamera-like class for the detection of the power emitted by a point-like source.
This small source lies in a much bigger volume empty volume. This new MeshCamera could exploit the information about the position of the source to adjust the distribution of the launched rays to have it peaked in the direction of the source. A much smaller number of rays will be wasted.
Thank you very much in advance.

Thanks @matteomoscheni, this sounds like a needed feature, especially for applications with small point like sources. We already have the required functionality in the TargettedPixel() observer.
https://raysect.github.io/documentation/api_reference/optical/observers.html#raysect.optical.observer.nonimaging.targetted_pixel.TargettedPixel
This observer basically allows a single pixel to use targeted sampling towards small sources. So for your application you basically need an array/mesh version of this observer. It should be fairly easy to add based on our existing code. We will implement this feature as a new observer class for Raysect.

Dear @mattngc,
I would like to ask you few questions about the procedure followed by TargettedPixel.
(i) Given that the TargettedPixel only targets a given input list of primitives (say one single small source of radiation A), is it able to detect the possible presence of other "external" primitives? For example, if an absorbing wall is placed between the TargettedPixel and that source, is its presence taken into account?
(ii) If (i) is true, then I have this further doubt. If TargettedPixel targets the source primitive A but another source primitive B lies in between them, B should be detected as well ((i) is true). Then, TargettedPixel will directly target B only: is the source contribution B detected twice?
(iii) Let's say TargettedPixel is oriented in such a way that it can not see the source. For example, this can be the case for a TargettedPixel placed on the divertor region: the complicated geometry can lead to such a situation.
I got the error message 'Ray direction probability is zero. The target object extends beyond the pixel horizon.' (I suppose line 281 in 'targetted_pixel.pyx') and the simulation stopped. It should be that 'pdf <= 0' because of the orientation of the normal to the TargettedPixel with respect to the vector from a point of the TargettedPixel to the source position. The angle between these two vectors is greater than 90 degrees ('beyond the pixel horizon'), is that right?
If this is the case, then I should need a zero result for that TargettedPixel instead of the error. Maybe it is sufficient to assign a zero-weight if the condition 'pdf<=0' is met. I believe that 'pdf <= 0' is a sort of sanity-check command: if a source is placed, say, outside the tokamak chamber then it should not be detected.
I am sure you can shed some light on this.
Regards,
Matteo Moscheni

Hi @matteomoscheni,
(i) and (ii): The primitives in the target list are simply used to guide the selection of solid angle over which to sample. Each primitive specified is converted into a cone of solid angle as seen by the pixel. If you have a list of multiple primitives it leads to a list of multiple overlapping cones in solid angle space. For scenes with small light sources this is efficient because you don't waste time sampling rays over the whole hemisphere.
Once ray launch vectors have been chosen, the ray-tracing proceeds normally. So normal intersection rules apply. If something is in the way, occluding the trajectory to the target primitive, the ray will never reach that primitive and terminate normally. So the results should always turn out as you expect, it just improves the ray efficiency if you know your problem.
(iii) hmm... this is a physically occurring scenario and for a single pixel might make sense. If a user has specified a single target, but that target is behind the observing surface, then its impossible for the rays to target that source. So in this case an error is returned. But for a camera with many more pixels, some will see it, others won't. In that case it makes more sense to return 0 for rays that can't see the target. Let's think more about this, if it seems better to return 0 we could fix this issue.

Thank you very much for the clear explanation! I look forward to hearing from you soon about point (iii).


Refactor parsing in bevy_reflect path module

Objective

Follow up to #8887
The parsing code in bevy_reflect/src/path/mod.rs could also do with some cleanup

Solution

Create the parse.rs module, move all parsing code to this module
The parsing errors also now keep track of the whole parsed string, and are much more fine-grained

Detailed changes

Move PathParser to parse.rs submodule
Rename token_to_access to access_following (yep, goes from 132 lines to 16)
Move parsing tests into the parse.rs file

I found out that parsing in "u8 mode" is much better in terms of performance. And there is no reason to not do it (apart from maintainability). What I will do is remove the comment wrt alternative impl and open a performance PR with all the unsafe.

This is the optimized version I came up with:
https://gist.github.com/nicopap/9a1c04bf549914036ea5cc5d6e23269b

This is the optimized version I came up with:
https://gist.github.com/nicopap/9a1c04bf549914036ea5cc5d6e23269b

Are you planning to add this optimized version in this PR or opening a followup PR?


fix(pulumi): #423 check with namespaces

Your welcome, I had some difficulties with Pulumi and made this. Next, I hope to track Helm resources properly.


Set the dataLocation using absolute path in config_template.yml

Change the config_template.yml so it is possible to run it out-of-the-box from a different directory.

It is already possible to use environment variables in our config.yml file. This could fix our problem. @GernotMaier just need to put
dataLocation: "$GAMMASIM_DATA"
in your config file and set the GAMMASIM_DATA location before running it from another directory.
Note that GAMMASIM_PATH="(...)/gammasim-tools/data/"
The way I interpret environment variables is very simple. There can be only a single variable, starting with $. Something like $PATH/data would not work. I found this package that allows for more complicated patterns, even for setting default values. However, it is a small package and I am not sure whether it is a good idea in terms of maintainability. What do you think?

I don't think we should add a dependency for this, but I will let @GernotMaier voice his opinion.

Is there no simple way in Python to resolve a path like $PATH/data? It doesn't need to happen on the level of the yamlfile, as the indicated package does.

Is there no simple way in Python to resolve a path like $PATH/data? It doesn't need to happen on the level of the yamlfile, as the indicated package does.

Yes, I just found it.
We can use os.path.expandvar for that. I will implement it.


jest-resolver: figure out a mechanism for passing the target platform to Jest

Description
Figure out a mechanism for passing the target platform to Jest.
Your environment

Affected package: @rnx-kit/jest-resolver
Version: 0.1.0
Tested platforms:

[ ] Android
[ ] iOS
[ ] macOS
[ ] Windows

System information: n/a

Steps to reproduce
n/a

Closed by #433.


Release v1.25.1

Changelog:

Logging improvements for api errors #543

Hold on there's a pending PR we could aggregate into this. #545


ioredis doesn't emit error for ioredis:connection error: Error: Client network socket disconnected before secure TLS connection was established

What happened
We use ioredis within bullmq and observe errors doing message processing when the load on the instance is high. Our redis is configured as follows:
import Redis, { RedisOptions } from "ioredis";
import { env } from "../../env";
import { logger } from "../logger";

export const createNewRedisInstance = () => {
  const instance = new Redis(env.REDIS_CONNECTION_STRING, {
        maxRetriesPerRequest: null,
        enableAutoPipelining: env.REDIS_ENABLE_AUTO_PIPELINING === "true",
        retryStrategy: (times: number) => {
          // Retries forever. Waits at least 1s and at most 20s between retries.
          logger.warn(`Connection to redis lost. Retry attempt: ${times}`);
          return Math.max(Math.min(Math.exp(times), 20000), 1000);
        },
        reconnectOnError: (err) => {
          // Reconnects on READONLY errors and auto-retries the command.
          logger.warn(`Redis connection error: ${err.message}`);
          return err.message.includes("READONLY") ? 2 : false;
        },
      });

  instance?.on("error", (error) => {
    logger.error("Redis error", error);
  });

  return instance;
};

As we see in the logs below, eventually we see the following error:
"2024-10-29T07:18:09.787Z",,"""worker""","}"
"2024-10-29T07:18:09.787Z",,"""worker"""," localAddress: undefined"
"2024-10-29T07:18:09.787Z",,"""worker"""," port: 6379,"
"2024-10-29T07:18:09.787Z",,"""worker"""," host: 'master.prod-eu-redis-cluster.<id>.euw1.cache.amazonaws.com',"
"2024-10-29T07:18:09.787Z",,"""worker"""," path: undefined,"
"2024-10-29T07:18:09.787Z",,"""worker"""," code: 'ECONNRESET',"
"2024-10-29T07:18:09.787Z",,"""worker"""," at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {"
"2024-10-29T07:18:09.787Z",,"""worker"""," at endReadableNT (node:internal/streams/readable:1696:12)"
"2024-10-29T07:18:09.787Z",,"""worker"""," at TLSSocket.emit (node:events:531:35)"
"2024-10-29T07:18:09.787Z",,"""worker"""," at TLSSocket.onConnectEnd (node:_tls_wrap:1730:19)"
"2024-10-29T07:18:09.787Z",,"""worker""","2024-10-29T07:18:09.786Z ioredis:connection error: Error: Client network socket disconnected before secure TLS connection was established"
"2024-10-29T07:18:09.778Z",,"""worker""","Queue worker cloud-usage-metering-queue failed: Error: Client network socket disconnected before secure TLS connection was established"
"2024-10-29T07:18:09.778Z",,"""worker""","}"
"2024-10-29T07:18:09.778Z",,"""worker"""," localAddress: undefined"
"2024-10-29T07:18:09.778Z",,"""worker"""," port: 6379,"
"2024-10-29T07:18:09.778Z",,"""worker"""," host: 'master.prod-eu-redis-cluster..<id>.euw1.cache.amazonaws.com',"
"2024-10-29T07:18:09.778Z",,"""worker"""," path: undefined,"
"2024-10-29T07:18:09.778Z",,"""worker"""," code: 'ECONNRESET',"
"2024-10-29T07:18:09.778Z",,"""worker"""," at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {"
"2024-10-29T07:18:09.778Z",,"""worker"""," at endReadableNT (node:internal/streams/readable:1696:12)"
"2024-10-29T07:18:09.778Z",,"""worker"""," at TLSSocket.emit (node:events:531:35)"
"2024-10-29T07:18:09.778Z",,"""worker"""," at TLSSocket.onConnectEnd (node:_tls_wrap:1730:19)"
"2024-10-29T07:18:09.778Z",,"""worker""","2024-10-29T07:18:09.776Z ioredis:connection error: Error: Client network socket disconnected before secure TLS connection was established"

What I expect to happen
Given my code above, I would expect a log message Redis error for the on("error") handler and a Redis connection error warning from the reconnectOnError method. The only logs I observe from our application are the Connection to redis lost. Retry attempt logs, though.
How would I need to adjust the code to correctly log the ECONNRESET errors and then process them in my reconnectOnError?
Versions
<EMAIL_ADDRESS><EMAIL_ADDRESS>Logs
Date,Host,Service,Message
"2024-10-29T07:18:10.809Z","""unknown""","""worker""","2024-10-29T07:18:10.808Z ioredis:redis status[master.prod-eu-redis-cluster.<id>..euw1.cache.amazonaws.com:6379 (bull:Y2xvdWQtdXNhZ2UtbWV0ZXJpbmctcXVldWU=)]: reconnecting -> connecting"
"2024-10-29T07:18:10.806Z","""unknown""","""worker""","2024-10-29T07:18:10.806Z ioredis:redis status[master.prod-eu-redis-cluster.<id>..euw1.cache.amazonaws.com:6379 (bull:YmF0Y2gtZXhwb3J0LXF1ZXVl)]: reconnecting -> connecting"
"2024-10-29T07:18:10.806Z","""unknown""","""worker""","2024-10-29T07:18:10.806Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha('909ce352e2491c9082e6150ad29edff258ef9833,14,bull:legacy-ingestion-queue:wait,bull:legacy-ingestion-queue:active,bull:legacy-ingestion-queue:prioritized,bull:legacy-ingestion-queue:events,bull:legacy-i ... <REDACTED full-length=""770"">')"
"2024-10-29T07:18:10.806Z","""unknown""","""worker""","2024-10-29T07:18:10.805Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha('accb36ea979fc4cd20f4a56e3c4ad6df0ec0b14a,11,bull:ingestion-queue:wait,bull:ingestion-queue:active,bull:ingestion-queue:prioritized,bull:ingestion-queue:events,bull:ingestion-queue:stalled,bull:ingesti ... <REDACTED full-length=""472"">')"
"2024-10-29T07:18:10.800Z","""unknown""","""worker""","2024-10-29T07:18:10.800Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha('accb36ea979fc4cd20f4a56e3c4ad6df0ec0b14a,11,bull:trace-upsert:wait,bull:trace-upsert:active,bull:trace-upsert:prioritized,bull:trace-upsert:events,bull:trace-upsert:stalled,bull:trace-upsert:limiter,b ... <REDACTED full-length=""435"">')"
"2024-10-29T07:18:10.799Z","""unknown""","""worker""","2024-10-29T07:18:10.798Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha([ 'a8034c631460c6803c87966ce881636246de2398', '1', 'bull:legacy-ingestion-queue:', 'wait', 'paused', 'delayed', 'prioritized', 'waiting-children' ])"
"2024-10-29T07:18:10.799Z","""unknown""","""worker""","2024-10-29T07:18:10.798Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha('accb36ea979fc4cd20f4a56e3c4ad6df0ec0b14a,11,bull:ingestion-queue:wait,bull:ingestion-queue:active,bull:ingestion-queue:prioritized,bull:ingestion-queue:events,bull:ingestion-queue:stalled,bull:ingesti ... <REDACTED full-length=""472"">')"
"2024-10-29T07:18:10.798Z","""unknown""","""worker""","2024-10-29T07:18:10.797Z ioredis:redis write command[<IP_ADDRESS>:6379 (bull:dHJhY2UtdXBzZXJ0)]: 0 -> bzpopmin([ 'bull:trace-upsert:marker', '5' ])"
"2024-10-29T07:18:10.796Z","""unknown""","""worker""","2024-10-29T07:18:10.795Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha('c95b6be40a2863dd03813001fa069bcd90a2c6c7,8,bull:trace-upsert:wait,bull:trace-upsert:paused,bull:trace-upsert:meta,bull:trace-upsert:id,bull:trace-upsert:completed,bull:trace-upsert:active,bull:trace-u ... <REDACTED full-length=""660"">')"
"2024-10-29T07:18:10.796Z","""unknown""","""worker""","2024-10-29T07:18:10.795Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha('c95b6be40a2863dd03813001fa069bcd90a2c6c7,8,bull:trace-upsert:wait,bull:trace-upsert:paused,bull:trace-upsert:meta,bull:trace-upsert:id,bull:trace-upsert:completed,bull:trace-upsert:active,bull:trace-u ... <REDACTED full-length=""660"">')"
"2024-10-29T07:18:10.795Z","""unknown""","""worker""","2024-10-29T07:18:10.794Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha('accb36ea979fc4cd20f4a56e3c4ad6df0ec0b14a,11,bull:ingestion-queue:wait,bull:ingestion-queue:active,bull:ingestion-queue:prioritized,bull:ingestion-queue:events,bull:ingestion-queue:stalled,bull:ingesti ... <REDACTED full-length=""472"">')"
"2024-10-29T07:18:10.783Z","""unknown""","""worker""","2024-10-29T07:18:10.783Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha('accb36ea979fc4cd20f4a56e3c4ad6df0ec0b14a,11,bull:trace-upsert:wait,bull:trace-upsert:active,bull:trace-upsert:prioritized,bull:trace-upsert:events,bull:trace-upsert:stalled,bull:trace-upsert:limiter,b ... <REDACTED full-length=""435"">')"
"2024-10-29T07:18:10.771Z","""unknown""","""worker""","2024-10-29T07:18:10.770Z ioredis:redis write command[<IP_ADDRESS>:6379 (bull:dHJhY2UtdXBzZXJ0)]: 0 -> client([ 'setname', 'bull:dHJhY2UtdXBzZXJ0' ])"
"2024-10-29T07:18:10.770Z","""unknown""","""worker""","2024-10-29T07:18:10.769Z ioredis:redis write command[<IP_ADDRESS>:6379 (bull:aW5nZXN0aW9uLXF1ZXVl)]: 0 -> client([ 'setname', 'bull:aW5nZXN0aW9uLXF1ZXVl' ])"
"2024-10-29T07:18:10.769Z",,"""worker""","2024-10-29T07:18:10.768Z ioredis:connection set the connection name [bull:aW5nZXN0aW9uLXF1ZXVl]"
"2024-10-29T07:18:10.769Z","""unknown""","""worker""","2024-10-29T07:18:10.768Z ioredis:redis status[<IP_ADDRESS>:6379 (bull:aW5nZXN0aW9uLXF1ZXVl)]: connect -> ready"
"2024-10-29T07:18:10.769Z","""unknown""","""worker""","2024-10-29T07:18:10.769Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha('accb36ea979fc4cd20f4a56e3c4ad6df0ec0b14a,11,bull:ingestion-queue:wait,bull:ingestion-queue:active,bull:ingestion-queue:prioritized,bull:ingestion-queue:events,bull:ingestion-queue:stalled,bull:ingesti ... <REDACTED full-length=""472"">')"
"2024-10-29T07:18:10.768Z",,"""worker""","2024-10-29T07:18:10.767Z ioredis:connection send 1 commands in offline queue"
"2024-10-29T07:18:10.768Z",,"""worker""","2024-10-29T07:18:10.767Z ioredis:connection set the connection name [bull:dHJhY2UtdXBzZXJ0]"
"2024-10-29T07:18:10.768Z","""unknown""","""worker""","2024-10-29T07:18:10.767Z ioredis:redis status[<IP_ADDRESS>:6379 (bull:dHJhY2UtdXBzZXJ0)]: connect -> ready"
"2024-10-29T07:18:10.768Z","""unknown""","""worker""","2024-10-29T07:18:10.768Z ioredis:redis write command[<IP_ADDRESS>:6379 (bull:dHJhY2UtdXBzZXJ0)]: 0 -> bzpopmin([ 'bull:trace-upsert:marker', '5' ])"
"2024-10-29T07:18:10.751Z",,"""worker""","2024-10-29T07:18:10.751Z ioredis:connection send 1 commands in offline queue"
"2024-10-29T07:18:10.751Z",,"""worker""","2024-10-29T07:18:10.750Z ioredis:connection resend 0 unfulfilled commands"
"2024-10-29T07:18:10.751Z",,"""worker""","2024-10-29T07:18:10.750Z ioredis:connection set the connection name [bull:ZXZhbHVhdGlvbi1leGVjdXRpb24tcXVldWU=]"
"2024-10-29T07:18:10.751Z","""unknown""","""worker""","2024-10-29T07:18:10.751Z ioredis:redis write command[<IP_ADDRESS>:6379 (bull:ZXZhbHVhdGlvbi1leGVjdXRpb24tcXVldWU=)]: 0 -> bzpopmin([ 'bull:evaluation-execution-queue:marker', '2.514' ])"
"2024-10-29T07:18:10.751Z","""unknown""","""worker""","2024-10-29T07:18:10.750Z ioredis:redis status[<IP_ADDRESS>:6379 (bull:ZXZhbHVhdGlvbi1leGVjdXRpb24tcXVldWU=)]: connect -> ready"
"2024-10-29T07:18:10.745Z","""unknown""","""worker""","2024-10-29T07:18:10.744Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha('909ce352e2491c9082e6150ad29edff258ef9833,14,bull:legacy-ingestion-queue:wait,bull:legacy-ingestion-queue:active,bull:legacy-ingestion-queue:prioritized,bull:legacy-ingestion-queue:events,bull:legacy-i ... <REDACTED full-length=""770"">')"
"2024-10-29T07:18:10.745Z","""unknown""","""worker""","2024-10-29T07:18:10.744Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha('accb36ea979fc4cd20f4a56e3c4ad6df0ec0b14a,11,bull:ingestion-queue:wait,bull:ingestion-queue:active,bull:ingestion-queue:prioritized,bull:ingestion-queue:events,bull:ingestion-queue:stalled,bull:ingesti ... <REDACTED full-length=""472"">')"
"2024-10-29T07:18:10.744Z","""unknown""","""worker""","2024-10-29T07:18:10.743Z ioredis:redis write command[<IP_ADDRESS>:6379 (bull:dHJhY2UtdXBzZXJ0)]: 0 -> info([])"
"2024-10-29T07:18:10.743Z","""unknown""","""worker""","2024-10-29T07:18:10.743Z ioredis:redis write command[<IP_ADDRESS>:6379 (bull:dHJhY2UtdXBzZXJ0)]: 0 -> auth([ 'default', '<pw>' ])"
"2024-10-29T07:18:10.743Z","""unknown""","""worker""","2024-10-29T07:18:10.743Z ioredis:redis status[<IP_ADDRESS>:6379 (bull:dHJhY2UtdXBzZXJ0)]: connecting -> connect"
"2024-10-29T07:18:10.734Z","""unknown""","""worker""","2024-10-29T07:18:10.733Z ioredis:redis write command[<IP_ADDRESS>:6379 (bull:aW5nZXN0aW9uLXF1ZXVl)]: 0 -> info([])"
"2024-10-29T07:18:10.734Z","""unknown""","""worker""","2024-10-29T07:18:10.733Z ioredis:redis write command[<IP_ADDRESS>:6379 (bull:aW5nZXN0aW9uLXF1ZXVl)]: 0 -> auth([ 'default', '<pw>' ])"
"2024-10-29T07:18:10.733Z","""unknown""","""worker""","2024-10-29T07:18:10.733Z ioredis:redis status[<IP_ADDRESS>:6379 (bull:aW5nZXN0aW9uLXF1ZXVl)]: connecting -> connect"
"2024-10-29T07:18:10.730Z","""unknown""","""worker""","2024-10-29T07:18:10.729Z ioredis:redis write command[<IP_ADDRESS>:6379 (bull:ZXZhbHVhdGlvbi1leGVjdXRpb24tcXVldWU=)]: 0 -> auth([ 'default', '<pw>' ])"
"2024-10-29T07:18:10.730Z","""unknown""","""worker""","2024-10-29T07:18:10.729Z ioredis:redis status[<IP_ADDRESS>:6379 (bull:ZXZhbHVhdGlvbi1leGVjdXRpb24tcXVldWU=)]: connecting -> connect"
"2024-10-29T07:18:10.730Z","""unknown""","""worker""","2024-10-29T07:18:10.730Z ioredis:redis write command[<IP_ADDRESS>:6379 (bull:ZXZhbHVhdGlvbi1leGVjdXRpb24tcXVldWU=)]: 0 -> info([])"
"2024-10-29T07:18:10.726Z","""unknown""","""worker""","2024-10-29T07:18:10.726Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha('909ce352e2491c9082e6150ad29edff258ef9833,14,bull:trace-upsert:wait,bull:trace-upsert:active,bull:trace-upsert:prioritized,bull:trace-upsert:events,bull:trace-upsert:stalled,bull:trace-upsert:limiter,b ... <REDACTED full-length=""619"">')"
"2024-10-29T07:18:10.715Z","""unknown""","""worker""","2024-10-29T07:18:10.714Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha('accb36ea979fc4cd20f4a56e3c4ad6df0ec0b14a,11,bull:ingestion-queue:wait,bull:ingestion-queue:active,bull:ingestion-queue:prioritized,bull:ingestion-queue:events,bull:ingestion-queue:stalled,bull:ingesti ... <REDACTED full-length=""472"">')"
"2024-10-29T07:18:10.715Z","""unknown""","""worker""","2024-10-29T07:18:10.715Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha([ 'a8034c631460c6803c87966ce881636246de2398', '1', 'bull:legacy-ingestion-queue:', 'wait', 'paused', 'delayed', 'prioritized', 'waiting-children' ])"
"2024-10-29T07:18:10.714Z",,"""worker""","Processing legacy ingestion"
"2024-10-29T07:18:10.711Z","""unknown""","""worker""","2024-10-29T07:18:10.710Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha([ 'a8034c631460c6803c87966ce881636246de2398', '1', 'bull:trace-upsert:', 'wait', 'paused', 'delayed', 'prioritized', 'waiting-children' ])"
"2024-10-29T07:18:10.695Z","""unknown""","""worker""","2024-10-29T07:18:10.694Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha('909ce352e2491c9082e6150ad29edff258ef9833,14,bull:ingestion-queue:wait,bull:ingestion-queue:active,bull:ingestion-queue:prioritized,bull:ingestion-queue:events,bull:ingestion-queue:stalled,bull:ingesti ... <REDACTED full-length=""665"">')"
"2024-10-29T07:18:10.695Z","""unknown""","""worker""","2024-10-29T07:18:10.695Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha('909ce352e2491c9082e6150ad29edff258ef9833,14,bull:legacy-ingestion-queue:wait,bull:legacy-ingestion-queue:active,bull:legacy-ingestion-queue:prioritized,bull:legacy-ingestion-queue:events,bull:legacy-i ... <REDACTED full-length=""770"">')"
"2024-10-29T07:18:10.694Z","""unknown""","""worker""","2024-10-29T07:18:10.694Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha('accb36ea979fc4cd20f4a56e3c4ad6df0ec0b14a,11,bull:ingestion-queue:wait,bull:ingestion-queue:active,bull:ingestion-queue:prioritized,bull:ingestion-queue:events,bull:ingestion-queue:stalled,bull:ingesti ... <REDACTED full-length=""472"">')"
"2024-10-29T07:18:10.690Z",,"""worker""","Processing legacy ingestion"
"2024-10-29T07:18:10.681Z",,"""worker""","Processing legacy ingestion"
"2024-10-29T07:18:10.600Z","""unknown""","""worker""","2024-10-29T07:18:10.600Z ioredis:redis status[<IP_ADDRESS>:6379 (bull:dHJhY2UtdXBzZXJ0)]: reconnecting -> connecting"
"2024-10-29T07:18:10.599Z","""unknown""","""worker""","2024-10-29T07:18:10.598Z ioredis:redis status[<IP_ADDRESS>:6379 (bull:aW5nZXN0aW9uLXF1ZXVl)]: reconnecting -> connecting"
"2024-10-29T07:18:10.597Z","""unknown""","""worker""","2024-10-29T07:18:10.596Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha([ 'a8034c631460c6803c87966ce881636246de2398', '1', 'bull:ingestion-queue:', 'wait', 'paused', 'delayed', 'prioritized', 'waiting-children' ])"
"2024-10-29T07:18:10.597Z","""unknown""","""worker""","2024-10-29T07:18:10.596Z ioredis:redis status[<IP_ADDRESS>:6379 (bull:ZXZhbHVhdGlvbi1leGVjdXRpb24tcXVldWU=)]: reconnecting -> connecting"
"2024-10-29T07:18:10.587Z",,"""worker""","Processing legacy ingestion"
"2024-10-29T07:18:10.285Z","""unknown""","""worker""","2024-10-29T07:18:10.284Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha([ 'a8034c631460c6803c87966ce881636246de2398', '1', 'bull:legacy-ingestion-queue:', 'wait', 'paused', 'delayed', 'prioritized', 'waiting-children' ])"
"2024-10-29T07:18:10.284Z","""unknown""","""worker""","2024-10-29T07:18:10.284Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha('909ce352e2491c9082e6150ad29edff258ef9833,14,bull:ingestion-queue:wait,bull:ingestion-queue:active,bull:ingestion-queue:prioritized,bull:ingestion-queue:events,bull:ingestion-queue:stalled,bull:ingesti ... <REDACTED full-length=""665"">')"
"2024-10-29T07:18:10.007Z",,"""worker""","Processing legacy ingestion"
"2024-10-29T07:18:10.003Z","""unknown""","""worker""","2024-10-29T07:18:10.003Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha('c95b6be40a2863dd03813001fa069bcd90a2c6c7,8,bull:trace-upsert:wait,bull:trace-upsert:paused,bull:trace-upsert:meta,bull:trace-upsert:id,bull:trace-upsert:completed,bull:trace-upsert:active,bull:trace-u ... <REDACTED full-length=""660"">')"
"2024-10-29T07:18:09.993Z","""unknown""","""worker""","2024-10-29T07:18:09.992Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha([ 'a8034c631460c6803c87966ce881636246de2398', '1', 'bull:ingestion-queue:', 'wait', 'paused', 'delayed', 'prioritized', 'waiting-children' ])"
"2024-10-29T07:18:09.992Z","""unknown""","""worker""","2024-10-29T07:18:09.991Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha([ 'a8034c631460c6803c87966ce881636246de2398', '1', 'bull:trace-upsert:', 'wait', 'paused', 'delayed', 'prioritized', 'waiting-children' ])"
"2024-10-29T07:18:09.992Z","""unknown""","""worker""","2024-10-29T07:18:09.992Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha('accb36ea979fc4cd20f4a56e3c4ad6df0ec0b14a,11,bull:ingestion-queue:wait,bull:ingestion-queue:active,bull:ingestion-queue:prioritized,bull:ingestion-queue:events,bull:ingestion-queue:stalled,bull:ingesti ... <REDACTED full-length=""472"">')"
"2024-10-29T07:18:09.833Z","""unknown""","""worker""","2024-10-29T07:18:09.833Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha('909ce352e2491c9082e6150ad29edff258ef9833,14,bull:trace-upsert:wait,bull:trace-upsert:active,bull:trace-upsert:prioritized,bull:trace-upsert:events,bull:trace-upsert:stalled,bull:trace-upsert:limiter,b ... <REDACTED full-length=""619"">')"
"2024-10-29T07:18:09.833Z","""unknown""","""worker""","2024-10-29T07:18:09.832Z ioredis:redis queue command[<IP_ADDRESS>:6379 (bull:dHJhY2UtdXBzZXJ0)]: 0 -> bzpopmin([ 'bull:trace-upsert:marker', '5' ])"
"2024-10-29T07:18:09.817Z","""unknown""","""worker""","2024-10-29T07:18:09.816Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha('909ce352e2491c9082e6150ad29edff258ef9833,14,bull:legacy-ingestion-queue:wait,bull:legacy-ingestion-queue:active,bull:legacy-ingestion-queue:prioritized,bull:legacy-ingestion-queue:events,bull:legacy-i ... <REDACTED full-length=""770"">')"
"2024-10-29T07:18:09.804Z",,"""worker""","2024-10-29T07:18:09.803Z ioredis:connection reconnect in 1000ms"
"2024-10-29T07:18:09.804Z","""unknown""","""worker""","2024-10-29T07:18:09.803Z ioredis:redis status[master.prod-eu-redis-cluster.<id>..euw1.cache.amazonaws.com:6379 (bull:Y2xvdWQtdXNhZ2UtbWV0ZXJpbmctcXVldWU=)]: close -> reconnecting"
"2024-10-29T07:18:09.803Z",,"""worker""","Connection to redis lost. Retry attempt: 2"
"2024-10-29T07:18:09.803Z","""unknown""","""worker""","2024-10-29T07:18:09.802Z ioredis:redis status[master.prod-eu-redis-cluster.<id>..euw1.cache.amazonaws.com:6379 (bull:YmF0Y2gtZXhwb3J0LXF1ZXVl)]: close -> reconnecting"
"2024-10-29T07:18:09.802Z",,"""worker""","2024-10-29T07:18:09.802Z ioredis:connection reconnect in 1000ms"
"2024-10-29T07:18:09.802Z",,"""worker""","Connection to redis lost. Retry attempt: 2"
"2024-10-29T07:18:09.802Z","""unknown""","""worker""","2024-10-29T07:18:09.801Z ioredis:redis status[master.prod-eu-redis-cluster.<id>..euw1.cache.amazonaws.com:6379 (bull:YmF0Y2gtZXhwb3J0LXF1ZXVl)]: connecting -> close"
"2024-10-29T07:18:09.801Z","""unknown""","""worker""","2024-10-29T07:18:09.801Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha('909ce352e2491c9082e6150ad29edff258ef9833,14,bull:legacy-ingestion-queue:wait,bull:legacy-ingestion-queue:active,bull:legacy-ingestion-queue:prioritized,bull:legacy-ingestion-queue:events,bull:legacy-i ... <REDACTED full-length=""770"">')"
"2024-10-29T07:18:09.801Z","""unknown""","""worker""","2024-10-29T07:18:09.800Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha('909ce352e2491c9082e6150ad29edff258ef9833,14,bull:ingestion-queue:wait,bull:ingestion-queue:active,bull:ingestion-queue:prioritized,bull:ingestion-queue:events,bull:ingestion-queue:stalled,bull:ingesti ... <REDACTED full-length=""665"">')"
"2024-10-29T07:18:09.801Z","""unknown""","""worker""","2024-10-29T07:18:09.801Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha('accb36ea979fc4cd20f4a56e3c4ad6df0ec0b14a,11,bull:trace-upsert:wait,bull:trace-upsert:active,bull:trace-upsert:prioritized,bull:trace-upsert:events,bull:trace-upsert:stalled,bull:trace-upsert:limiter,b ... <REDACTED full-length=""435"">')"
"2024-10-29T07:18:09.800Z","""unknown""","""worker""","2024-10-29T07:18:09.800Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> ping([])"
"2024-10-29T07:18:09.800Z","""unknown""","""worker""","2024-10-29T07:18:09.799Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha([ 'a8034c631460c6803c87966ce881636246de2398', '1', 'bull:trace-upsert:', 'wait', 'paused', 'delayed', 'prioritized', 'waiting-children' ])"
"2024-10-29T07:18:09.800Z","""unknown""","""worker""","2024-10-29T07:18:09.800Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha('accb36ea979fc4cd20f4a56e3c4ad6df0ec0b14a,11,bull:ingestion-queue:wait,bull:ingestion-queue:active,bull:ingestion-queue:prioritized,bull:ingestion-queue:events,bull:ingestion-queue:stalled,bull:ingesti ... <REDACTED full-length=""472"">')"
"2024-10-29T07:18:09.798Z","""unknown""","""worker""","2024-10-29T07:18:09.797Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha([ 'a8034c631460c6803c87966ce881636246de2398', '1', 'bull:legacy-ingestion-queue:', 'wait', 'paused', 'delayed', 'prioritized', 'waiting-children' ])"
"2024-10-29T07:18:09.787Z",,"""worker""","Queue worker batch-export-queue failed: Error: Client network socket disconnected before secure TLS connection was established"
"2024-10-29T07:18:09.787Z",,"""worker""","}"
"2024-10-29T07:18:09.787Z",,"""worker"""," localAddress: undefined"
"2024-10-29T07:18:09.787Z",,"""worker"""," port: 6379,"
"2024-10-29T07:18:09.787Z",,"""worker"""," host: 'master.prod-eu-redis-cluster.<id>.euw1.cache.amazonaws.com',"
"2024-10-29T07:18:09.787Z",,"""worker"""," path: undefined,"
"2024-10-29T07:18:09.787Z",,"""worker"""," code: 'ECONNRESET',"
"2024-10-29T07:18:09.787Z",,"""worker"""," at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {"
"2024-10-29T07:18:09.787Z",,"""worker"""," at endReadableNT (node:internal/streams/readable:1696:12)"
"2024-10-29T07:18:09.787Z",,"""worker"""," at TLSSocket.emit (node:events:531:35)"
"2024-10-29T07:18:09.787Z",,"""worker"""," at TLSSocket.onConnectEnd (node:_tls_wrap:1730:19)"
"2024-10-29T07:18:09.787Z",,"""worker""","2024-10-29T07:18:09.786Z ioredis:connection error: Error: Client network socket disconnected before secure TLS connection was established"
"2024-10-29T07:18:09.778Z",,"""worker""","Queue worker cloud-usage-metering-queue failed: Error: Client network socket disconnected before secure TLS connection was established"
"2024-10-29T07:18:09.778Z",,"""worker""","}"
"2024-10-29T07:18:09.778Z",,"""worker"""," localAddress: undefined"
"2024-10-29T07:18:09.778Z",,"""worker"""," port: 6379,"
"2024-10-29T07:18:09.778Z",,"""worker"""," host: 'master.prod-eu-redis-cluster..<id>.euw1.cache.amazonaws.com',"
"2024-10-29T07:18:09.778Z",,"""worker"""," path: undefined,"
"2024-10-29T07:18:09.778Z",,"""worker"""," code: 'ECONNRESET',"
"2024-10-29T07:18:09.778Z",,"""worker"""," at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {"
"2024-10-29T07:18:09.778Z",,"""worker"""," at endReadableNT (node:internal/streams/readable:1696:12)"
"2024-10-29T07:18:09.778Z",,"""worker"""," at TLSSocket.emit (node:events:531:35)"
"2024-10-29T07:18:09.778Z",,"""worker"""," at TLSSocket.onConnectEnd (node:_tls_wrap:1730:19)"
"2024-10-29T07:18:09.778Z",,"""worker""","2024-10-29T07:18:09.776Z ioredis:connection error: Error: Client network socket disconnected before secure TLS connection was established"
"2024-10-29T07:18:09.709Z","""unknown""","""worker""","2024-10-29T07:18:09.709Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha('accb36ea979fc4cd20f4a56e3c4ad6df0ec0b14a,11,bull:trace-upsert:wait,bull:trace-upsert:active,bull:trace-upsert:prioritized,bull:trace-upsert:events,bull:trace-upsert:stalled,bull:trace-upsert:limiter,b ... <REDACTED full-length=""435"">')"
"2024-10-29T07:18:09.709Z","""unknown""","""worker""","2024-10-29T07:18:09.708Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha('accb36ea979fc4cd20f4a56e3c4ad6df0ec0b14a,11,bull:ingestion-queue:wait,bull:ingestion-queue:active,bull:ingestion-queue:prioritized,bull:ingestion-queue:events,bull:ingestion-queue:stalled,bull:ingesti ... <REDACTED full-length=""472"">')"
"2024-10-29T07:18:09.708Z","""unknown""","""worker""","2024-10-29T07:18:09.708Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha([ 'a8034c631460c6803c87966ce881636246de2398', '1', 'bull:ingestion-queue:', 'wait', 'paused', 'delayed', 'prioritized', 'waiting-children' ])"
"2024-10-29T07:18:09.693Z",,"""worker""","Processing ingestion event"
"2024-10-29T07:18:09.676Z",,"""worker""","Processing legacy ingestion"
"2024-10-29T07:18:09.528Z",,"""worker""","Connection to redis lost. Retry attempt: 1"
"2024-10-29T07:18:09.528Z",,"""worker""","2024-10-29T07:18:09.528Z ioredis:connection reconnect in 1000ms"
"2024-10-29T07:18:09.528Z",,"""worker""","2024-10-29T07:18:09.527Z ioredis:connection reconnect in 1000ms"
"2024-10-29T07:18:09.528Z","""unknown""","""worker""","2024-10-29T07:18:09.527Z ioredis:redis status[<IP_ADDRESS>:6379 (bull:aW5nZXN0aW9uLXF1ZXVl)]: close -> reconnecting"
"2024-10-29T07:18:09.528Z","""unknown""","""worker""","2024-10-29T07:18:09.528Z ioredis:redis status[<IP_ADDRESS>:6379 (bull:dHJhY2UtdXBzZXJ0)]: close -> reconnecting"
"2024-10-29T07:18:09.528Z","""unknown""","""worker""","2024-10-29T07:18:09.528Z ioredis:redis status[<IP_ADDRESS>:6379 (bull:dHJhY2UtdXBzZXJ0)]: ready -> close"
"2024-10-29T07:18:09.527Z",,"""worker""","Connection to redis lost. Retry attempt: 1"
"2024-10-29T07:18:09.527Z",,"""worker""","2024-10-29T07:18:09.526Z ioredis:connection reconnect in 1000ms"
"2024-10-29T07:18:09.527Z","""unknown""","""worker""","2024-10-29T07:18:09.526Z ioredis:redis status[<IP_ADDRESS>:6379 (bull:ZXZhbHVhdGlvbi1leGVjdXRpb24tcXVldWU=)]: ready -> close"
"2024-10-29T07:18:09.527Z","""unknown""","""worker""","2024-10-29T07:18:09.527Z ioredis:redis status[<IP_ADDRESS>:6379 (bull:aW5nZXN0aW9uLXF1ZXVl)]: ready -> close"
"2024-10-29T07:18:09.527Z","""unknown""","""worker""","2024-10-29T07:18:09.526Z ioredis:redis status[<IP_ADDRESS>:6379 (bull:ZXZhbHVhdGlvbi1leGVjdXRpb24tcXVldWU=)]: close -> reconnecting"
"2024-10-29T07:18:09.526Z",,"""worker""","Connection to redis lost. Retry attempt: 1"
"2024-10-29T07:18:09.524Z","""unknown""","""worker""","2024-10-29T07:18:09.524Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha('a2b97b03b7d21d69b152446117520ec099d22a84,9,bull:cloud-usage-metering-queue:stalled,bull:cloud-usage-metering-queue:wait,bull:cloud-usage-metering-queue:active,bull:cloud-usage-metering-queue:failed,bu ... <REDACTED full-length=""452"">')"
"2024-10-29T07:18:09.487Z","""unknown""","""worker""","2024-10-29T07:18:09.487Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha([ '99b6eb3d7a4777fb2271fd6cdf3ac5a290a4aac4', '2', 'bull:ingestion-queue:30470371:lock', 'bull:ingestion-queue:stalled', '7b8d9abf-b30f-463b-9ce3-3dfd7637d2ad:37478', '30000', '30470371' ])"
"2024-10-29T07:18:09.484Z","""unknown""","""worker""","2024-10-29T07:18:09.483Z ioredis:redis status[<IP_ADDRESS>:6379 (bull:YmF0Y2gtZXhwb3J0LXF1ZXVl)]: reconnecting -> connecting"
"2024-10-29T07:18:09.483Z","""unknown""","""worker""","2024-10-29T07:18:09.482Z ioredis:redis status[<IP_ADDRESS>:6379 (bull:Y2xvdWQtdXNhZ2UtbWV0ZXJpbmctcXVldWU=)]: reconnecting -> connecting"
"2024-10-29T07:18:00.918Z",,"""worker""","Processing legacy ingestion"
"2024-10-29T07:18:00.916Z","""unknown""","""worker""","2024-10-29T07:18:00.915Z ioredis:redis write command[<IP_ADDRESS>:6379 (bull:aW5nZXN0aW9uLXF1ZXVl)]: 0 -> bzpopmin([ 'bull:ingestion-queue:marker', '5' ])"
"2024-10-29T07:18:00.911Z",,"""worker""","Processing legacy ingestion"
"2024-10-29T07:18:00.897Z","""unknown""","""worker""","2024-10-29T07:18:00.896Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha([ 'a8034c631460c6803c87966ce881636246de2398', '1', 'bull:legacy-ingestion-queue:', 'wait', 'paused', 'delayed', 'prioritized', 'waiting-children' ])"
"2024-10-29T07:18:00.897Z","""unknown""","""worker""","2024-10-29T07:18:00.896Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha([ 'a8034c631460c6803c87966ce881636246de2398', '1', 'bull:legacy-ingestion-queue:', 'wait', 'paused', 'delayed', 'prioritized', 'waiting-children' ])"
"2024-10-29T07:18:00.897Z",,"""worker""","Processing ingestion event"
"2024-10-29T07:18:00.896Z","""unknown""","""worker""","2024-10-29T07:18:00.895Z ioredis:redis queue command[<IP_ADDRESS>:6379 (bull:Y2xvdWQtdXNhZ2UtbWV0ZXJpbmctcXVldWU=)]: 0 -> bzpopmin([ 'bull:cloud-usage-metering-queue:marker', '5' ])"
"2024-10-29T07:18:00.880Z","""unknown""","""worker""","2024-10-29T07:18:00.879Z ioredis:redis status[<IP_ADDRESS>:6379 (bull:YmF0Y2gtZXhwb3J0LXF1ZXVl)]: close -> reconnecting"
"2024-10-29T07:18:00.880Z",,"""worker""","2024-10-29T07:18:00.879Z ioredis:connection reconnect in 1000ms"
"2024-10-29T07:18:00.879Z","""unknown""","""worker""","2024-10-29T07:18:00.879Z ioredis:redis status[<IP_ADDRESS>:6379 (bull:YmF0Y2gtZXhwb3J0LXF1ZXVl)]: ready -> close"
"2024-10-29T07:18:00.879Z","""unknown""","""worker""","2024-10-29T07:18:00.879Z ioredis:redis status[<IP_ADDRESS>:6379 (bull:Y2xvdWQtdXNhZ2UtbWV0ZXJpbmctcXVldWU=)]: close -> reconnecting"
"2024-10-29T07:18:00.879Z","""unknown""","""worker""","2024-10-29T07:18:00.878Z ioredis:redis status[<IP_ADDRESS>:6379 (bull:Y2xvdWQtdXNhZ2UtbWV0ZXJpbmctcXVldWU=)]: ready -> close"
"2024-10-29T07:18:00.879Z",,"""worker""","Connection to redis lost. Retry attempt: 1"
"2024-10-29T07:18:00.879Z",,"""worker""","2024-10-29T07:18:00.879Z ioredis:connection reconnect in 1000ms"
"2024-10-29T07:18:00.879Z",,"""worker""","Connection to redis lost. Retry attempt: 1"
"2024-10-29T07:18:00.877Z","""unknown""","""worker""","2024-10-29T07:18:00.876Z ioredis:redis write command[<IP_ADDRESS>:6379 (bull:dHJhY2UtdXBzZXJ0)]: 0 -> bzpopmin([ 'bull:trace-upsert:marker', '5' ])"
"2024-10-29T07:18:00.877Z","""unknown""","""worker""","2024-10-29T07:18:00.876Z ioredis:redis write command[<IP_ADDRESS>:6379]: 0 -> evalsha('accb36ea979fc4cd20f4a56e3c4ad6df0ec0b14a,11,bull:cloud-usage-metering-queue:wait,bull:cloud-usage-metering-queue:active,bull:cloud-usage-metering-queue:prioritized,bull:cloud-usage-metering-queue:even ... <REDACTED full-length=""602"">')"

Potentially related issues:

https://github.com/nodejs/node/issues/21088
https://github.com/redis/node-redis/issues/2810
https://github.com/redis/ioredis/issues/754 which points to https://github.com/redis/ioredis#special-note-aws-elasticache-clusters-with-tls


Minor updates to tf.org maintenance process.

Add the TSC as the ultimate escalation point for maintainer disagreements.

Remove outdated information about TF-A and TF-M CI systems. These are now mature enough to accept contributions.

LGTM!
Reviewed-by: Joakim Bech<EMAIL_ADDRESS>


Wrong example

[Enter feedback here]
The examples section contains a wrong example.
This line:
dotnet test --logger:trx
should have been:
dotnet test --logger trx

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 61ca4064-7481-a263-60fc-c70127f2ad16
Version Independent ID: 2878af44-8208-7bd8-46d4-3acda3e82ce0
Content: dotnet test command - .NET Core CLI
Content Source: docs/core/tools/dotnet-test.md
Product: dotnet-core
Technology: dotnet-cli
GitHub Login: @mairaw
Microsoft Alias: mairaw

Thank you for your feedback @boskjoett. I've opened PR #14374 to fix this issue.
You can also make edits to our content when you find issues. For more information, see https://docs.microsoft.com/en-us/contribute/#quick-edits-to-existing-documents.


CUDA out of memory error for long sequences

When I try to run the esm model with large sequences( > 4700), I get error:
RuntimeError: CUDA out of memory. Tried to allocate 746.00 MiB (GPU 0; 15.78 GiB total capacity; 12.68 GiB already allocated; 718.75 MiB free; 13.63 GiB reserved in total by PyTorch)
I have tried setting the chunk size all the way down to 1 with no improvements.  I'm wondering if there are any other ways to reduce memory usage for large sequences.

4700 length on 16GB gpu simply won't fit I'm afraid. We're looking more memory efficient versions of ESMFold, but timeline to release is unclear

Where are the main memory bottlenecks coming from? Is it just the O(N^2) transformers? I am trying to run some of the smaller ESM-2 models on larger proteins and I am running out of memory on a 32G GPU.

For the LM yes it's the O(N^2) self-attention. For ESMFold there's the O(N^3) in axial attention, but the computation can be chopped into independent chunks to circumvent that, see model.set_chunk_size(128) instructions in frontpage README

how to deal?Can i use 2gpus?

Hi,
facing same issue for a sequence of length 2180.
using nvidia l4 x2 gpu (48)
already tried with multiple chunk sizes but still getting out of memory,
is there any other way or will the gpu size here not suffice for this sequence length
reference code:
tokenizer = AutoTokenizer.from_pretrained("facebook/esmfold_v1")
model = EsmForProteinFolding.from_pretrained("facebook/esmfold_v1")
model = model.cuda()
model.esm = model.esm.half()
torch.backends.cuda.matmul.allow_tf32 = True
model.trunk.set_chunk_size(8)
tokenized_input = tokenized_input.cuda()
model.eval()
with torch.no_grad():
output = model(tokenized_input)
can you please help with this.