[BUG] Creates ugly lines on screen for no reason

I have several possible suspects for this issue

legendary tooltips
Eclectic Trove (Legendary Tooltips resource pack)
blur

but everything working great without tooltipfix

any news?


Space between the pararms fails the execution and gives an unclear message

In the following screenshot you can see two executions which the only difference between them is the space before the second parameter.

No problem with requiring to not put a space, but the message is super-unclear.

as far as I know this is Spring Shell's implementation so I don't know how we can change that message

Duplicate #129


First Submission Branch

Hello Rohan and Tidbyt team - here is my first cut at stepcounter.   Tracking your steps (with coaching message) from Google Fit.
Per Rohan, submitting the app with the code that assumes a refresh token is passed via config.   He said that y'all would paste in the Oauth code (user auth, token exchange)  from your Google Calendar app.

I have read the CLA Document and I hereby sign the CLA

Rebasing to fix tests.

Closing in preference of #294.


1flow Web Actions going into Public Beta

Proposed changes
Removes the hidden frontmatter
Merge timing
asap

Sorry for the delay @tcgilbert


Revert url is only called after chunk uploads are complete, not during uploads?

 Is there a way to during chunk uploads processing when cancelled.

Hi @rikschennink, is this behaviour by design? I noticed this while working on chunked upload support for the Python Django filepond backend.
At present, if a chunked upload fails after uploading some number of chunks, you can retry or cancel the upload. If you cancel, as @prince272 has pointed out, the upload disappears from the filepond UI but no call is made to the server to notify it of the cancellation. This potentially results in a number of chunk files of a partially completed and then cancelled upload floating around on the server.
If there is no specific design reason for not making a revert call when cancelling a partially complete chunked upload, I could potentially have a go at adding this to filepond, along with an option to switch the functionality on, if you'd prefer the default case to remain as is.
Thanks.

@jcohen02 Would be a nice addition, I wouldn’t mind if it was default.

Ok, thanks @rikschennink, happy to take a look at this but its likely to take a few days for me to get round to this.

@jcohen02 no worries, take your time 👍

@rikschennink it would be useful to get your thoughts on the implementation of this feature - I'm still trying to familiarise myself with the design of filepond. I can see you've put a lot of work into the library design and so I'd like any update to stick within the current model. I can see a few different ways to solve this but any advice on which you think is the most suitable would be useful.
What's the issue?
When a user is uploading a file, if the upload fails (potentially after automatic retries), the user sees the file object in the UI with the red bar and two options - the "x" on the left hand side to cancel the upload or the retry button on the right hand side. If the "x" is clicked, the upload is cancelled, however, if a chunked upload was taking place and 1 or more chunks have been successfully uploaded, this may leave partially uploaded file data stored on the server disk.
What do we want to achieve?
If a chunked upload was taking place and one or more chunks have been successfully uploaded, when the button is clicked to cancel the upload, a revert request should be sent to the server so that the server-side has the opportunity to remove any stored file data relating to the cancelled upload.
Some possible implementation options:

In the REMOVE_ITEM handler in actions.js, add an additional branch to the if statement (from L884) to check if the upload is chunked and if one or more chunks have been successfully uploaded, dispatch a REQUEST_REVERT_ITEM_PROCESSING event to revert the upload. However, this results in REVERT_ITEM_PROCESSING being called and another REMOVE_ITEM being dispatched so we get into a loop! This could be solved by forcing state.options.instantUpload to be false. This prevents dispatch of another REMOVE_ITEM event but it feels like a hack so I don't think this is the right way to solve things.

The above option could possibly be solved by adding a chunkedUpload boolean parameter to the REQUEST_REMOVE_ITEM logic and then using this to trigger the revert event, passing the chunkedUpload parameter through to REVERT_ITEM_PROCESSING and using this to prevent REMOVE_ITEM being triggered again in this case. Again this doesn't feel quite right.

Another possibility may be to create a different button handler associated with a button with a new ID (but the same icon) that is displayed when a chunked upload fails. This would trigger a different event, say REQUEST_REMOVE_CHUNKED_ITEM which would in turn dispatch REVERT_CHUNKED_ITEM_PROCESSING. This would revert the upload without triggering REMOVE_ITEM since this is handled by the original event dispatch.

Hope this makes sense! Thanks.

@jcohen02 appreciate the detail, will get back to you asap, currently swamped with work.

No worries @rikschennink, whenever you get the chance.

@jcohen02 I think option 1 would be best, if we can prevent the looping situation (I think we can by calling item.revert instead of dispatching the revert request.
I've added reverting for limbo/input files in the past two releases to fix another issue, it might have "magically" fixed this as well (not sure haven't tested). If not, I think calling item.revert will solve the loop issue.


Fixes #4377

If imageryRectangle and imageryBounds do not overlap, continue with the next imagery tile.

@bagnell, @shunter, or @kring can one of you review/merge this?
Thanks @erikmaarten.

This clearly fixes the problem, so I want to get it into today's release.  If anyone else has more thoughts on a different approach for this, let me know.  Thanks @erikmaarten!
I'll update CHANGES in master.


Form Template Upload does not immidiatley save template

Description
If a user attempts to modify the Delivery Settings for a form that has been uploaded, but not yet saved, a user is shown an error 552.
Steps to reproduce

Upload a form JSON through the form-builder
(user is immediately brought to the preview of the form)
Click on "Settings" in left navigation
Click on "Receive responses by email" under delivery options
Click on "Save changes"
A toast message with error 552 should appear

Expected behaviour
The user can modify the form settings immediately after uploading the form.
Screenshots or videos
There is a video, it just takes time to load

Additional context
Any other information about the problem here.

We seen roughly this type of thing happen a few different ways.
@Abi-Nada @srtalbot --- thoughts on starting folks on the edit screen vs preview after the initial upload?
That's not a total solve but for sure would be an improvement ...wanted to suggest it before we work on this fix.

Button does say edit a form :)

Can keep as preview  -

Closing this issue as a duplicate - tackling 3628 will solve this


Test: Address warning in ansible when running molecule

Issue Summary
Following error message is observed when running molecule tests for eos_designs_unit_tests.
Possibly other scenarios as well.
[WARNING]: Invalid characters were found in group names but not replaced, use
-vvvv to see details

Which component(s) of AVD impacted
eos_designs
How do you run AVD ?
Ansible CLI (with virtual-env or native python)
Steps to reproduce
molecule converge --scenario-name eos_designs_unit_tests

Relevant log output
INFO     Running from /workspaces/ansible-avd/ansible_collections/arista/avd : ansible-galaxy collection install -vvv --force /workspaces/ansible-avd/ansible_collections/arista/avd
INFO     Running eos_designs_unit_tests > syntax
[WARNING]: Invalid characters were found in group names but not replaced, use
-vvvv to see details

playbook: /workspaces/ansible-avd/ansible_collections/arista/avd/molecule/eos_designs_unit_tests/converge.yml
INFO     Running eos_designs_unit_tests > create
[WARNING]: Invalid characters were found in group names but not replaced, use
-vvvv to see details

Contributing Guide

[X] I agree to follow this project's Code of Conduct

Change all ansible groups to be upper case with underscores. Maybe require adjustments of fabric_name variables as well. There should be no changes to configs from this.


Cherry picked fixes for proposed 1.1.x release

This PR merges the two fixes from the master branch proposed for a new 1.1.x patch.
The changes to the apps (related to OpenImageIO) add a requirement for C++11 that was not previously present, so that impact needs weighing before we decide to merge this. I have successfully built and tested this against the latest OIIO release on Linux, but can also confirm it will not build with an earlier CXX standard. One solution may be to support both paths with some CMake magic.
Commits cherry picked from:
https://github.com/imageworks/OpenColorIO/commit/718cc7f9f7ebb189643b8691df7ef4d9a735adac
https://github.com/imageworks/OpenColorIO/commit/1ff22741ff577bd1be504452eb3fa4ce1a652370
(plus a later correction made to two ImageInput/ImageOutput mismatches)

Hitting same CI failures as #677
RB-1.1 CI env might just need some tweaking. AppVeyor is failing as the VS2015 image is missing Boost. The Travis Mac Clang build is failing to "find a suitable distribution" of markupsafe.

Oh, oops, I didn't realize I was changing C++ standard requirements. (C++ 03 compat in 2019, ok...)
I don't know how to reconcile that. OIIO 2.x certainly is only C++11 or higher, but a patch release should definitely not force a different compiler.

I wonder if we could extend the use of the OIIO version macro to revert to the C++ 03 friendly constructors when appropriate. That or an explicit opt-in cmake property to choose.

I'll cherry pick #682 into this PR once that is merged into master

I'll cherry pick #682 into this PR once that is merged into master

As no C++ version is explicitly mentioned in the makefiles, the gcc default version is C++98 (for gcc version < 6.1) while Vc++ default is latest (i.e. depends on the Visual Studio version).  Bottom line, the ‘auto’ is problematic in that context.
gcc link: http://gcc.gnu.org/projects/cxx-status.html (bottom of the page)

'auto' in this case is a 'unique_ptr', so in either case, C++11 is the minimum.
With the latest commit ( https://github.com/imageworks/OpenColorIO/commit/65a08dbedb771adc62b35b71e2a8b637776b8504 ), C++11 is only required when building with OIIO_VERSION >= 10903. The default behavior would be to not build at all unless OIIO_VERSION < 10903 were being used, which is consistent with OCIO 1.1.
Keeping the OIIO related commits would allow OCIO to be built against the latest OIIO optionally. The builder would have to explicitly specify a newer C++ standard on the commandline with -DCMAKE_CXX_FLAGS, but they would have the option to do so. We could update the build instructions and make this requirement clear.
The alternative is to remove these commits from the potential patch release and not support a newer OIIO until OCIO v2, which is acceptable if this is deemed beyond the scope of a patch.

...or we target a 1.2 release and officially support building with a newer C++ standard, without changing library functionality.

One potential solution to benefit from OIIO newer versions for OCIO v1, would be to explicitly enforce the C++11 constraint.
While keeping the current changes, one could add the following code somewhere in the ocioconvert and ociodisplay main files:
#if OIIO_VERSION >= 10903 && __cplusplus < 201103
#error "While not officially supported, compile the library with C++11 (or higher) to benefit from newer OIIO versions"
#endif

I tested the added #error directive with builds against older and newer OIIO. To note, when I did try to build OCIO with OIIO 2.0 and C++03, Both the compiler and OIIO gave clear errors about the incompatibility before OCIO did.

This PR will be merged in 24 hours if there is not additional feedback.


where to get the .so files?

I want to Enable PHP extensions for mysql driver

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 9ba00ed4-0fb2-ae6f-dec8-fc1e6a77c6e0
Version Independent ID: 0b72e7cc-958d-378d-34b5-029c187db2a5
Content: Configure PHP apps - Azure App Service
Content Source: articles/app-service/containers/configure-language-php.md
Service: app-service
GitHub Login: @cephalin
Microsoft Alias: cephalin

@shadyshrif Thanks for the question! We are investigating and will update you shortly.

@shadyshrif The .so files should be located inside your php extension directory.
You can use phpinfo() function to find the location of your php extension directory.

@shadyshrif Since we have not heard back from you we will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly reopen the issue and continue the discussion.


tracking: Improving hdfs support.

For details, please refer to #10575.

[ ] Create two images for different hadoop versions: 2.x and 3.x
[x] Create an example to use k8s's configmap to allow user to pass hadoop config dirs to risingwave.
[x] Update doc to tell user to create new image based on risingwave's image so that they can use patched hadoop client.

cc @wcy-fdu @arkbriar

Is it possible to add the test in intergration tests? I don't think cloud team has bandwith to add HDFS as storage backend for us.

I believe the HDFS support is mature enough, both kernel's and operator/helm chart's. The problem left should be the support from the test driver, a.k.a. the kube-bench cc. @huangjw806

I believe the HDFS support is mature enough,

+1, currently we has supported docker compose on HDFS and k8s on HDFS, and the remain work is to create real HDFS cluster and build test pipeline for it. We can go ahead when test team has bandwidth.


Display block name on non collapsed blocks in formwidget

Now the user can see the blockname  on unfolded blocks to identify the right block to change content

@xitara could you post a before/after screenshot?

before:

after:

the red arrows show the changes

How does it look when collapsed? This is why the core blocks provided by this plugin all have "section" fields at the top of their field configs to act as the block titles in the backend.

Collapsed it's look like before, no changes.
Yes you can do it with a section. My thoughts are that this work can be saved with this little patch

Hmm, my main concerns at this point would be:

The existing demo blocks all use the section approach, so either we'd implement this PR and remove the sections so that they aren't doubling up, or we just say "Use sections" and don't merge this.
I'm not a fan of using <h*> tags purely for visuals, they have semantically meaning and slapping them in the middle of the page's markup can cause issues, especially with accessibility. Are you able to switch it to a <strong> or something that's more explicitly styled rather than relying on the implicit styling of h tags?

I think, we use sections and don't merge, this is the more flexible solution


Site Restart Setting Update is waiting on Slot Runtime Task before executing kusto query

The performance of this detector can be improved by following the model that most other detectors use. The current implementation blocks on the SlotRunTimeMap before it makes the kusto call, so that is can get a list of runtime sitenames.
AntaresRuntimeWorkerEvents
| where PreciseTimeStamp >= datetime(2017-06-19 18:10:00) and PreciseTimeStamp <= datetime(2017-06-20 18:10:00) and Tenant =~ "bd91c5193454456facb025f7d19c1f33"
| where EventId == 15028 and SiteName in ("pulseapi__2b83")
| where (LogStatement contains "Need to recycle worker processes due to the new environment variable" or LogStatement contains "ApplicationPool settings change requires worker processes to be restarted")
| summarize count() by SiteName, bin(PreciseTimeStamp, 5m)
Change the bold part above to "(SiteName=~ "pulseapi" or SiteName startswith "pulseapi__")"
This means you won't have to block on the Slot Runtime Map Task that can take some time. Then all you have to do is use:
DiagnosticUtilities.MergetSlotTimeAndSiteEvents(, slotTimeRange[Parameters.Slot])

Andrey let me know if you don't own this detector but I thought you did

SiteRestartUserInitiatedDetector is also doing something that should be improved, though it is not hurting performance very much:
foreach (var siteName in runtimeSiteNames)
 {
                filter += string.Format("\"{0}\",", siteName);
//cover case when in Kusto site is in lower case
if (siteName.Any(char.IsUpper))
{
 filter += string.Format("\"{0}\",", siteName.ToLower());
}
}
You can change the query from:
SiteName in ("CSSGlobalRetailBrandsHouse","cssglobalretailbrandshouse")
to
SiteName =~ "CSSGlobalRetailBrandsHouse"
=~ is the case insensitive equality operator


queueless Ragna?

While responding to all issues since the launch (thank you everyone that took the time to get in touch!) it became somewhat evident that the queue backend is confusing / limiting.
In general, the task queue is used to scale deployments of Ragna. When auto scaling worker nodes on the queue length, that is all there is to do to load balance Ragnas backend, while keeping the REST API nodes constant (of cource they also need to be autoscaled at some point if you have a lot of concurrent users). Other options like a service mesh were a lot more complicated when we want to have one consistent REST API the user can hit extend with plugins.
Still, this doesn't answer why the queue is part of the Python API. Let me give you some historical context on how it came into being. In the very beginning, Ragna had no public Python API (#17). The REST API was the only real entrypoint for users. After I rewrote the project in #28, I failed to cleanly separate the two. Back then, the Python API even still had the database baked in (#65). I removed that in #67.
Meaning, right now, I can't come up with a good reason for why we have a task queue as part of the Python API. Not having it could potentially simplify the implementation and enable us to implement a few nice to have features, such as #185.
The only downside I can currently come up with is that parallelizing tasks the Ragna API will be harder in the future. Right now, you can simply select a non-memory task queue and start a worker and be done with it. Without a task queue the user has to implement their own parallelization scheme. That being said, since the Python API is mostly meant for experimentation, I'm not sure if this is a strong argument.
I'm going to start a branch soon to move the task queue to the Python API to see if there are more downsides that I'm currently not seeing. Will post updates here.

thank you everyone that took the time to get in touch!

In my experience, contributions and engagement are tightly linked to how open the core dev team is to feedback. The ragna team have been extremely receptive IMO (this issue being a perfect example), thank you for sowing the seeds of good culture! :)

I'm going to start a branch soon to move the task queue to the Python API to see if there are more downsides that I'm currently not seeing. Will post updates here.

Thanks for making this public. IMO It's the right approach to have two branches in parallel - there are so many 'breakthroughs' right now in Gen AI that it feels like a good idea to explore both paths in parallel for as long as possible.
That being said, I'm not fully sure I understand the rationale for removing the queue only from the Python API.
My current understanding is that a queue is useful for CPU-bound tasks in Python. The most common CPU-bound tasks that I am familiar with for RAG applications are i) loading and chunking the documents, ii) computing the embeddings and iii) performing a vector-similarity search or lexical-search. There are other less common (for now!) tasks that I haven't mentioned such as re-ranking, but I'll ignore these. These three sub-tasks are currently (mostly) abstracted behind the prepare() interface in ragna. (Of course, all of these become I/O-bound tasks if they are computed on a server elsewhere eg using OpenAI to compute the embeddings.)
For I/O bound tasks and for an application that already uses async methods eg FastAPI web-server, I don't really see the benefit of using a queue (at least for workloads that I have experienced in the past). When interacting with an external API like OpenAI, I don't think there is much benefit to waiting for a response on a worker, as opposed to waiting for a response on the main thread. Disadvantages include issues like #185.
So, my point is, why restrict the queue experimentation branch only to the Python API? Surely the web server/API would also benefit from this. The key for me is distinguishing between I/O bound and CPU bound tasks, and then executing them in the right place ie on the main thread, or on a worker.
Specific Questions:

Is there any benefit to having the interaction with the LLM take place via the queue/on a worker, rather than on the main thread ? It prevents features like #185, and it's taking up resources idling on the queue.
Is there some pattern/trick that could be adopted to only execute CPU-bound tasks on the queue, and everything else on the main thread. For example, FastAPI executes def functions on a separate thread (I think this is a reasonable analogy).

@nenb You are correct that the task queue is mostly useful for CPU bound tasks. But since we allow generic extensions, we cannot make any assumptions on the nature of the task. And since I/O bound tasks still work correctly with a task queue, this is just the common denominator. Indeed we lose nice-to-have features like #185, but it lowers the maintenance burden by only having one way to do everything.

Is there some pattern/trick that could be adopted to only execute CPU-bound tasks on the queue, and everything else on the main thread. For example, FastAPI executes def functions on a separate thread (I think this is a reasonable analogy).

Yes, that could work, but we would need a bigger design for this. Let's say for example I have an CPU bound SourceStorage.store, but an I/O bound Assistant.answer. I could mark the former with def and the latter async def. But what are we going to do for Chat.answer? Should we mark this sync and block over a mostly I/O operation or do we mark it async and accept the fact that users might be confused why awaiting the result blocks for a long time?

Answering my own question here:

I could mark the former with def and the latter async def. But what are we going to do for Chat.answer?

Well, I guess we can have both as discussed in https://github.com/Quansight/ragna/issues/183#issuecomment-1804834416.

Just "vomiting" my thoughts here for later:

All components are free to define their abstract methods either as sync or async. Sync marks the method as CPU bound and async as I/O bound.
On the Python API, we have both sync and async endpoints, i.e. def answer and async def aanswer. We leave it to the user to select the right method for the given components.
Meaning, if we have a CPU bound task and the user chooses the async endpoint, there won't be any async behavior and the execution will just block until the result is computed. On the flip side, if a user selects the sync endpoint, but the task is async, we still block and thus lose the async advantage.
Maybe we can find a way to warn in these suboptimal cases, but I don't want this to get annoying. We can't automate
On the REST API, the story is a little different. If we have the metadata whether a task is CPU or I/O bound, we can always call the correct endpoint. CPU bound in this case means enqueuing the task as await the result. For I/O bound tasks, we can await directly and thus don't need to use the task queue.

I think this can work :slightly_smiling_face:

We had a lengthy internal discussion about this last night and came to the following conclusion: the task queue is a premature optimization for an architecture that has changed quite a bit since we started on Ragna. As such, it has very little going for it. In addition, it causes real issues for users. And just because they are just nice-to-haves from a functionality perspective, it doesn't mean that users turn away because of them. Thus, our current plan is to remove the task queue completely and only later revisit this if the need arises.
@nenb I think the FastAPI analogy is pretty neat here. Sync functions will be run on a separate thread and async ones are just awaited in the main thread. That makes the Python API fully async and thus directly usable by the REST API as well.
One thing that I worry about is DX while debugging multithreaded code. Not sure if that is an issue, but I would prefer being able to use a debugger if possible. Still, this is not blocking for the design.

Development for this is happening in #205.

@pmeier Thanks for the updates, this sounds great. I'll follow #205 with considerable interest, and will post here if I have any further comments.
PS I do think that the task queue has a place for heavier workloads, or if someone is using ragna to orchestrate RAG across many concurrent users (which I think is an intended use-case). So, I do think it could still have a role to play in the future, especially as huey is so lightweight. I think you are aware of this already though!

Hi @nenb @pmeier I noticed your team removed the LLM requests queue? I'm curious why ?
I'm the maintainer of LiteLLM https://github.com/BerriAI/litellm and we have a queue implementation that handles 100+ req/seconds for Any LLM. Would it be helpful to use our request queue here - if not I'd love your feedback on why not ?
Here's the quick start on the litellm request queue
docs: https://docs.litellm.ai/docs/routing#queuing-beta
Quick Start

Add Redis credentials in a .env file

REDIS_HOST="my-redis-endpoint"
REDIS_PORT="my-redis-port"
REDIS_PASSWORD="my-redis-password" # [OPTIONAL] if self-hosted
REDIS_USERNAME="default" # [OPTIONAL] if self-hosted

Start litellm server with your model config

$ litellm --config /path/to/config.yaml --use_queue

Here's an example config for gpt-3.5-turbo
config.yaml (This will load balance between OpenAI + Azure endpoints)
model_list:
  - model_name: gpt-3.5-turbo
    litellm_params:
      model: gpt-3.5-turbo
      api_key:
  - model_name: gpt-3.5-turbo
    litellm_params:
      model: azure/chatgpt-v-2 # actual model name
      api_key:
      api_version: 2023-07-01-preview
      api_base: https://openai-gpt-4-test-v-1.openai.azure.com/

Test (in another window) → sends 100 simultaneous requests to the queue

$ litellm --test_async --num_requests 100

Available Endpoints

/queue/request - Queues a /chat/completions request. Returns a job id.
/queue/response/{id} - Returns the status of a job. If completed, returns the response as well. Potential status's are: queued and finished.

I'm going to start a branch soon to move the task queue to the Python API to see if there are more downsides that I'm currently not seeing. Will post updates here.

@pmeier I'd love to help here - we implemented a queue for LLM requests to handle 100+ request/second. If this is not a good enough solution- I'd love to know what's missing

@ishaan-jaff

I noticed your team removed the LLM requests queue?

We didn't remove an "LLM requests queue", but rather the task queue that handled everything inside the RAG workflow from extracting text, embedding it, retrieving it, and finally sending it to an LLM.

I'm curious why ?

You can read this thread for details. TL;DR it was confusing to users and limited our ability to implement things like streaming API responses. I see that LiteLLM supports streaming, but that is likely only possible because you hard depend on Redis as queue. We didn't want to take a hard dependency on that since it is not pip installable.
Since hitting an LLM API is heavily I/O bound, we just went vanilla async execution instead.

Would it be helpful to use our request queue here - if not I'd love your feedback on why not ?

I don't want to have a mechanism only for the LLMs (assistants in Ragna lingo) unless absolutely necessary. Whatever abstraction we build should work for everything. Right now we have that.
What I think would be totally possible is to implement a class LiteLlmAssistant(ragna.core.Assistant). That way we are no longer bound to the requirement that the queue needs to handle more than just the LLMs. Still, this would be an extension, i.e. not land in Ragna core, since one needs extra infrastructure requirements (redis-server) that cannot be installed by pip. Happy to give pointers or help out if you want to give it a shot.


Is there a dmg distribution?

@djyde No, but why you need dmg? 😳

zip is ok, too 😅  Just lazy for compiling by myself.

@djyde You can download the macOS App from https://github.com/Sunnyyoung/Suohai/releases/download/1.0/Suohai.zip, no compile needed.


DEVX-2740: Patch Credential Onboarding bug

Patch a bug where service credential on-boarding overwrites previously attached services.

Related Issue

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[ ] I have signed the Adobe Open Source CLA.
[ ] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[ ] I have read the CONTRIBUTING document.
[ ] I have added tests to cover my changes.
[ ] All new and existing tests passed.

Tests passed locally but not in PR. Looking.


Explore Django Rest Framework extension using ModelSerializers

I was thinking it'd be pretty feasible to use the DRF ModelSerializers to greatly simplify some complex tasks (related to issue #1375 )  like importing/exporting subsets of data with arbitrary depth.
So we could add a higher level of Resource that uses ModelSerializers or Serializers to import an model instance with its entire hierarchy of relationships intact, e.g. SerializerResource or the like.
It's entirely possible that Serializer could allow moving away from needing an underlying dataset engine, although I'm not sure if performance is there at scale or not. They seem pretty fast though.
(Possible that expanding to support another library in DRF is too big a scope, just an idea I had that might be worth some cycles)

I would very much welcome this for projects where DRF is already heavily used. I am facing this now. I have defined serializers for allmost all resources, including the creation/update of nested objects. If one could use those serializers for exporting/importing, this would greatly simplify the handling of nested objects.

My thinking on the way to implement this would be to create a ModelSerializerResource that inherits from ModelResource, and overrides a lot of the functions. My recollection is that DRF has a way to find the default serializer for a given model, but you could also have a _meta option for specifying a specific serializer to use.
One challenge here is that I think it would only work cleanly for Json and Yaml, because CSV/Excel are very unreliable at nested data structures past a depth of one, unless the serializerfields are Json/yaml themselves.
The ModelSerializerResource would need to to override a lot of the functions on ModelResource and would probably not be able to use widgets and fields - because serializers essentially do what those do.
The first step would probably be to add DRF and a DRF app to the core test implementation, and then start fiddling with how serializers import. A really nice part about this is, at least as I recall, DRF serializers can handle creating related models for you, which includes an often-requested piece of functionality (creating related model instances).
Maybe I can spin up a branch for this one to do some brainstorming in the next couple months as I get some time.

It would help for issues like #1425, we could use the DRF DecimalField class to format and process numbers correctly.  At the moment we are in danger of duplicating logic.

If we're open to the possibility of adding DRF as a dependency I could see that being rather useful. I'm not sure architecturally speaking if we want to have DRF be required for import-export for people who are using vanilla django, but honestly DRF seems to be rapidly becoming the default for Django?
DRF's ModelSerializers do a lot of what import-export does natively so it'd probbly be a pretty major redesign though.

I don't know anything about the actual implementation of this library, but I've seen esupport libraries which add "DRF" capabilities to another library.
One small example is django-polymorphic and django-rest-polymorphic
But this would a.s. mean feature duplication.

This is my solution.
Hope it help.
core/mixins.py
from django.utils.translation import gettext_lazy as _

from import_export.formats.base_formats import CSV, HTML, JSON, ODS, TSV, XLS, XLSX, YAML
from import_export.resources import ModelResource
from rest_framework import status
from rest_framework.decorators import action
from rest_framework.response import Response
from tablib import Dataset

from .exceptions import ExportError

EXPORT_FORMATS_DICT = {
    "csv": CSV.CONTENT_TYPE,
    "xls": XLS.CONTENT_TYPE,
    "xlsx": XLSX.CONTENT_TYPE,
    "tsv": TSV.CONTENT_TYPE,
    "ods": ODS.CONTENT_TYPE,
    "yaml": YAML.CONTENT_TYPE,
    "json": JSON.CONTENT_TYPE,
    "html": HTML.CONTENT_TYPE,
}
IMPORT_FORMATS_DICT = EXPORT_FORMATS_DICT

class ExportMixin:
    """Export Mixin"""

    export_filename:str = "Default"
    export_resource: ModelResource = None

    @action(detail=False, methods=["get"])
    def export(self, request, *args, **kwargs):
        filename = self.export_filename
        eformat = request.query_params.get("eformat", "csv")

        queryset = self.filter_queryset(self.get_queryset())

        dataset = self.get_resource().export(queryset)

        if not hasattr(dataset, eformat):
            raise ExportError(
                detail=_("Unsupport export format"), code="unsupport_export_format"
            )

        data, content_type = (
            getattr(dataset, eformat),
            EXPORT_FORMATS_DICT[eformat],
        )

        response = HttpResponse(data, content_type=content_type)
        response["Content-Disposition"] = (
            f'attachment; filename="{filename}_{TODAY}.{eformat}"',
        )
        return response

    def get_resource(self):
        if not self.export_resource:
            raise ExportError(detail=_("Pleause set export resource"))
        return self.export_resource()

class ImportMixin:
    """Import Mixin"""

    import_resource: ModelResource = None

    @action(methods=["post"], detail=False)
    def import_data(self, request, *args, **kwargs):
        file = request.FILES["file"]
        extension = file.name.split(".")[-1].lower()
        import_resource = self.get_import_resource()
        dataset = Dataset()

        if extension in IMPORT_FORMATS_DICT:
            dataset.load(file.read(), format=extension)
        else:
            raise ImportError("Unsupport import format", code="unsupport_import_format")

        result = import_resource.import_data(
            dataset,
            dray_run=True,
            collect_failed_rows=True,
            raise_errors=True,
        )

        if not result.has_validation_errors() or result.has_errors():
            result = import_resource.import_data(
                dataset, dry_run=False, raise_errors=True
            )
        else:
            raise ImportError("Import data failed", code="import_data_failed")

        return Response(
            data={"message": "Import successed"}, status=status.HTTP_201_CREATED
        )

    def get_import_resource(self):
        if not self.import_resource:
            raise ImportError(detail=_("Pleause set import resource"))
        return self.import_resource()

core/exceptions.py
from rest_framework import exceptions

class ExportError(exceptions.APIException):
    status_code = 400
    default_detail = _("Export error")
    default_code = "export_error"

class ImportError(exceptions.APIException):
    status_code = 500
    default_detail = _("Import error")
    default_code = "import_error"


Architect a "Verified Reproducible Build Attestation"

Architect a "Verified Reproducible Build Attestation".
Some useful links:

https://www.cisa.gov/sites/default/files/2024-03/CISA_RSAA_User_Guide_18_March_2024.pdf
https://cyclonedx.org/capabilities/attestations/
https://github.com/CycloneDX/specification/blob/master/tools/src/test/resources/1.6/valid-attestation-1.6.json
https://github.com/in-toto/attestation/blob/main/README.md
https://www.cisa.gov/resources-tools/resources/secure-software-development-attestation-form (and related DB for attestations and artifacts)

TODO, check if similar exists for EU

https://www.cisa.gov/sites/default/files/2024-04/Self_Attestation_Common_Form_FINAL_508c.pdf

TODO, look at fields/requirements of the form and see if all are also valid for verification builds / as opposed to official published build)

https://www.activestate.com/blog/everything-developers-need-to-know-about-attestations/
https://cyclonedx.org/docs/1.6/json/#declarations_attestations

Also:

https://www.cisa.gov/resources-tools/resources/secure-software-development-attestation-form (and related DB for attestations and artifacts)

TODO, check if similar exists for EU

https://www.cisa.gov/sites/default/files/2024-04/Self_Attestation_Common_Form_FINAL_508c.pdf

TODO, look at fields/requirements of the form and see if all are also valid for verification builds / as opposed to official published build)

https://www.activestate.com/blog/everything-developers-need-to-know-about-attestations/
https://cyclonedx.org/docs/1.6/json/#declarations_attestations

Sigstore seems to be a center of gravity in the secure supply chain processes.
We should seriously consider using the formats supported by Rekor to be able to use that infrastructure downstream.

Sigstore seems to be a center of gravity in the secure supply chain processes. We should seriously consider using the formats supported by Rekor to be able to use that infrastructure downstream.

An interesting read, this is what I understand from reading their various docs:

CoSign's primary aim is in providing signing of "containers" stored in an OCI Registry.
CoSign signing process is usually interactive involving opening of a Browser window for Identity verification during the signing
CoSign keyless signing uses Rekor attestation ledger allowing "Signature" to be "verified" against the Rekor ledger entry identity and timestamp, hence does not require a "key"
Rekor provides a append only transparency log, for things like attestations
Rekor provide a public server: https://rekor.sigstore.dev

I take from that we could maybe:

Using CoSign seems to be specific to keyless "Identity" based Signatures, it's usage for generic blob/artifact signing is not its main aim
Rekor transparency log provides various pluggable formats including in-toto-attestations : https://github.com/sigstore/rekor/tree/main/pkg/types#currently-supported-types

Rekor also support generic types: https://docs.sigstore.dev/logging/sign-upload/

Using Rekor transparency log, possibly the public one, could be a means for 3rd party reproducible attestation uploading. The public instance is limited to 100Kb attestation blobs

https://github.com/in-toto/attestation/issues/82

in-toto/attestation#82 in-toto/attestation#129

Thanks Shelley, I am going to follow up with some questions to in-toto in their Slack

Further research, and references:

in-toto attestation's are primarily targeted at automated policy engines, like in-toto-verify (and the associated in-toto-run'type interfaces) or Google Binary-Authorization engine, which capture "attestations at the time of execution" : ref: https://github.com/in-toto/attestation/blob/main/spec/README.md#in-toto-attestation-framework-spec
CycloneDX 1.6 attestations provide a full metadata declaration for attestations in the CDXA format : https://cyclonedx.org/capabilities/attestations/

I am currently thinking of this outline architecture:

Temurin build (temiurin-build) constructs a CycloneDX CDXA "provenance" attestation document at the end of the build along side the existing SBOM document, and links the SBOM->CDXA. The "Adoptium signed" CDXA will state the "provenance" of the built JDK tarball from an Adoptium "Producer" perspective, ie. something along the lines of a statement with evidence of "reproducible build", eg. binary hashes, and a "re-build" reproducible build evidence log?? Both SBOM and CDXA are published at release time.
A 3rd party performs a successful "Reproducible Verification Build", and creates their own "3rd party signed" consumer CDXA "provenance" attestation document, which documents their statement of evidence of "reproducible build" for the given Temurin release JDK tarball. The 3rd party then publishes this CDXA attestation document to the Adoptium marketplace: https://adoptium.net/marketplace/.
The last step by the Adoptium community as a consequence of a 3rd party publishing a "consumer" CDXA, is to run a "Verify" step, which compares the "producer" CDXA with the "consumer" CDXA, comparing the references and evidence, and upon success then lists the given 3rd party as "attesting" to the reproducibility of the given Temurin JDK release. This last step could also update a "reference", maybe a "bom-link" in the Temurin SBOM, to reference the "verified" consumer CDXA, the "version" number of the SBOM would be incremented by 1, as with accordance to the "CycloneDX Authorative Guide to SBOMs".

Future:

Add capability of 3rd party uploading a CycloneDX CDXA document to the public Sigstore Rekor instance, and allowing the step(3) "Verify" stage to reference the Rekor transparency log entry. The "bom-link" would then reference the Rekor entry.

From talking to members of the in-toto community, their suggestion points to looking at https://github.com/in-toto/witness
Which from the front cover looks a possibility.

Option 2:

Temurin ci.adoptium.net pipeline produces an in-toto attestation of reproducible-build provenance using the in-toto-witness tooling
3rd party performs a Temurin re-build, and produces their own in-toto attestation using in-toto-witness
Eclipse Adoptium runs an in-toto-policy verification using in-toto-witness that verifies the Adoptium attestation and the 3rd party attestation evidence matches, to attest the claim of an independent Verified Reproducible Build

Plan for 1Q 25:

Create a CycloneDX CDXA json document
Provide a temurin-build cycloneDX tooling CLI to create CDXA

@tellison @smlambert @jiekang My thoughts on what is in the CDXA: https://github.com/adoptium/temurin-build/issues/3950#issuecomment-2488344806

I think the Target->component entry should have the full name of the file (e.g. OpenJDK21U-jdk_x64_linux_hotspot_21.0.5_11.tar.gz)
Though that made me look at the releases page and now wonder what we should plan for staticlibs, debugimage, testimage, and jre artifacts...

I think the Target->component entry should have the full name of the file (e.g. OpenJDK21U-jdk_x64_linux_hotspot_21.0.5_11.tar.gz)

Not opposed to the suggestion, but want to mention that the full name of the artifact will feature in the url to the binaries repo

I think the Target->component entry should have the full name of the file (e.g. OpenJDK21U-jdk_x64_linux_hotspot_21.0.5_11.tar.gz)

Not opposed to the suggestion, but want to mention that the full name of the artifact will feature in the url to the binaries repo

yes that was my original thought too

I think the Target->component entry should have the full name of the file (e.g. OpenJDK21U-jdk_x64_linux_hotspot_21.0.5_11.tar.gz)

Not opposed to the suggestion, but want to mention that the full name of the artifact will feature in the url to the binaries repo

yes that was my original thought too

Or preferably, a stable URL link to the asset via the Adoptium API rather than straight to the storage (just to avoid building dependencies on the storage implementation into the CDXA)
e.g. https://api.adoptium.net/v3/binary/version/jdk-21.0.5+11/linux/x64/jdk/hotspot/normal/eclipse

A generated example:
{
  "bomFormat" : "CycloneDX",
  "specVersion" : "1.6",
  "version" : 1,
  "declarations" : {
    "assessors" : [
      {
        "thirdParty" : true,
        "organization" : {
          "name" : "Acme Inc"
        },
        "bom-ref" : "assessor-1"
      }
    ],
    "attestations" : [
      {
        "summary" : "Eclipse Temurin Attestation",
        "assessor" : "assessor-1",
        "map" : [
          {
            "claims" : [
              "claim-1"
            ]
          }
        ]
      }
    ],
    "claims" : [
      {
        "target" : "target-jdk-1",
        "predicate" : "VERIFIED_REPRODUCIBLE_BUILD",
        "bom-ref" : "claim-1"
      }
    ],
    "targets" : {
      "components" : [
        {
          "type" : "application",
          "bom-ref" : "target-jdk-1",
          "name" : "Temurin jdk-21.0.5+11 x64 Linux",
          "externalReferences" : [
            {
              "type" : "distribution",
              "url" : "https://api.adoptium.net/v3/binary/version/jdk-21.0.5+11/linux/x64/jdk/hotspot/normal/eclipse",
              "hashes" : [
                {
                  "alg" : "SHA-256",
                  "content" : "abcdefg0123456789"
                }
              ]
            }
          ]
        }
      ]
    },
    "affirmation" : {
      "statement" : "Acme confirms a verified reproducible build",
      "signatories" : [
        {
          "organization" : {
            "name" : "Acme Inc"
          },
          "externalReference" : {
            "type" : "website",
            "url" : "https://acme-inc.com"
          }
        }
      ]
    }
  }
}

And here is the same using XML output:
<?xml version="1.0" encoding="UTF-8"?>
<bom serialNumber="urn:uuid:4068785d-f37f-4c56-b0c5-1951d2ec375b" version="1" xmlns="http://cyclonedx.org/schema/bom/1.6">
  <declarations>
    <assessors>
      <assessors bom-ref="assessor-1">
        <thirdParty>true</thirdParty>
        <organization>
          <name>Acme Inc</name>
        </organization>
      </assessors>
    </assessors>
    <attestations>
      <attestation>
        <summary>Eclipse Temurin Attestation</summary>
        <assessor>assessor-1</assessor>
        <map>
          <claims>
            <claim>claim-1</claim>
          </claims>
        </map>
      </attestation>
    </attestations>
    <claims>
      <claim bom-ref="claim-1">
        <target>target-jdk-1</target>
        <predicate>VERIFIED_REPRODUCIBLE_BUILD</predicate>
      </claim>
    </claims>
    <targets>
      <components>
        <component type="application" bom-ref="target-jdk-1">
          <name>Temurin jdk-21.0.5+11 x64 Linux</name>
          <externalReferences>
            <reference type="distribution">
              <url>https://api.adoptium.net/v3/binary/version/jdk-21.0.5+11/linux/x64/jdk/hotspot/normal/eclipse</url>
              <hashes>
                <hash alg="SHA-256">abcdefg0123456789</hash>
              </hashes>
            </reference>
          </externalReferences>
        </component>
      </components>
    </targets>
    <affirmation>
      <statement>Acme confirms a verified reproducible build</statement>
      <signatories>
        <signatory>
          <organization>
            <name>Acme Inc</name>
          </organization>
          <externalReference type="website">
            <url>https://acme-inc.com</url>
          </externalReference>
        </signatory>
      </signatories>
    </affirmation>
  </declarations>
</bom>

For Q1 '25, the above CDXA.xml will be used: https://github.com/adoptium/temurin-build/issues/3950#issuecomment-2503602280


Mockup collection for Room Resources

Please do work for this task in a branch called issue-021.

Room Resources Collection:
room_number,
capacity,
chairs,
desks,
tv,
phone_number,
data_jacks,


Missing powershell commands for complex APIs

Hi, powershell commands for the below APIs are missing in our documentation here: https://learn.microsoft.com/en-us/graph/api/resources/authenticationeventsflow?view=graph-rest-beta
Under Userflows for Azure AD for customers (Preview):

All APIs under Identity Providers
All APIs under Authentication Condition Applications
List attributes, add attributes and remove attributes under User flow Attributes

Attaching the screenshots for clarity:

Could you please help generate the powershell commands for the one missing and also make sure that the example from HTTPs is translated as well?

Here's another question regarding the path:
/identity/authenticationEventsFlows/{authenticationEventsFlow-id}/conditions/applications/includeApplications/

applications is a complex property contained in the entity type authenticationConditions which is the type for the complex property conditions.
By adding a ReadRestrictions property to the complex property conditions, we can generate the path: /identity/authenticationEventsFlows/{authenticationEventsFlow-id}/conditions
However we cannot expand into the entity type of this complex property. We only expand containment navigation properties.

Complex types are keyless nominal structured types. The lack of a key means that complex types cannot be referenced, created, updated or deleted independently of an entity type

Doc. reference
Hence this path is not feasible: /identity/authenticationEventsFlows/{authenticationEventsFlow-id}/conditions/applications
Is this a case of poor API design?
cc: @darrelmiller @baywet

@irvinesunday I'm not sure I understand your comment about poor API design. Looking at this path with the metadata
/identity/authenticationEventsFlows/{authenticationEventsFlow-id}/microsoft.graph.externalUsersSelfServiceSignUpEventsFlow/onAuthenticationMethodLoadStart/microsoft.graph.onAuthenticationMethodLoadStartExternalUsersSelfServiceSignUp/identityProviders/

for each segment we get:

singleton of type identity identityContainer
nav prop of type collection of authenticationEventsFlow
indexing into the collection
downcast to externalUsersSelfServiceSignUpEventsFlow
property externalUsersSelfServiceSignUpEventsFlow of the cast type, of type onAuthenticationMethodLoadStartHandler
downcast to onAuthenticationMethodLoadStartExternalUsersSelfServiceSignUp
property identityProviders of the cast type

Which to me respects OData semantics, type cast segments are added to navigation properties so properties of the cast type can be accessed. We're never accessing a property from the base type after a type cast segment.
The only thing from our conversion conventions is that we might be inverting the indexer and type cast segments.
Did I miss anything?

@irvinesunday I'm not sure I understand your comment about poor API design. Looking at this path with the metadata
/identity/authenticationEventsFlows/{authenticationEventsFlow-id}/microsoft.graph.externalUsersSelfServiceSignUpEventsFlow/onAuthenticationMethodLoadStart/microsoft.graph.onAuthenticationMethodLoadStartExternalUsersSelfServiceSignUp/identityProviders/

for each segment we get:

singleton of type identity identityContainer
nav prop of type collection of authenticationEventsFlow
indexing into the collection
downcast to externalUsersSelfServiceSignUpEventsFlow
property externalUsersSelfServiceSignUpEventsFlow of the cast type, of type onAuthenticationMethodLoadStartHandler
downcast to onAuthenticationMethodLoadStartExternalUsersSelfServiceSignUp
property identityProviders of the cast type

Which to me respects OData semantics, type cast segments are added to navigation properties so properties of the cast type can be accessed. We're never accessing a property from the base type after a type cast segment. The only thing from our conversion conventions is that we might be inverting the indexer and type cast segments.
Did I miss anything?

From the above clarification, to achieve the expected paths below:

GET /identity/authenticationEventsFlows/{authenticationEventsFlow-id}/microsoft.graph.externalUsersSelfServiceSignUpEventsFlow/onAuthenticationMethodLoadStart/microsoft.graph.onAuthenticationMethodLoadStartExternalUsersSelfServiceSignUp/identityProviders/
POST /identity/authenticationEventsFlows/{authenticationEventsFlow-id}/microsoft.graph.externalUsersSelfServiceSignUpEventsFlow/onAuthenticationMethodLoadStart/microsoft.graph.onAuthenticationMethodLoadStartExternalUsersSelfServiceSignUp/identityProviders/$ref
DELETE /identity/authenticationEventsFlows/{authenticationEventsFlow-id}/microsoft.graph.externalUsersSelfServiceSignUpEventsFlow/onAuthenticationMethodLoadStart/microsoft.graph.onAuthenticationMethodLoadStartExternalUsersSelfServiceSignUp/identityProviders/$ref
Question: Should the above path be:
DELETE   /identity/authenticationEventsFlows/{authenticationEventsFlow-id}/microsoft.graph.externalUsersSelfServiceSignUpEventsFlow/onAuthenticationMethodLoadStart/microsoft.graph.onAuthenticationMethodLoadStartExternalUsersSelfServiceSignUp/identityProviders/{identityProviderBase-id}/$ref
(i.e, indexing into the identityProviders collection before deleting the $ref)

The below annotations will need to be added:
<NavigationProperty Name="authenticationEventsFlows" Type="Collection(graph.authenticationEventsFlow)" ContainsTarget="true">
  <Annotation Term="Org.OData.Validation.V1.DerivedTypeConstraint">
    <Collection>
      <String>microsoft.graph.externalUsersSelfServiceSignUpEventsFlow</String>
    </Collection>
  </Annotation>
</NavigationProperty>

<EntityType Name="externalUsersSelfServiceSignUpEventsFlow" BaseType="graph.authenticationEventsFlow">
  <Property Name="onAuthenticationMethodLoadStart" Type="graph.onAuthenticationMethodLoadStartHandler">
    <Annotation Term="Org.OData.Validation.V1.DerivedTypeConstraint">
      <Collection>
        <String>microsoft.graph.onAuthenticationMethodLoadStartExternalUsersSelfServiceSignUp</String>
      </Collection>
    </Annotation>
    <Annotation Term="Org.OData.Capabilities.V1.ReadRestrictions">
      <Record>
        <PropertyValue Property="Readable" Bool="true" />
      </Record>
    </Annotation>
  </Property>
</EntityType>

<ComplexType Name="onAuthenticationMethodLoadStartExternalUsersSelfServiceSignUp" BaseType="graph.onAuthenticationMethodLoadStartHandler">
  <NavigationProperty Name="identityProviders" Type="Collection(graph.identityProviderBase)">
    <Annotation Term="Org.OData.Capabilities.V1.NavigationRestrictions">
      <Record>
        <PropertyValue Property="Referenceable" Bool="true" />
      </Record>
    </Annotation>
  </NavigationProperty>
</ComplexType>

The conversion lib will also need to be updated to retrieve complex properties of derived types. This issue will track this work: https://github.com/microsoft/OpenAPI.NET.OData/issues/437


prune reports missing data blobs

Output of restic version

restic 0.10.0 compiled with go1.15.2 on linux/amd64

I'm running this on Debian 10.6 ("Buster"), using an official Restic binary (as opposed to a self-compiled or distro-packaged version).

The following 6-line script:

#!/bin/bash
export RESTIC_REPOSITORY=b2:WWULibBackupSpecColl
export RESTIC_PASSWORD=[redacted]
export B2_ACCOUNT_ID=[redacted]
export B2_ACCOUNT_KEY=[redacted]
/usr/local/bin/restic prune -o b2.connections=50

Output is attached, with link in the Actual behavior section below.
I have not changed Restic versions recently, although the repository was created and early backups were done with an earlier version than I'm using now.
In case it matters: Backups are performed using Rclone's Restic REST server mode as the backend seen by Restic -- only Rclone talks directly to the repository in Backblaze B2 during backups.  In contrast, maintenance operations (forget, prune, check) are performed by the Rclone server (not the machine that's been backed up), directly against the repository on B2 without involving any version of the Restic REST server.

What backend/server/service did you use to store the repository?
Backblaze B2
Expected behavior
Prune the repository and report success.
This is both what I expect and want.

Actual behavior
The prune operation appears to finish building a new index (reaches 100%), and finding data used by existing snapshots (reaches 100%). Then it reported missing data blobs, and asked me to report the problem here.  Output is attached:  out.txt

Steps to reproduce the behavior
I ran prune again and got the same result, so presumably I can reproduce at will by running prune against this repository. (Perhaps check would also produce the same error, but I haven't tried that.)
This repository has been pruned and checked monthly for well over a year; this is the first time I've encountered this problem. The repository is approximately 2 TB.  I'm guessing that you might need access to this repository in order to reproduce this problem. (I can try to arrange that if you like.)

Do you have any idea what may have caused this?
The most obvious possibility is that the repository is truly damaged, although I don't know how that might have happened.
Do you have an idea how to solve the issue?
No, but if it would help I can try to build a debug-enabled copy of Restic.  Also, I can probably get permission from the data owner to let you examine the repository.
Did restic help you today? Did it make you happy in any way?
Backing up large amounts of data has been a headache for me for many years. Restic, plus Backblaze's affordable B2 storage, is by far the best solution I've ever found, and that literally makes me happy every day when I get my script-driven backup status report and remember that I'm not replacing failed or filled disks in my own miserable backup server any more.  Today I'm having my first serious problem with Restic, and although the problem per se is not making me happy today, it also highlights how trouble-free Restic has generally been. Restic is a tremendous gift -- thank you so much!

This could be same issues as in #3266. Please run restic cat pack --no-cache --quiet 46609e589ae9d0aa4f42c1175a16f5a8643482b219724492739d811cfb7695c6 > temp-46609e5[...] to download all pack files reported as incomplete pack file. Then use shasum -a256 temp-46609e5[...]  [...] to check the hash of these files. For files whose hash does not match the filename, please lookup these in the B2 web ui. The creation/modification time and the expected sha1 checksum would be the most interesting. Then compare these sha1 checksums to the output of sha1sum temp-[...].

Thank you very much Michael. It will take me some time to read issue 3266 and follow your advice, but I will report back when I've done that. It might be a few days.

Just popping my head in to say I'm still alive and still care about this issue. Today was earmarked to work on this in earnest, but now B2 is down  :-(
I can only work on this a couple days each week, so I apologize for slow progress. I'll be back with an update soon, for large-ish values of "soon".

The main investigation of this B2 issue is now in #3268. The current state seems to be that the problem reproduces without restic and B2 is currently investigating: https://github.com/restic/restic/issues/3268#issuecomment-780172480

I can now get to B2 again, so I finished the first test that I had ready to go. I downloaded the 6 packs that prune identified as incomplete. 4 of them have correct SHA256 hashes; 2 are incorrect.
If it would be helpful, I'd be happy to continue with the B2 web UI lookup, or other steps. But I wonder if this would really be valuable considering issue #3268 looks like it might be on the verge of a solution. Please let me know if you'd like me to do more.

I have some multi-terabyte repositories, so I was very excited to try the performance improvements in newer Restic versions. I upgraded to Restic 0.12.0 and tried both prune and check --no-cache on my problem repository and neither found any problems. Moreover, both took seconds or minutes instead of many hours  :-D
I realize this is a bad troubleshooting method, but it appears issue #3268 is essentially solved (if not formally solved yet), and I did not have much to contribute to its solution. I hope the fact that my problem now seems to be fixed supports the theory that #3268 is also essentially fixed.
A huge THANK YOU to Michael and the Restic developers. I will close this issue in a few days if there's no new activity.

@tomm87 Closing this issue, as I'm pretty sure that your prune issues were caused by the temporarily invalid pack files returned by B2. Feel free to reopen the issue is the issue resurfaces.
Btw, I wouldn't recommend using the --no-cache flag with check. The check command by default already downloads a fresh copy of all files into a new temporary cache. And when using a cache, check is a lot faster than without.


⚠ Bofrost.es has degraded performance

In 34fe56e, Bofrost.es (https://www.bofrost.es) experienced degraded performance:

HTTP code: 200
Response time: 1480 ms

Resolved: Bofrost.es performance has improved in 90b6fa3 after 6 minutes.


Add search widget feature

Add the search widget feature.
Business scenarios:

As a Customer I can open search widget by clicking the button on the top;
As a Customer, I can type searched product (phrases like name of the product, SKU number, product category etc.) and as the result, I will instantly see the grid of products in search widget;
As a Customer, I can modify searched phrase and the product grid will be instantly updated;
As a Customer, I can click any product on the grid in search widget and move to the product card.

The design of the view:

Probably need to user search action from  /store/modules/catalog.js

I can complete this task. What do you think about including https://community.algolia.com/vue-instantsearch/ as a secondary search service option?

@collymore Cool! We’re using elastic search so we do have search capabilities already included :) I mean - Algolia is kind of cool product for search but as we’re on MIT we shouldn’t have any paid services included by default. Would be great to prepare optional extension for Algolia that user can use as a inplace replacement for the defaults.
What do you think? We already have a mechanism for extensions so I believe if you add proper events/hooks or simply use Mixins to override the default search I would work.
But the default search must base on ES.
Please take a look at src/pages/Category.vue on how we’re building sophisticated ES queries using product vuex store and bodybuilder package for elasticsearch-JS
In case of any questions just ask!

Thanks for the info, that makes sense.
I'll get this done next week then on ES.
What's the best way to request other issues to be assigned to me?
Should I Comment on them or message on Slack?

Cool to hear that! Yes, just please comment them and put me @pkarw in the comment to notify me


Problem with unique ids

First of all thank you for this excellent project! It was suuuper easy to build beautiful and useful API docs for my project.
I noticed that slate makes equal header ids unique, but it doesn't work in all cases. For example this works:
## Foo

## Foo

Both foos get an unique id and I can link to them using #foo-<something>. However, this doesn't work:
## Foo

## $Foo

Both of these get the same id foo.

Actually I just noticed that I links to generated ids like [foo](#foo-18) don't work. The actual id of the header is still non-unique. There is some
 that is somehow used by the TOC?

Hey, thanks for reporting! I think header id generation happens in redcarpet, and IIRC they're aware of the problem. For now, if you want to manually fix it, you can always manually create a header:
$Foo
.


Release/v1.0.1

Assuming the v1.0.0 -> v1.0 change is intentional (as opposed to v1.0.1)

Correct. We decided that API docs will not change between patch releases and therefore don't warrant an iteration.


Could not take images

GET /v1/images/:file_key
Error
{"params":["file_key"],"query":["ids"]} are required.

Did you properly run the command (node main.js <FILE_ID_HERE><API_TOKEN_HERE>)? Also make sure that all the dependencies properly installed and that you can access the file fine through the web or desktop clients


Set Default ColorPickerType

is there a way I can set the default ColorPickerType?

but the wheel is opening by default

but I can't primary to open by default

i was using this that's why

Hi @imSaharukh, thanks for your question.
The picker will open on the page that has the color you pass to it when you open it. So if you have a box for selecting a color, and you click on it to open the picker, and pass that color it has to the picker when you open it, it will open the picker with the section (picker type) selected where it can first find the color you passed in, including any shade.
So if you passed it a color value that matched e.g. primary Red[200], that will be selected and the Primary picker shown.
If you pass it some custom color that does not exist in Material, Accent or in eg. custom colors you made for the custom tab, then it will always open the wheel and select the color there, that is if you have the wheel enabled.
If wheel is not enabled and there is no matching color on any of the enabled picker types, it will open the first picker type you have enabled.
BR Mike 😃


fixed pa scraper

I ended up using Chris's code instead of mine for most of it, but I changed the output_rows to a list and accounted for some order switching and missing values.
Also learned that Pennsylvania is a commonwealth.

Broadly speaking, what needed to change?

Writing a dict to csv was having some issues, so I converted it to a list and used the default write_rows_to_csv
Also, the dates were being placed in separate columns, so I put them in a comma separated string and placed them in one column.

Triggering tests by closing and reopening.

Getting a sphinx error on tests
40 scrapers found 21 scrapers have docs     state["source"] = module.__source__ AttributeError: module 'warn.scrapers.pa' has no attribute '__source__' make: *** [Makefile:127: tally-sources] Error 1 Error: Process completed with exit code 2.


Logitech+ options

I am not able to install logitech+ options after using bloatynosy... ia m not sure it is happening because of your scriot or because of ChrisTitusTechwinutil..... if possible guide me .

I'd say let's rule out Bloatynosy for now. Give the successor to Bloatynosy, Winpilot, a shot.


Clean more code smells

Proposed Changes

Clean more code smells.

:white_check_mark: Build concoct 0.1.119 completed (commit https://github.com/Concoctist/concoct/commit/ae13b64e0b by @ldilley)


Menus not showing… only Brackets and Window and no coding window!!!

Strange… Brackets is not showing any menus but the Brackets and Windows menus. Hence, no Debug selection! I deinstalled Brackets and all the related files, then reinstalled. Same problem. Also tried restarting the computer.
Also, I can open only one file, and it opens as though it's a browser, no coding window!!
Here's my computer info, and an attached screenshot:
Model Name:	MacBook Pro
Model Identifier:	MacBookPro11,4
Processor Name:	Intel Core i7
Processor Speed:	2.5 GHz
Memory:	16 GB
System Version:	macOS 10.12.1 (16B2657)

I guess you are opening html file using Shift+Click


Improve messaging when running a task with no workspace

When running the Run Task command from the menu, you are prompted with the following notification:

For new users, they might not have a concept of a "workspace" or what a "task" is in the context of VS Code. We should add a link that points to our Tasks docs for more information.
Additionally, our docs should have an explanation of how tasks work with workspaces.

Fix 6 months ago by https://github.com/microsoft/vscode/pull/71476

Verified this works now, would've expected an inline text link instead but this works:


Improve username behavior

Fixes #10149

It's a good idea to open an issue first for discussion.

[ ] Tests pass
[ ] Appropriate changes to README are included in PR

@googlebot I signed it!


Check for pointer validity when casting raw pointer to reference.

Requested feature: Kani should automatically check if a raw pointer is valid before any conversion to a reference, strengthening the verification of unsafe code.
Use case: This will help enable modular verification of safe abstractions. Kani detects invalid memory access only when the reference or dangling pointer is used.
Link to relevant documentation (Rust reference, Nomicon, RFC):
Test case:
struct Store<'a, const LEN: usize> {
    data: [&'a i128; LEN],
}

impl<'a, const LEN: usize> Store<'a, LEN> {
    pub fn from(var: &i64) -> Self {
        let ref1: *const i64 = var;
        let ref2: *const i128 = ref1 as *const i128;
        unsafe {
            Store { data: [&*ref2; LEN] }  // ---- THIS LINE SHOULD FAIL
        }
    }
}

#[cfg_attr(kani, kani::proof)]
#[cfg_attr(not(kani), test)]
pub fn check_store() {
    let val = 1;
    let broken = Store::<3>::from(&val);
    assert_eq!(broken.data.len(), 3)
}

Note: The harness above succeeds under playground run (version 1.75.0), but correctly fails under MIRI.

I.e.: The pointer is not dangling and the allocation size is greater or equal to the reference type.


Fortinet VPN leaked passwords

more info at https://therecord.media/fortinet-warns-customers-after-hackers-leak-passwords-for-87000-vpns/

Thank you!


Highlight background of current choose option

Kamiel had made this a while back, but unfortunately he lost the code for it.

This GIF also showcases the menu option switched for the newer RS look, but what I would mainly like to see is the background highlighted.

Just a note that this cannot be done without internal access.

High Detail choose option menu was added in 1.5.21, which works for me.


Automatic rush update --full

This is an automatic PR generated weekly with changes from running the command rush update --full

There looks to be some bug in code coverage report writing on Windows.  Going to wait for a few days to see if anything is fixed.

Logged https://github.com/istanbuljs/istanbul-lib-processinfo/issues/24


Custom relationship attribute and ajax search

Hello
Im using custom relationship attribute ex : full_name as described in docs . when creating bread im using full_name as display column , relationship is belongsToMany so , i have multiselect with ajax search . but when trying to search getting error
SQLSTATE[42S22]: Column not found: 1054 Unknown column 'full_name' in 'where clause' (SQL: select count(*) as aggregate from playerswherefull_name LIKE %asd%)
i cant use name or surname as relationship display column ( as you now there can be duplicates in names and surnames too )
see below images .
what can i do to fix it ?
`protected $translatable = ['first_name', 'last_name', 'text'];
public $additional_attributes = ['full_name'];

public function getFullNameAttribute()
{
    $locale = app()->getLocale();
    $firstName = $this->getTranslatedAttribute('first_name', $locale);
    $lastName = $this->getTranslatedAttribute('last_name', $locale);
    return $firstName . ' ' . $lastName;
}`

POSSIBLE FIX
add this json as relationship details {"model":"App\\Player","table":"players","type":"belongsToMany","column":"id","key":"id","label":"full_name","label_custom":"last_name","pivot_table":"poll_questions","pivot":"1","taggable":"0"}
and edit VoyagerBaseController@relation to
`    public function relation(Request $request)
{
$slug = $this->getSlug($request);
$page = $request->input('page');
$on_page = 50;
$search = $request->input('search', false);
$dataType = Voyager::model('DataType')->where('slug', '=', $slug)->first();
    $rows = $request->input('method', 'add') == 'add' ? $dataType->addRows : $dataType->editRows;
    foreach ($rows as $key => $row) {
        if ($row->field === $request->input('type')) {
            $options = $row->details;

            $skip = $on_page * ($page - 1);

            // If search query, use LIKE to filter results depending on field label
            if ($search) {
                $label = $options->label_custom ? $options->label_custom : $options->label;
                $total_count = app($options->model)->where($label, 'LIKE', '%'.$search.'%')->count();
                $relationshipOptions = app($options->model)->take($on_page)->skip($skip)
                    ->where($label, 'LIKE', '%'.$search.'%')
                    ->get();
            } else {
                $total_count = app($options->model)->count();
                $relationshipOptions = app($options->model)->take($on_page)->skip($skip)->get();
            }

            $results = [];
            foreach ($relationshipOptions as $relationshipOption) {
                $results[] = [
                    'id'   => $relationshipOption->{$options->key},
                    'text' => $relationshipOption->{$options->label},
                ];
            }

            return response()->json([
                'results'    => $results,
                'pagination' => [
                    'more' => ($total_count > ($skip + $on_page)),
                ],
            ]);
        }
    }

    // No result found, return empty array
    return response()->json([], 404);
}`

@emptynick , Given above is fix by @snowket works. This issue is also with belongsTo relationship.


SyntaxError: Expected token ')' - (use double quotes in XPath)

raising from user's email

I did a simple implementation to try to automate pubmed queries. But I encountered the following errors:
==============================
SyntaxError: Expected token ')'

  phantomjs://code/try1.txt.js:1229 in injectJs
  phantomjs://code/bootstrap.js:456
==============================

The code is as follows:
==============================
https://www.ncbi.nlm.nih.gov/pubmed/
click term
enter term as "redcap"
click search

read (//p[@class='title']))[1] to pubTitle1
echo pubTitle1
LIVE
==============================

Would it be because of the javascript version?

Try below - the quotes in XPath you can use double quotes, and there was an extra ) bracket in it. The XPath using double quotes for text is mentioned in readme but the readme page is very long, so people may miss that.
Also, try downloading this cutting edge version and overwrite your existing copy. This version will not restart your Chrome browser when the automation is running, and that is very helpful if user uses Chrome to search reference info and their session gets restarted every time automation runs.
https://www.ncbi.nlm.nih.gov/pubmed/
click term
enter term as "redcap"
click search

read (//p[@class="title"])[1] to pubTitle1
echo pubTitle1
live


Inventory#contains() and similar not implemented

Trying to check if an Inventory contains a certain item type or item stack returns a static "false". This is a really, really useful feature that needs to be implemented.

@Faithcaio thank you omg

hope you dont need "exact" contains which checks for the stack size too

That would be nice, but this will work for now. I'm not sure why I even needed contain ATM but I'll figure it out.

Well with a contains WITH amount you could check if a player has at least x of an item.
You can still do that now by comparing: inventory.query(stack).totalItems()


Feature/issue 1060 change credential type

Fixes # . Support extend RBAC authentication
Changes proposed in this PR:

change credential type from enum to string to support any credential type that control in the market
new EventAccessControl class to handle RBAC related api call
isConsumable to call RBAC api to validate event access

The only issue here seems to be that allow and deny list tests are failing in the marketplace flow, otherwise I'm happy to approve. Do you already know what the cause of the failed tests is?

isConsumable require barge to start up latest version of RBAC server


embedded: support class existentials with generic classes

For example:
protocol P: AnyObject {}
class C<T> : P {}

let e: P = C<Int>()

Generic class existentials are supported in embedded swift by specializing witness tables.
This PR enables the very basic generic class existentials. More work needs to be done to e.g. support protocol inheritance, etc.
Also, I refactored the VTableSpecializer: I moved the VTableSpecializer pass into MandatoryPerformanceOptimizations. MandatoryPerformanceOptimizations already did most of the vtable specialization work.
So it makes sense to remove the VTableSpecializerPass completely and do everything in MandatoryPerformanceOptimizations.
This is a follow-up of https://github.com/swiftlang/swift/pull/76572

@swift-ci test

@swift-ci test

@swift-ci please test

@swift-ci test

@swift-ci test


authors.md is dead

It’s currently dead. https://github.com/himynameisdave/postcss-plugins/blob/master/docs/authors.md
This commit killed it. https://github.com/himynameisdave/postcss-plugins/commit/b82337afaebbcc43fb69815c46f4259a215f8e00#diff-365b0403c12f88b9d3bbdfb3f3b077a4

@jonathantneal thanks, fixed now!


TODO

may be use https://grammy.dev/hosting/cloudflare-workers-nodejs to reconstruct it, after I can't bear the code.

fixed by #4

well, size
https://www.npmjs.com/package/grammy 1.08M vs https://www.npmjs.com/package/@codebam/cf-workers-telegram-bot 54.4 kB, but since
Total Upload: 647.83 KiB / gzip: 108.46 KiB
so telegramify-markdown 60kb, this 40kb, acceptable.


Update armeria-grpc to 1.17.1

Updates com.linecorp.armeria:armeria-grpc from 1.17.0 to 1.17.1.
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "com.linecorp.armeria", artifactId = "armeria-grpc" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "@monthly" },
  dependency = { groupId = "com.linecorp.armeria", artifactId = "armeria-grpc" }
}]

labels: test-library-update, early-semver-patch, semver-spec-patch, commit-count:1

Superseded by #124.


Crashing whenever i leave a server

After injecting or requiring autorun, leaving any server instantly crashes/closes my game. No warning no error.
The version i used is 0.7.0 latest release.

could try https://github.com/Vurv78/Autorun-rs/releases/tag/v1.0.0-beta and see if it crashes

Don't use 0.7.0. Get a build from the latest github workflow of 1.0.0-beta5

Also note 1.0.0 is switching the folder from sautorun-rs to just "autorun"

Closing since I closed the other issues for being fixed in the beta versions. https://github.com/Vurv78/Autorun-rs/actions/runs/1877444171. I'll make a release if I don't get to a stable version soon  (Only thing I want for 1.0.0 right now is 32 bit to be fixed.)


Add the ability to lock to a profile

As a checkbox in the settings or toggleable tile based on #23 the option to lock down the current volume settings would be appreciated.
This would prevent unintended changes outside the app, meaning applying a profile or manually changing values inside the app would be unaffected.
Locking down the volume may work by reapplying the current profile on value change.

Thanks for the idea.
That would probably require a background service with a constant notification in the notification bar.

As far as I can tell there already is a constant notification around for jumping into the app. Maybe that one can be reused?
Thanks for the useful app.

@Idrinth this thing is optional you can disable it using "notification widget" check-box. If profile freeze is enabled, the widget will be always there.
Thanks for your support :)


Add support for native Mysql blob types and fallback for binary

Hi,
This PR came from discussion here : https://github.com/cakephp/migrations/issues/359
There's no breaking change.
The extended behavior is the ability with MysqlAdapter to :

use binary or varbinary type with any length and get the best fitted column type
directly use tinyblob, mediumblob or longblob with no limit to get this column type and right length settled

In case of conflict between required type and limit, the limit is used to select the best fitted column type.
Aside this, I've also fix some tests that were relying on hard coded db name and CR/LF issues for string comparisons
Hope it can be useful and that I've not mistaken any guideline :smile:

There seems to be a conflict.

There's no conflict in my opinion. Previous commit simply replace hard coded db name with another one. There's no way to have a phinx_testing nor a phinx database settled in these tests apart config DSN config setup in phpunit.xml.
Im' pretty confident that my replacements should be kept as it uses DSN db name and will work in any setup.

It isn’t an opinion. git just requires merging master to your branch because of edits in the same file.

Not really, you've switched back to hard coded db name that will make tests fails in local dev setup if not choosing 'phinx' as db name.

Conflicting files
tests/Phinx/Db/Adapter/MysqlAdapterTest.php

Maybe you didnt make the changes on top of latest master?
You might want to rebase.

Conflict is from a PR merged after mine was submitted. I propose to leave hard-coded way :smile:

The merge looks good. It should use the new config array like you have it.

Some errors seem left:
There were 4 errors:

1) Test\Phinx\Db\Adapter\MysqlAdapterTest::testCreateTableWithForeignKeys
Undefined index: table_name

/home/runner/work/phinx/phinx/tests/Phinx/Db/Adapter/MysqlAdapterTest.php:207

2) Test\Phinx\Db\Adapter\MysqlAdapterTest::testAddPrimarykey
Undefined index: constraint_name

...

Those failures started with mysql 8.0.21 in other PRs.

Weird, master is green ( https://travis-ci.com/github/cakephp/phinx/builds/176148908 )
Maybe this PR might need another merge of master?

Thx @othercorey
I was wondering how the hell it may have been related to this PR.

This only started with the mysql 8.0.21 release. You can see my draft PR showing 8.0.20 builds.

Merging then, and we figure out to solve travis in the separate PR.


Plugin problem

if I render the following html, the 2. display($parms) call for data-sd="organizer.Person" returns an empty string. Any idea?

....

blabla

 ...

If I remove " .... " it works.
it looks as if the div structure does not correspond to the schema structure
if I, change,
  blah
to
   blah
it works! because the schema is switched back to "Event"

@peters59 thanks for the report

If I remove " .... " it works.

Could you provide an complete example of how do you use the semantics and the output you are expecting please? (or link to your code somewhere) From the example you've provided I can't understand where is the problem.
By the way you are using "address.PostalAddress"" at the end with two quotes.

it works! because the schema is switched back to "Event"

Yes in some cases you need to declare again the Type in order to switch back to the parent, take a look at this example please https://github.com/PAlexcom/PHPStructuredData#usage-example-1, the <?php echo $scope; ?> is declared twice as you will see.
Once the library changes the current Type to use it does not know when to change it back to the previous  type, this is the reason why you need to specify it manually.


[KBFS-1773] Fix race with enabling journaling and login

Initialize the journal even if getting the current
user info fails, which is needed to enable existing
journals. That way, when the user does log in, the
existing journals will be enabled then.

Ah, don't look at this quite yet...

Okay, this is ready for review now.


Add more flexibility to configure credentials

Hi,
Currently, you can only set a secret name and dbman assumes the keys are username and password.
To cover all possible usecase it would be great to have more flexibility on how to configure credentials :

inline username / password in the Database resource to allow using flux substitutes for example
secret & key in secret for username / password to allow custom key (eg. share the app secret and reuse the app's env vars)

Here is an example of what I have in mind :
---
apiVersion: dbman.hef.sh/v1alpha2
kind: Database
metadata:
  name: test
  namespace: default
spec:
  credentials:
    # assumes username / password keys
    secret: test-secret
  database_name: test
  database_server_ref:
    namespace: infrastructure
    name: apps
---
apiVersion: dbman.hef.sh/v1alpha2
kind: Database
metadata:
  name: test
  namespace: default
spec:
  credentials:
    # Inline username & password can be useful if you use flux substitute vars for example
    username: test
    password: "${SOME_FLUX_VAR}"
  database_name: test
  database_server_ref:
    namespace: infrastructure
    name: apps
---
apiVersion: dbman.hef.sh/v1alpha2
kind: Database
metadata:
  name: test
  namespace: default
spec:
  credentials:
    # manually specifying the secret name and key allows reusing an existing app secret
    usernameSecretRef:
      name: test-secret
      key: DB_USERNAME
    passwordSecretRef:
      name: test-secret
      key: DB_PASSWORD
  database_name: test
  database_server_ref:
    namespace: infrastructure
    name: apps

3e87d43cdc11aa735a4f5ec99f023a0279f2b666  starts addressing this

added docs in dcda17b4e5cb6cbd21cf64ccdad6724e01f39e36


HUD background not removed if animation does not complete, makes app unusable until restarted

https://github.com/jdg/MBProgressHUD/pull/366 already covers the issue I am seeing, but since I could not find a corresponding Issue, I figured I would report this.
In my case I have a tab controller and I can reproduce this by performing an action in one tab that would trigger the HUD and then switching to another tab before the HUD is properly dismissed. On navigating back to the tab with the HUD, the user may no longer interact with the view as the HUD background has not been removed.
This appears to be because removeFromSuperview is not being called because the finished is false.
In doneFinished:
    if (finished) {
        self.alpha = 0.0f;
        if (self.removeFromSuperViewOnHide) {
            NSLog(@"HUD removeFromSuperViewOnHide");
            [self removeFromSuperview];
        }
    }

This log shows the flow when the tab is switched before the HUD animation can complete:
2016-05-19 14:23:27.396 [9307:4270521] HUD hideAnimated: true
2016-05-19 14:23:27.397 [9307:4270521] HUD hideUsingAnimation started
2016-05-19 14:23:27.400 [9307:4270521] HUD hideUsingAnimation animation completed
2016-05-19 14:23:27.401 [9307:4270521] HUD hideUsingAnimation doneFinished: false
I don't see any benefit to checking for finished here, unless I am missing some functionality of the HUD.  Commenting out the check for finished appears to work for me.

I modify it to :
self.alpha = 0.0f;
if (self.removeFromSuperViewOnHide) {
NSLog(@"HUD removeFromSuperViewOnHide");
[self removeFromSuperview];
}

Should be resolved now with #366 merged.


Growing number of NioEventLoopGroup threads

Currently I am working on Redis Cluster interaction, so I tried out a couple of client modules: Jedis and lettuce. So far I prefer lettuce over Jedis in terms of handiness and robustness.
I, however, bumped into a strange behavior when I closed connection(s) via RedisAdvancedClusterConnection#close. I understand it creates new thread(s) for new connection handling, but it does not seem to be removed when the connection is closed. Below is the code fragment my colleague gave to me.
@Slf4j
public class Main {
    public static void main(String[] args) throws Exception {
        List<RedisURI> redisUris = IntStream.rangeClosed(7000, 7005)
                .mapToObj(i -> "redis://localhost:" + i)
                .map(RedisURI::create)
                .collect(Collectors.toList());
        RedisClusterClient redisClusterClient = new RedisClusterClient(redisUris);

        //noinspection InfiniteLoopStatement
        while (true) {
            RedisAdvancedClusterConnection<String, String> connection = redisClusterClient.connectCluster();
            String key = Long.toString(System.currentTimeMillis(), 10);
            connection.set(key, "foo");
            assert connection.get(key).equals("foo");
            connection.close();
            Thread.sleep(Duration.ofSeconds(5).toMillis());
        }
    }
}

Try trace the count of threads and see it grows.
3.3.1.Final and 3.3.2.Final both reproduce this issue.
I tried to see if they were reused after reaching some kind of threshold, but no luck.
$ jps | grep -F AppMain | cut -f1 -d' ' | xargs jstack | grep '^"nio' | wc -l
8
$ jps | grep -F AppMain | cut -f1 -d' ' | xargs jstack | grep '^"nio' | wc -l
10
$ jps | grep -F AppMain | cut -f1 -d' ' | xargs jstack | grep '^"nio' | wc -l
12

I also found a similar issue on the old repo, https://github.com/wg/lettuce/issues/34 .
Is it going to be fixed? Or is there any solution or workaround?

Hi @oklahomer
Threads in lettuce are bound to a particular RedisClient/RedisClusterClient instance (in 3.4 and 4.1 to ClientResources, but that's a new thing) and not to connections. Threads are not removed when you close a connection but when a client instance is shutdown. Please test your case with calling RedisClusterClient.shutdown() at the end.
The EventLoop threads are an infrastructure resource that is expensive to create and to remove so they are kept alive until the client resource is closed. A thread is not bound to a particular connection, but a connection binds to one EventLoop thread which means that one thread can be used for more than one connection. To answer your question: This behavior is by design to facilitate scalability.
wg/lettuce#34 is different because it's about daemon threads vs. regular threads. 3.4 and 4.1 are using daemon threads.

Thanks @mp911de for quick reply.
That should help.

@mp911de Just to make sure, let me ask a bit more.
With the code fragment above, we only have one client instance at all time. In the while loop, it calls connectCluster and try establish connection with the same setting every time. Based on my understanding about how client tries to reuse EventLoop threads, it should reuse the created EventLoop threads to minimize the resource cost. At least that's what I thought when I took a glance at AbstractRedisClient#getEventLoopGroup; reuse event loop group for the same connection point.
Am I right?
I think it's neat that client instance manages those threads' life-cycle, but got a bit confused to see how it's actually working.

The answer is yes and no.
The underlying facility that provides Threads to the client is the NioEventLoopGroup (for TCP connections). That's a Thread group with a max-capacity that is currently set to Runtime.getRuntime().availableProcessors() * 4. It's not a pool; Threads can be requested by different "users" such as Channels or custom events but are not exclusively used by that "user". The number of Threads cannot grow beyond Runtime.getRuntime().availableProcessors() * 4 for one RedisClient or one RedisClusterClient instance. Multiple instances of RedisClient create their own NioEventLoopGroups and can cause unlimited threads.
The thing of multithreaded EventLoopGroups is that the client has no (or only very little) control over the  Thread on which a particular thing runs on. There is an EventExecutorChooser but it does not work like a pool when a Thread is "free" that it is reused (see https://github.com/netty/netty/blob/43ebbc3fa065155fa67732b0cbd7c12843b0f3f7/common/src/main/java/io/netty/util/concurrent/MultithreadEventExecutorGroup.java#L234), but the Threads are selected in a particular order. This means that you probably end up with up to Runtime.getRuntime().availableProcessors() * 4 Threads after using the client a while.
There are two things you can do to limit Threads:

Wait for lettuce 3.4/4.1 with an implementation you can control the max number of Threads programmatically
You can set the system property io.netty.eventLoopThreads to specify the number of Threads, but you could also run  into issues if you use netty for other things in your JVM instance

Come back to me if you need further support.

Thanks again for detailed comment. I checked with the code fragment above and the number actually stopped at 32, where Runtime.getRuntime().availableProcessors() * 4 = 32. I'd steer away from modifying system property and wait for next version.
Thanks.


Support for events emitted in an arbitrary transaction

Similar to:

https://docs.openzeppelin.com/test-helpers/0.5/api#intransaction
https://github.com/rkalis/truffle-assertions#truffleassertcreatetransactionresultcontract-transactionhash

It would be great to have support for checking events that occur within a transaction hash to assert that certain events are emitted.
Using the current form await expect(trxRequest).to.emit(emitter, 'Event') fails with:
Error: invalid hash (argument="value", value=undefined, code=INVALID_ARGUMENT, version=providers/5.0.17)

When attempting to check an event was emitted if a transaction hash is returned instead of a receipt.
potential related issues:

https://github.com/EthWorks/Waffle/issues/196
https://github.com/EthWorks/Waffle/issues/119

Released in 3.2.2

Released in 3.2.2