Python: Chat service will always put the first message as system message, how to handle it with text completion?

Describe the bug
GPT4 has no text completion capability and I used it for single text completion at previous version, which was working fine. For the new beta version, the chat_history will always put the first message as system role instead of user role.

To Reproduce
            kernel.add_service(
                AzureChatCompletion(
                    service_id="azure_openai_chat4_service",
                    deployment_name=chat4_api_config.deployment_model_id,
                    ad_token_provider=get_bearer_token_provider(
                        DefaultAzureCredential(),
                        "https://cognitiveservices.azure.com/.default",
                    ),
                    endpoint=chat4_api_config.endpoint,
                )

        execution_settings = AzureChatPromptExecutionSettings(
            service_id="azure_openai_chat4_service",
        )
        query_expansion = self.get_semantic_function(kernel, "skills", "QueryExpansion")
        result = await query_expansion.invoke(
            kernel=kernel, input=condensed_question, settings=execution_settings
        )

Expected behavior
The first message should be user role. Or we should be able to configure the system message in config.json or skprompt.txt.
Screenshots
If applicable, add screenshots to help explain your problem.
Platform
OS: Windows
IDE: VS Code
Language: Python
Source: semantic-kernel==0.9.2b1
Additional context
Add any other context about the problem here.

Right now I figured a work arounds for single prompt shot is to add a dummy user message:

Wondering if this is the only way to do it? It adds extra redundancy

Hi @HuskyDanny, I am working on responding to your other post in the Q&A section, but wanted to respond to this here. Can you help me understand something important: the title of this issue says you want to handle a system message for text completion? In your code snippet you're create an AI service for AzureChatCompletion and not AzureTextCompletion. Is that intentional? Text completion does not deal with chat history, it takes a prompt only.

Hi @moonbox3, thanks for replying. So I am using gpt4 and gpt3.5turbo which only supports chat completion. But I have some functionalities like summary or query rewrite which will use these models like text completion. If I go to text completion class, the api will return not supported error. So I could only go with chat completion for these 2 models.
And I see the first message will be automatically set as system role, so I intentionally add chat_history and add one dummy user message, so that the chat_history will divide the prompt into system + user message and roles will be assigned correctly.
To illustrate:
Prompt:
Instruction: You are a summarizer to summarize the below input  ---> system message
{{$chat_history}} --->Add dummy user message
Input: {{$input}} ---> user message

Thanks for your reply, @HuskyDanny. Can you please try 0.9.4b1? We made the update to have the single message (without a chat history) to have the user role instead of the system role.

Hey @moonbox3, this is awesome to react this quick to fix it. I am trying with the new version. There are some breaking changes, I used to get over with it by the blog showing the summary and examples of these breaking changes. I am wondering if there is such one for this change? And what is the general recommendation for us to deal with the breaking changes? (Dedicated blog section for breaking changes etc..)

With the new version, I could not intentionally add system message. Both system and user message will be combined as a single user message:

The root cause seems to be the from_rendered_prompt of chat_history, where it failed to parse the XML prompt. This exposes internal logic on how to handle prompt, not sure if this is user's responsibility to construct the prompt with XML tags? so posting here to see if this should be fixed.

Hey @moonbox3, this is awesome to react this quick to fix it. I am trying with the new version. There are some breaking changes, I used to get over with it by the blog showing the summary and examples of these breaking changes. I am wondering if there is such one for this change? And what is the general recommendation for us to deal with the breaking changes? (Like dedicated blog section for breaking changes? etc..)

Since we're still in beta, there may be further breaking changes until we get to v1.0 in May. We're working fast to try and get the breaking changes out of the way ASAP; however, things are still in flux before we get to v1, when we don't want any breaking changes. We're putting more focus on code velocity right now, and we're doing our best to document larger changes as we go. We are working on updating the docs, and when we land at v1, the docs will reflect a healthy state.

With the new version, I could not intentionally add system message. Both system and user message will be combined as a single user message:
The root cause seems to be the from_rendered_prompt of chat_history, where it failed to parse the XML prompt. Should we have a XML invalid or special token removal so that the retrieved content will not interfere the flow?

You can use XML as you see to generate a prompt with a system tag. You can use a chat history that can be explicitly set with a system message, like chat_history = ChatHistory(system_message="<some message>") or chat_history.add_system_message("").
Could you help me understand why couldn't structure your prompt in the following way (as an example?
chat_function = kernel.create_function_from_prompt(
    prompt="""{{$chat_history}}{{$user_input}}""",
    function_name="chat",
    plugin_name="chat",
    prompt_execution_settings=req_settings,
)

history = ChatHistory()
history.add_system_message("You are a summarizer to summarize the below input.")

answer = await kernel.invoke(
        chat_function,
        user_input=user_input,
        chat_history=history,
    )

This gets parsed into two distinct messages that will be sent to the model -- one system and one user.

Yes, I did structure this way, in my case, one more piece is the CONTEXTS, which pulls the data from vector db.
So the prompt is like """{{$chat_history}}{{$contexts}}{{$user_input}}""". And I have confirmed the contexts's some special character could break the XML parse, like adding the extra "<>" in the content.
So by leveraging XML parse library, it introduces a point of failure especially for RAG system where the retrieval might have special characters for XML. Will the semantic kernel consider handling it?

Thanks for your reply, @HuskyDanny. Tagging @eavanvalkenburg so we can chat about this.

@HuskyDanny are you still having issues with this?

@HuskyDanny, haven't heard back from you in a while. Closing this issue. If needed, please re-open or create a new one. Thank you.


removing system prompt from message payload for bedrock

Title
removing system prompt from message payload for bedrock

PR Type
Bug fix, Enhancement

Description

Removed the inclusion of system prompts in the message payload for Bedrock to address compatibility issues.
Introduced a system_prompt parameter in _create_request_params to allow explicit handling of system-level prompts.
Updated the ask_call method to derive and pass system_prompt when context is provided as a string.
Commented out the previous logic in _prepare_message_payload that added system prompts to the payload.

Changes walkthrough 📝
Relevant filesBug fix

bedrock_client.pyRefactor Bedrock client to exclude system prompts and add system_prompt parameter.

py/genai_client/text_generation/bedrock_client.py
Removed the inclusion of system prompts in the message payload for Bedrock.  Added support for a system_prompt parameter in _create_request_params.  Updated ask_call to handle system_prompt derived from context.  Commented out the previous logic for handling system prompts in _prepare_message_payload.

+20/-15

💡 PR-Agent usage: Comment /help "your question" on any pull request to receive relevant information

Changelog updates: 🔄
2024-12-19
Changed

Updated Bedrock client to remove system prompts from message payloads and handle them separately.

to commit the new content to the CHANGELOG.md file, please type:
'/update_changelog --pr_update_changelog.push_changelog_changes=true'

PR Code Suggestions ✨

CategorySuggestion                                                                                ScoreSecurity

Sanitize the context value before using it to construct the system_prompt to prevent injection or formatting issues

Verify that context is sanitized or escaped when constructing the system_prompt to prevent potential injection or formatting issues.
py/genai_client/text_generation/bedrock_client.py [294-295]
 if context is not None and isinstance(context, str):
-    system_prompt = [{'text':context}]
+    sanitized_context = self._sanitize_context(context)
+    system_prompt = [{'text': sanitized_context}]

Suggestion importance[1-10]: 9
Why: Sanitizing the context value is crucial for security, as it prevents potential injection or formatting vulnerabilities. This suggestion addresses a critical issue and enhances the safety of the code.
9
General

Add validation for the system_prompt parameter to ensure it is of the expected type before adding it to the request parameters

Ensure that the system_prompt parameter is properly validated before being added to the params dictionary in _create_request_params to avoid potential issues with invalid or unexpected data types.
py/genai_client/text_generation/bedrock_client.py [217-218]
-if system_prompt:
-    params["system"]=system_prompt
+if system_prompt and isinstance(system_prompt, list):
+    params["system"] = system_prompt

Suggestion importance[1-10]: 8
Why: Adding validation for system_prompt ensures robustness by preventing unexpected data types from being added to the params dictionary, which could lead to runtime errors or incorrect behavior. This is a significant improvement in terms of code reliability.
8
Possible issue

Handle cases where system_prompt might be None to ensure consistent behavior in streaming responses

Ensure that the system_prompt is correctly handled in both streaming and non-streaming responses to avoid inconsistencies in behavior.
py/genai_client/text_generation/bedrock_client.py [322-323]
 response = client.converse_stream(
     **self._create_request_params(
-        messages, inference_config, guardrail_config, system_prompt
+        messages, inference_config, guardrail_config, system_prompt or []
     )
 )

Suggestion importance[1-10]: 7
Why: Ensuring that system_prompt is handled consistently, even when it is None, improves the reliability of the streaming response functionality. While not critical, this change enhances the robustness of the code.
7


Most recent version is actually outdated

On https://github.com/marketplace/actions/changelog-from-conventional-commits I see that v1 is supposed to be the "most recent version". Looking at this repo, I see that version 1.3.2 is actually newer. I do not know what or why, but it does feel that the marketplace should show the most recent version as 1.3.2 at this point.

No, they both point to the same commit (0ea1d18) and are identical. Because of how marketplace releases work, two releases must be created for the v1 and the full version number. As such, one will always be created after the other, although they are the same.


memory leak?

The following code creates a cyclic assertion. Should it??
var assertvanish = require('assertvanish');
io.sockets.on('connection', function (socket) {
	    // setup listeners for this socket
	    setup_socket_events(socket);
});

function setup_socket_events(socket)
{
	socket.on('disconnect', event_disconnect(socket) );
     // do more things...
}
function event_disconnect(socket)
{

return function() {
    console.log("disconnect");
	// do something with socket
	assertvanish(socket, 5000);
    }
}

apparently it certainly does :)
I didn't realize that the THIS keyword for all the socket.on events pointed to that socket, so I was passing in the socket object as part of a closure.
changing things to
socket.on('disconnect', event_disconnect);

function event_disconnect()
{
    var socket = this;
    // ...
 }

seems to have helped


How would you like to edit object translations ?

Hello Daniel,
In the last release (2.11.5), you asked where we would like to translate objects :
"How would you like to edit object translations, perspective membership, etc. in an Excel sheet?"
I don't know if you had already some replies but I described my point of view about translations in this blog post :
https://thebipower.fr/index.php/2020/07/05/translations-life-cycle-management/
It could be great to share opinions on this topic
Regards
Didier

Hi Didier,
Did you already try exporting/importing translations to the TSV format using the ExportProperties / ImportProperties methods as described here? Please let me know if you have any issues or have any feedback on this approach.

Hello Daniel,
Yes I checked the export properties approach but I think the file format will be more complex once I export several languages. There will be groups of 3 columns (name, description, display folder) for each language ? I would prefer to have name, description & display folder in different rows and one column by language. The script I wrote works in this way.

Have you tried the Tabular Translator tool, with a .json file exported from Tabular Editor? (Translations > Export translations...). In my opinion, this is probably the best option, if all you need is to edit object translations and no other properties.

I know there is an issue right now that the file does not include "empty" translations, making it difficult to work with the .json file by hand, but I think the Translator-tool should work just fine.

Yes, I tried Tabular translator. It works but it doesn't meet  some of the requirements I explained in this blog post :   https://thebipower.fr/index.php/2020/07/05/translations-life-cycle-management/ :

copy and paste
automatic translation
easy identification of new labels
easy identification of the labels needing a review after modification
splitting translations between translators with an appropriate level of security

and most of all, it doesn't help translating visuals labels like titles.

Closing this issue as no further additions to this version of Tabular Editor are planned in this regard.


rxui: Add WireUpControls to Avalonia.ReactiveUI

What does the pull request do?

This PR adds this.WireUpControls() extension method to Avalonia controls. WireUpControls is a ReactiveUI feature available in ReactiveUI.AndroidX package, and its main purpose is to associate controls that have a Name is XML markup with strongly typed mutable C# properties declared in the code-behind file https://www.reactiveui.net/docs/handbook/data-binding/xamarin-android/wire-up-controls
I assume having this extension method is reasonable until we build a source generator that will generate strongly typed x:Name directives. With WireUpControls, one could write the following code with typed ReactiveUI bindings:
public class SignUpView : ReactiveWindow<SignUpViewModel>
{
    // We are leaving these properties uninitialized.
    // this.WireUpControls() initializes them for us.
    public TextBlock UserNameValidation { get; set; }
    public TextBlock PasswordValidation { get; set; }

    public SignUpView()
    {
        AvaloniaXamlLoader.Load(this);
        this.WireUpControls();
        this.WhenActivated(disposables =>
        {
            // We can now access the properties because this.WireUpControls() initializes them.
            this.BindValidation(ViewModel, x => x.UserName, x => x.UserNameValidation.Text)
                .DisposeWith(disposables);
            this.BindValidation(ViewModel, x => x.Password, x => x.PasswordValidation.Text)
                .DisposeWith(disposables);
        });
    }
}

What is the current behavior?

Currently, there is no WireUpControls extension method in Avalonia.ReactiveUI support package.
What is the updated/expected behavior with this PR?

Now, the WireUpControls extension method exists in Avalonia.ReactiveUI support package.
How was the solution implemented (if it's not obvious)?

This implementation is based on WireUpControls for Android implementation.
Checklist

[x] Added unit tests (if possible)?
[x] Added XML documentation to any related classes?

I feel like this would contradict with the "zero reflection usage" Avalonia is striving for (https://github.com/AvaloniaUI/Avalonia/issues/2359)

@pr8x ReactiveUI bindings are using reflection under the hood anyway e.g. in PropertyBinderImplementation (used by Bind, OneWayBind, BindValidation and friends). However, these code-behind typed bindings are completely optional (and WireUpControls implemented in this PR is optional as well), so System.Reflection could be excluded from .rd.xml if these extension methods aren't used. this.WireUpControls() is just an extension method for the convenience of those who decide to use ReactiveUI bindings that use reflection anyway (at least until we build that SourceGenerator for typed x:Name references generation)

@pr8x ReactiveUI is a lost cause anyway, so in ReactiveUI integration packages one can do anything they wish as long as they refrain from System.Reflection.Emit which isn't really compatible with Xamarin.iOS.

Built a C# source generator POC that we could adopt after net5 official release. Guessing this would be Avalonia 0.11.x https://github.com/worldbeater/XamlNameReferenceGenerator


Exclude files from search based on filename extension

Add control to exclude certain filename pattern from the search, e.g. only search files with the suffix *f90 and *F2003 or ignore all files ending with *sh, *bash and *py.

4353e7c5715f19a278513e75c8d4abbe2e99037a added the option to excluded files based on their extension.

The option to exclude files should be documented in the README file.

The documentation has been added to the README file.

Completed


[5.4] Add helper function 'me()' as an alias for 'auth()->user()'

I'm taking a queue from the common API endpoint: /user/me
I think me() is an appropriate metaphor for the concept of auth()->user()
I think people would love to start using this.
Thanks for taking a peek!

Caleb

I used this helper but with name user :)

It seems that 'me' makes sense in the context of an endpoint, as the end user will be interacting with it. A core 'me' helper seems to lose its meaning though.

This seems pretty easy to define in any project where you want it, especially since there's already an example in the comments of someone who did the same thing but preferred a different name.

I feel like signedIn or loggedIn would be a better name, because it makes more sense to reason about the code from the point of view of the interpreter, not the end-user. For example:
if(signedIn()->subscriptionExpired()) {
    signedIn()->notify(new SubscriptionExpired);
}

It makes more sense to say:

If the signed-in user's subscription has expired, notify him of that.

instead of

If my subscription is expired, notify me.

Nevertheless, I like this idea.

I myself had this dilemma, I use current all over the place.
This is not going to be merged, cheers.

I dunno this is pretty ballsy, lol. If anything I would prefer user.

@taylorotwell yea, user would be best with an exception that if you list multiple users you "want" to reserve user... foreach users as user.


Autorest Regen Preview to 3.0.0-alpha.20231019.2 by Dapeng Zhang from refs/pull/3852/merge

Triggered from https://github.com/Azure/autorest.csharp/pull/3852

API change check
API changes are not detected in this pull request.


Simplify RomTable handling so we get some proper data output

Signed-off-by: Daniel Egger<EMAIL_ADDRESS>

LGTM


[BUG] HTML ID should not start with a digit to be CSS compliant

Describe the bug
The problem is that the HTML ID generated for each element can start with a number, which is not CSS compliant, making it impossible to customize the element's style.
The current workaround is to use the in-browser element inspector and generate duplicates of an element until they get an ID starting with a [a-zA-Z] character.
Product Flavor

[x] Managed (https://rxresu.me)
[ ] Self Hosted

To Reproduce

Try to change the CSS for an element through the right sidebar by specifying the ID.
Expected behavior
IDs should not start with a digit.
Desktop (please complete the following information):

OS: Manjaro Linux
Browser Firefox
Version 101.0.1 (64-bit)

Cheers for the awesome tool anyway.

Should be fixed now :)


could mpiexec works on a gpu?

I run the cmd 'mpiexec -n 16  python scripts/image_train.py XXXXXX' while only has one gpu. and got some error. have  anybody tried it with one gpu?

@huangyehui using mpiexec -n 1 to run image_train,py if you only has one GPU

thanks a lot!

@huangyehui using mpiexec -n 1 to run image_train.py if you only has one GPU

@huangyehui using mpiexec -n 1 to run image_train.py if you only has one GPU


Keep viewBox attribute on optimized SVGs

Closes #1759
This was introduced with #1745, svgo defaults strip-out the viewBox attribute, which is functional and not a safe optimization to blindly apply.
For reference: https://developer.mozilla.org/en-US/docs/Web/SVG/Attribute/viewBox

When an SVG contains a viewBox attribute (often in combination with a preserveAspectRatio attribute), a transform stretches or resizes the SVG viewport to fit a particular container element.

Preview fixes issue #1759


fix_rotate.py

The way it was, you couldn't pass in a rotation higher than the length of the string doubled without getting a blank string in return. This way allows you to pass in any positive integer and get a result.
Also,
The last line of the description said that it took 2 strings and returned a boolean. This is not correct. The first line of the comment is correct.

Nice fix, could you update the algorithm description, please?


gdal: 3.3.2 -> 3.3.3

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/OSGeo/gdal/releases.
meta.description for gdal is: "Translator library for raster geospatial data formats"
meta.homepage for gdal is: "https://www.gdal.org/"
meta.changelog for gdal is: ""
Updates performed

Version update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/gdal_proximity.py passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/.gdal_retile.py-wrapped passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/gdalmdimtranslate passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/.gdal_polygonize.py-wrapped passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/gdalinfo passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/.gdal_proximity.py-wrapped passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/gdalmove.py passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/.gdal_pansharpen.py-wrapped passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/pct2rgb.py passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/.gdalmove.py-wrapped passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/gdal_create passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/gdalsrsinfo passed the binary check.
Warning: no invocation of /nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/.gdalattachpct.py-wrapped had a zero exit code or showed the expected version
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/nearblack passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/.gdal_merge.py-wrapped passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/gdal_edit.py passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/gdalbuildvrt passed the binary check.
Warning: no invocation of /nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/ogrtindex had a zero exit code or showed the expected version
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/gdal_retile.py passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/gdal_calc.py passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/.rgb2pct.py-wrapped passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/gdaladdo passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/.gdal2xyz.py-wrapped passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/gdal2tiles.py passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/.pct2rgb.py-wrapped passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/.gdal_calc.py-wrapped passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/rgb2pct.py passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/gdalwarp passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/gdal_contour passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/.gdalcompare.py-wrapped passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/gdal_viewshed passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/gdalenhance passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/gdal_rasterize passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/.gdal_sieve.py-wrapped passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/gdal2xyz.py passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/gnmanalyse passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/gdal_polygonize.py passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/ogrinfo passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/gdal_translate passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/ogr2ogr passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/gdalmanage passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/gdal_grid passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/.gdal_edit.py-wrapped passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/.gdal2tiles.py-wrapped passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/gdaldem passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/gdalmdiminfo passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/gdal_pansharpen.py passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/gnmmanage passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/gdal_sieve.py passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/ogrmerge.py passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/ogrlineref passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/gdalcompare.py passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/gdaltransform passed the binary check.
Warning: no invocation of /nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/gdalattachpct.py had a zero exit code or showed the expected version
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/gdallocationinfo passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/gdal_fillnodata.py passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/gdal_merge.py passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/gdal-config passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/.ogrmerge.py-wrapped passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/gdaltindex passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/testepsg passed the binary check.
/nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin/.gdal_fillnodata.py-wrapped passed the binary check.
42 of 62 passed binary check by having a zero exit code.
58 of 62 passed binary check by having the new version present in output.
found 3.3.3 with grep in /nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3
found 3.3.3 in filename of file in /nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3

Rebuild report (if merged into master) (click to expand)

133 total rebuild path(s)

37 package rebuild(s)

37 x86_64-linux rebuild(s)
34 i686-linux rebuild(s)
28 x86_64-darwin rebuild(s)
34 aarch64-linux rebuild(s)

First fifty rebuilds by attrpath
cloudcompare
gdal
gmt
gplates
grass
libLAS
mapcache
mapproxy
mapserver
merkaartor
openorienteering-mapper
openspace
paraview
pdal
postgresql11Packages.postgis
postgresql12Packages.postgis
postgresql13Packages.postgis
postgresqlPackages.postgis
python38Packages.cartopy
python38Packages.gdal
python38Packages.osmnx
python38Packages.pygmt
python38Packages.worldengine
python39Packages.cartopy
python39Packages.gdal
python39Packages.osmnx
python39Packages.pygmt
python39Packages.worldengine
pytrainer
qgis
qgis-unwrapped
qlandkartegt
qmapshack
saga
sumo
udig
worldengine-cli

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A gdal https://github.com/r-ryantm/nixpkgs/archive/78f3d7f2813ac25d7639111fb996f727627a1fe1.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3
ls -la /nix/store/zsxqjml0xrqpqxps0xj1ayj6v8mbngm3-gdal-3.3.3/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
nixpkgs-review took longer than 45m and timed out

Maintainer pings
cc @marcweber for testing.

Upstream is at v3.4.0

Upstream is at v3.4.0


Don't throw uncaught error on mdns connection

The mdns connection handler throws an error if the socket is not connected, which will result in an uncaught error. Should handle this gracefully without an uncaught error.

This refers to code that no longer exists.


[mitmweb] Combo all cookie headers to one Cookie header

Problem Description
Usually, like Chrome, send a web request with only one cookie header to the server, but all cookie headers are split one by one in the mitmweb.
When exporting as curl and paste to Postman with raw text, Postman only imports one cookie header and value.
Proposal
Combo all cookie headers to one cookie header is more compatible with other apps, like Postman, etc.
Alternatives
Just combo all cookie values with '; ' into one Cookie header.
Additional context

Thank you for the suggestion!
Maybe Chrome shows something else, but we are displaying the cookie headers exactly as they are sent over the wire. I don't want to change that as sometimes these exact details matter.
If you would like to see the header concatenated, you can instruct mitmproxy to do it like this (see https://docs.mitmproxy.org/dev/api/mitmproxy/http.html#Headers):
concat-headers.py
def requestheaders(flow):
    flow.request.headers["cookie"] = "; ".join(flow.request.headers.get_all("cookie"))

And then run mitmweb -s concat-headers.py. Does that help?

Yes, this just helps me.
Thanks very much!

I improve the script to fix request error when the request without cookie header.
concat-headers.py
def requestheaders(flow):
    cookies = flow.request.headers.get_all("cookie")
    if len(cookies) > 0:
        flow.request.headers["cookie"] = "; ".join(cookies)


Add support to customize the scope

added support for optional scope attribute for #50

Other option is to possibly update this PR to make the scope option empty be default and only add it if it has a value.

@mr-menno do you think you can add a test for this? a simpler one creating the strategy with a custom scope and check the scope is applied.

Honestly, I'm not good at writing tests.

Support for this was implemented in #89


Default call to net_if_up() will always fail because device is not ready

In my modem driver i define a network interface using NET_DEVICE_INIT macro.
After pulling from the main branch (on which i was already) i noticed that the interface associated with the net device is not up by default as it used to be, the call to net_if_up() is still done automatically but now it fails.
After some debugging i found that the issue is that inside net_if_up() there is a check on the fact that the device associated to the iface is ready (in subsys/net/ip/net_if.c at line 4486), this is always failing because the init function of the device (that sets the initialized flag to true) is called after net_if_up().
I am not sure which change caused this behavior.

It is probably this change d8ec9118b48232aecb285f31dea651f5847d32d2

this is always failing because the init function of the device (that sets the initialized flag to true) is called after net_if_up().

The device should be initialized before calling net_if_up()

@jukkar yes, i agree that the commit you linked it's what caused this behavior.

@jukkar should this be fixed within Zephyr?
I am just trying to understand if i am doing something wrong and need to fix my driver. If that's the case i am not sure how to control when the device gets initialized as at the moment the calls are triggered automatically because of the NET_DEVICE_INIT macro.

@AlessioLei94 do you have your driver somewhere it could be reviewed?
I do not quite understand what you mean by "calls are triggered automatically because of the NET_DEVICE_INIT macro". There should be no way that net_if_up() is called from action in network device init function.
The net_if_up() is called if network interface flag NET_IF_NO_AUTO_START is not set during the device booting. But at point the network device driver should be already set up properly in the system.
What network technology your driver is, Ethernet or something else?

@jukkar I mean that the device and the net_if are initialized and the net_if is enabled just by having the NET_DEVICE_INIT macro in my driver file to define the network interface that is part of the modem driver. As you also said the net_if is enabled because i do not set the NET_IF_NO_AUTOSTART flag.
I found the way to fix this behavior in the driver tho, i see that the net_if is initialized and then started at POST_KERNEL level with priority 80 (not set by me). While the priority passed to the NET_DEVICE_INIT is used to initialize the device associated to the net_if, again at POST_KERNEL level (this is not configurable). So just by giving a priority < 80 i get the same behavior as i did before, so the net_if is initialized and properly enabled.
It was just a bit tricky because i had to find out at what level the net_if was initialized.

@jukkar Sorry for the confusion, i managed to understand how it works now.
In my driver i define a modem device and a network interface that is controlled by the modem. I just need to make sure to have the network interface to have higher init priority (lower number) compared to the modem device and it will work properly.

Great, I will close this one then.


Hide keras model so that clone function from sklearn works

There are actually several changes I've made. I believe they can be useful for other people as well.
Hide keras model within the wrapper so that sklearn clone function works
It is essential that the sklearn.base.clone function works so that keras can be used with sklearn functions, such as BaggingClassifier or GridSearchCV, etc. Currently, the clone function has issues with keras. The way that function works is that it calls the get_params method from the classifier, and then does it recursively to each of the parameters in case that method is present. If not, it just deep copies the parameter.
It turns out that sklearn should only see the get_params from the wrapper class, but not the get_params from the keral model itself, as the latter doesn't follow the sklearn API (and it is not supposed to). By hiding this model inside a Hideout class, we can fix this issue.
Add a class_weight = 'balanced' (as in sklearn)
This way, when calling fit, the class_weight will be computed from the class frequency.
Created a decorator do use in fit(X, y) and score(X, y)
This code was already in the fit function. However it should also be in the score function. To avoid repeating it, I created a decorator to do the job.
Recall, this code turns a vector y into a one hot encoded output in case the loss_function is 'categorical_crossentropy'
Some minor fixes
There were some other minor changes, like moving some Classifier specific code from the BaseWrapper into the KerasClassifier, and some other minor stuff.
I've left the KerasRegressor code intact.

I've added some tests in a different branch: https://github.com/jnlaia/keras/blob/sklearn_tests/tests/test_sklearn.py
Some of those tests require sklearn to run. Let me know if there is interest in adding them to the main repo.

@jnlaia
It seems like it doesn't correctly work with sklearn.calibration.CalibratedClassifierCV()
Any insight?

You can move here any test that doesn't require sklearn.

In this PR I've avoided making any decision regarding the scikit learn API. Instead, I've build upon the one that was already in keras, and made some small changes to fix it. As it is now, keras scikit learn API is broken.
I still think it would be helpful to have, at some point, a deeper discussion on this API. For example, wouldn't it be nice to have some of the parameters from the class attribute "model" exposed to the outside, so that we would be able to do grid searches on them as well? As it is, it is not immediate to do a grid search where one changed the dropout or the geometry of the neural net. Changing the current API is outside of the scope of this PR though.

I will only be able to implement the changes you suggested by the end of the week that is now starting, or the following one.

Closing outdated PRs following the release of Keras 1.0.


add shutdown delay

waiting on green process to done, avoid client 408 error

Thank you for your PR, but I think the Sever::shutdown_delay method is not general enough. It seems that it can only use std::process::exit to force the program to exit after receiving the interrupt signal.
I think you can refer to https://docs.rs/warp/0.3.1/warp/struct.Server.html#method.bind_with_graceful_shutdown to let users customize the shutdown conditions, which is better. 🙂

Refactoring it like this:
#[tokio::main] async fn main() {     let app = route().at("/hello/:name", get(hello));     let server = Server::bind("<IP_ADDRESS>:3000").await.unwrap();     server.         graceful_shutdown(|| {             println!("i'm done")         }).         run(app).await.unwrap(); }


feat(ci): publish charts to gh-pages on master commits

adds build and publish jobs to brigade.js, intended to run on pushes to master
renames docs/ to dist/ as directory to place chart artifacts and index.yaml
once this is proven and charts are being successfully served from https://github.com/Azure/brigade-charts/tree/gh-pages, we can issue a follow-up PR to remove dist/ and untrack from version control

Closes https://github.com/Azure/brigade-charts/issues/28

Brigade-related question for the pros (@radu-matei @technosophos): Is there currently a way to prevent brigade builds for branches such as gh-pages that may only serve artifacts -- or, more pertinently, will not host the brigade.js file that exists on the master branch?
Currently, builds are generated because we have repo-wide webhooks (courtesy a GH App) but these will fail right off the bat for pushes to gh-pages, as no brigade.js exists to use.


[QUESTION] Is it possible to use radamsa as a library?

Hi!
I would like to embed radamsa in a few different places as a library instead of having to call a binary on the command line from my own fuzzers. Is there a (recommended) way of doing so?
Thanks in advance!

Hi,
You are probably thinking about a scenario where the linked radamsa would constantly get something to mutate? Supporting that kind of use is something I've been planning to add for a while now.
You already can start radamsa with a fixed set of samples to serve a stream of fuzzed data (radamsa -n inf -o 31337 samples/*), and embed a trivial wrapper which just grabs the next testcase from localhost:31337. The trouble is that that you can't extend the sample set on the fly this way.

Yes, something similar. Right now I'm using my own port of this: https://github.com/trailofbits/grr/tree/master/third_party/radamsa
...and I have written a Python wrapper for it. I have also used the socket mode for it but... is not actually what I want.

Grr seems to use a fun approach. The only issue I have with it, is that when you run radamsa once for each output it doesn't get a chance to collect data about the inputs. Therefore some mutations which may be useful will never occur. There should be some way to either pass state between the runs, or run the radamsa in a separate process and have the state there, which is why the TCP mode was originally added.
A few solutions would come to mind:

add an incremental mode to radamsa, so that it can store information gathered so far between runs, after which something like the current wrapper would work better
extend the TCP mode to handle the kind of use required in a library setting, and maybe bundle functions along which start a background radamsa automatically and call it
the same, but with stdio redirection to background process
add support for librarization upstream to owl, and get libradamsa as a result

Are there some issues why the background process approach doesn't work well in your test setups?

The main reason is performance. There is no comparison between using an in-memory mutation engine in N machines (running independently on each one) than using network sockets. The other one is that, often, the same mutation engine needs to use different data sources, thus, making it required to open a listening socket (i.e., a radamsa listening instance) for each and all formats I want to fuzz. An easy example: PDFs. They have a lot of different "formats" being used inside a single PDF document.

The last one is probably the correct solution. Owl should have some builtin support for building programs to be used as libraries in other C-programs. Then it would be possible to run radamsa incrementally without losing state from within one process.

Maybe you want to take a look at this: https://github.com/trailofbits/grr/tree/master/third_party/radamsa

Current plan: radamsa (and owl programs in general) work by decoding the program to run from a fasl image, encode the command line parameters as a corresponding lisp object, run the program on that data and return the likely integer value the program returns. When used as a library, the programs could have a boot/init function to decode the image, and you could then correspondingly have a lisp object -> lisp object library call for any compiled function, which automatically en- and decodes the object. This way the same heap state could be used, and the library function being called would even remain a purely functional one with state.
In practice, you'd link a libradamsa and have something like radamsa(void *ptr, size_t s, &result, &result_size).

That would be perfect!

That would end up working without radamsa-specific modifications in the planned solution, because you'd initially boot up the embedded radamsa anyway with a fake command line, on which you can give the seed and other settings as usual.

https://github.com/aoh/owl-lisp/issues/15 is waiting for spare time. I got DoS'd by various kinds of extra work in December.

I understand, don't worry :)

Oh, by the way, https://github.com/aoh/ni might be of interest here. It's a quick port of some radamsa-mutations to C, which should be easy to embed.

Thanks! But doesn't look like comparable. It seems "ni" doesn't try to infer the grammar from the inputs.

@aoh Could you explain this

That would end up working without radamsa-specific modifications in the planned solution, because you'd initially boot up the embedded radamsa anyway with a fake command line, on which you can give the seed and other settings as usual.

If we do radamsa seed.pdf -o mutated.pdf every time for each seed, are we missing on radamsa specific mutation?
What about running radamsa -r seeds/*.pdf -o mutated.pdf every time we want a mutated sample, are we still missing on radamsa specific mutation?

In both cases yes, though not by much. Some mutations are only possible if radamsa has had a chance to look at another file, or the same file from a different position. If you have sample files with '' and '', then the first output will never have something like '', because radamsa only learned about one of the attributes while generating the first fuzzed output.
As a workaround, if it's not easy to make sets of files at a time for testing, you can add --seek 2 especially in the latter case to your existing test scripts to allow more cross pollination between sample files. This won't be necessary after issue #24 is fixed, but you still should consider making sets of files at a time to allow radamsa to filter out duplicate testcases.

@aoh thank you very much for the explanation. According to you what could be the best way to run radamsa when I have like n seeds in a directory seeds and I want to get full benefits of radamsa mutation and generation considering that I need to get the mutated sample by giving a seed file and also the recursive mode in radamsa

In both cases you could generate a bunch of files and serve them as they are needed. Something like
# make a file for fuzzed files if necessary
mkdir -p fuzzed
# check if more files need to be generated to fuzzed
ls fuzzed | grep -q radamsa || radamsa -n 1000 -o fuzzed/radamsa-%n.out seeds/*
# give the next file
mv "$(ls fuzzed | head -n 1)" mutated.pdf

First tests of calling some compiled lisp code from c passed. The code looks roughly as what was planned above https://github.com/aoh/owl-lisp/blob/develop/c/lib.c#L49
Next step is to add a suitable function to radamsa and try it out from C using some similar wrapper.

Can't wait to have a working version. Do you have an estimated date to start testing it?
Thanks a lot!

need it as library too... hope to see this as soon as possible

In case this is still relevant, fix is mostly done at https://gitlab.com/akihe/radamsa/issues/28 . Next versions will likely have a libradamsa.c and radamsa.h.

That's awesome! Thank you very much!

Fantastic!!

Thank you!


Add missing semicolon to weechat example

Thanks! Could you prefix your commit message with doc:  so it's obvious from the commit message what it touches?

fixed commit title by using the squash option in the github interface.

@Mic92 While it's trivial to take care of yourself I lke to ask the commiter to do so.
Not only will it lead to subsequent PRs of higher quality but using the github web ui rewrites the commit history and any signatures are removed from the commit.


Incorrect computation of guest memory

Describe the bug
kraft run incorrectly interprets memory suffixes G and T to be 1000M and 1000000M, respectively.
The correct values are 1024M and 1048576M, respectively.
A similar issue exists for larger suffixes, but those are outside the realm of feasible RAM sizes today.
This is due to a confusion between SI and IEC units, see below for a detailed discussion.
Steps to reproduce
kraft run --plat qemu --log-level debug -M 1G on any unikernel image and observe the -m size=XXX argument in the resulting qemu commandline.
Expected behavior
Expected -m size=1024M for -M 1G and -m size=1048576M for -M 1T.
Extended context
TL;DR
Memory works best handled with IEC (base-2) suffixes due to how it's built and accessed at the hardware level. Using SI units will lead to memory fragmentation and inability to use cool performance features like hugepages.
Long explanation (grab a coffee):
Historically there has been much confusion about how to correctly express multiples of bits and bytes. Early computer engineers inconsistently piggy-backed on top of metric prefixes: sometimes 'kilo' meant 1000 (base-10; 10^3), sometimes 1024 (base-2, 2^10), whichever was most convenient for the task at hand -- some things are naturally base-2, some not. The marketing department probably got involved at some point too.
This confusion only got exponentially worse when larger multipliers appeared ('mega', 'giga', and so on), with absolutely no standard as to what each meant. 1M could mean 1000 * 1000 (e.g. 1 Mbps in networking), or 1024 * 1000 (e.g., 1.44MB floppies), or 1024 * 1024 (a 1MB file as declared by Windows). This became progressively less sustainable, as the relative difference between interpretations grew ever larger. Misinterpreting 1K is only a 2.4% error; for 1T can be as much as 10% -- enough for non-techies to notice and vocally complain, possibly to lawyers.
To solve this issue once and for all, two systems are agreed upon for measuring multiples of bytes:

SI (metric): kB, MB, TB, PB, EB, etc., representing powers of 1000
IEC: KiB, MiB, TiB, PiB, EiB, etc., representing powers of 1024

Note that case is important here! There's no such thing as KB; it's either kB or KiB (just like we have kg and km, not Kg and Km).
So why use one over the other? Common sense says to stick to what's most widely known: metric prefixes, and indeed most user-facing software has made that transition for storage media (with the notable exception of Windows shell, likely for obscure backwards-compat reasons). This makes sense as the size of a storage medium is not directly determined by powers of two. Networking too uses SI prefixes since forever, because telecommunications often don't care about powers of two either.
Where things change greatly is RAM -- the one place where IEC units are not only preferrable, but often mandated by hardware design and performance considerations. Without getting too deep into details, DRAM, even though it appears as one contiguous byte-addressable array (mostly), is actually addressed using a higher-dimensional tuple of addresses: <channel, DIMM, rank, bank, row, column>, with such a tuple addressing a memory word (which in DDR* is 64 bits).
The smallest usable unit of DRAM memory is the bank which consists of exactly 65536 rows by 1024 columns of 64-bit memory words. JEDEC DDR standards further impose the number of banks a DRAM device (or rank) must provide -- DDR3 has exactly 8 banks, DDR4 16. All DRAM of a particular standard has the exact same geometry. The way you get more DRAM is by pooling multiple devices in parallel, then MUXing between the lot: two ranks on the same DIMM with a rank select bit, two DIMMs on the same channel w/ a DIMM select bit, two channels on the same controller addressed independently, etc.
This MUXing operation further entrenches powers of two as the natural addressing mode of DRAM -- indeed most RAM configurations in the wild are pure powers of two for this exact reason, and the rare examples that aren't must perform various addressing tricks inside the memory controller in order to cleanly map physical DRAM into a linear address space.
AMD used to publish their memory controller mapping functions for microarchitectures prior to zen; feel free to look them up for a concrete example of this.
Besides all that we must consider how RAM is typically accessed in a modern system: paging. Because page translation of a virtual address must be done as fast as possible in hardware, pages too are naturally aligned on power-of-two boundaries so that bit-slicing may be used instead of more complex arithmetic. This goes doubly so for hugepages.
Allocating without regard to natural page boundaries can lead to wasted unaligned memory, or can prevent the use of more performant hugepages altogether.
Summing all up: DRAM is aggressively parallelized as well as addressed in powers of two, and conscientious allocation of memory resources should also follow powers of two if one wants to make the most of memory performance.
Which architectures were you using or does this bug affect?
No response
Which operating system were you using or does this bug affect?
No response
Relevant log output
No response

Thanks for the deep dive; super insightful!
I think this is due to kraftkit using Kubernetes apimachinery code for quantity handling, see:
https://github.com/kubernetes/apimachinery/blob/0d057e543013c79e20abb000df19bc16d286b777/pkg/api/resource/quantity.go#L100
The behavior you observed stems from the way Kubernetes handles memory units. Kubernetes distinguishes between SI and IEC units. For memory:
SI units like M, G, T are base-10. So, 1G in Kubernetes is actually 1,000,000,000 bytes.
IEC units like Mi, Gi, Ti are base-2. Hence, 1Gi is precisely 1,073,741,824 bytes.
So to get 1024M in the generated qemu command, you'd have to use 1Gi.
I believe this mimics Kubernetes behaviour.
Perhaps it would be a good idea to document this in the memory CLI flag.

Ok, I will add a better description with examples for this


How to retrieve images with different resolutions?

Hello there, in the docs it is stated that:

thumbnail_url: The Media's thumbnail image URL. Only available on VIDEO Media.

Does this mean that if I want to retrieve the feed images in a lower resolution I simply cannot?
I haven't found a way to retrieve the images in a resolution lower than 900x900...
Can anyone clear this up to me please?
Thanks in advance.

I fear it is not possible: https://stackoverflow.com/questions/59930227/instagram-basic-display-api-alternative-image-size

Too bad! :(((

Has anyone find a way to get small images directly from the source? Any ETA from Instagram? I definitely understand that this is not an Instafeed problem, nonetheless I think this is a big problem for this library too.
Thanks for any info!

I found the following url pattern works for retrieving images of different sizes:
Get the post permalink:
https://www.instagram.com/p/CEriQnOMwW9/
and append media?size=t to it like this:

media?size=t returns a 150x150 px image
media?size=m returns a 320x320 px image
media?size=l returns a 1080x1080 px image

This resolves to a corresponding image on their CDN e.g.:
https://scontent-lht6-1.cdninstagram.com/v/t51.2885-15/e35/s150x150/118725806_2684783481838717_5792380552439146436_n.jpg?_nc_ht=scontent-lht6-1.cdninstagram.com&_nc_cat=1&_nc_ohc=gi9Y2_xUxfAAX_82RD5&_nc_tp=15&oh=fc3a5ca4968ef2fa3dea1237584ce627&oe=5F7C05C9
So in instafeed we can use the template option and the {{link}} templating tag to get the image size we need like so:

Or add our own size attributes using the transform option and model tag:
var feed = new Instafeed({

    accessToken: InstagramToken,

    // Add image size attributes
    transform: function(item) {
        item.model.image_thumb  = item.link + 'media/?size=t';
        item.model.image_medium = item.link + 'media/?size=m';
        item.model.image_large  = item.link + 'media/?size=l';
        return item;
    },

    template: ''

});

feed.run();

I'm guessing this could break at any moment if Instagram stop supporting that url pattern though.

Good detective work, @shanomurphy!
Your observation:

I'm guessing this could break at any moment if Instagram stop supporting this (undocumented?) url pattern though.

is pretty much on-point though: Instagram did take extra steps to stop Instafeed v1 from using these undocumented URLs, so I would be very hesitant to use them in a production setting.
I'm going to close this issue, as there's no official support for image resolutions, so it's not something that Instafeed can look after. Thanks again!

I think already breaked :(
Any resolution?


Disappearing text block when :set rightleft (mini.indentscope)

Contributing guidelines

[X] I have read CONTRIBUTING.md
[X] I have read CODE_OF_CONDUCT.md
[X] I have updated 'mini.nvim' to latest version

Module(s)
mini.indentscope
Description
If the buffer direction is set to right to left with :set rightleft, when the animation starts, the text disappears.
Neovim version
0.9.4
Steps to reproduce
1- open any file with some indentation and mini.align activated
2- :set rightleft
3- move the cursor to a new block of text
4. the block starts to disappear
Expected behavior
No response
Actual behavior
indented blocks starts to disappear in rightleft mode.

Thanks for the issue!
I indeed can reproduce on Neovim 0.9.5, but it seems to be fixed on current Nightly version 0.10.
The reason this happens is the way the line is drawn to be more robust against left-right window scroll and it indeed implicitly assumes that there is an empty space to the left of it. As it is quite a fundamental implementation detail, I'm afraid my only suggestion would be to use Nightly for this (or wait until 0.10 release, which should be soon-ish enough).
Closing as fixed on latest Neovim version.


cluster client configuration

Hi all,
can anyone provide some documentation on how to setup a client for this cluster?
I try to tests some rbd use cases like creating mounting and cloning rbd images with a local cluster sat up with vagrant.
I tried copying the monitor configuration from mon1:/etc/ceph/* to my hostmachine:/etc/ceph/*
doing so ceph health outputs:
root@jck2:~# ceph health
HEALTH_OK

ceph -w
root@jck2:~# ceph -w
    cluster 4a158d27-f750-41d5-9e7f-26ce4c9d2d45
     health HEALTH_OK
     monmap e1: 3 mons at {ceph-mon0=<IP_ADDRESS>:6789/0,ceph-mon1=<IP_ADDRESS>:6789/0,ceph-mon2=<IP_ADDRESS>:6789/0}, election epoch 6, quorum 0,1,2 ceph-mon0,ceph-mon1,ceph-mon2
     osdmap e14: 6 osds: 6 up, 6 in
      pgmap v24: 192 pgs, 3 pools, 0 bytes data, 0 objects
            207 MB used, 65126 MB / 65333 MB avail
                 192 active+clean

2014-11-10 12:59:34.364045 mon.0 [INF] pgmap v24: 192 pgs: 192 active+clean; 0 bytes data, 207 MB used, 65126 MB / 65333 MB avail

ceph osd tree:
root@jck2:~# ceph osd tree
# id    weight  type name   up/down reweight
-1  0.05997 root default
-2  0.01999     host ceph-osd0
0   0.009995            osd.0   up  1
3   0.009995            osd.3   up  1
-3  0.01999     host ceph-osd1
2   0.009995            osd.2   up  1
4   0.009995            osd.4   up  1
-4  0.01999     host ceph-osd2
1   0.009995            osd.1   up  1
5   0.009995            osd.5   up  1

creating of rbd images works!
root@jck2:~# rbd create --image-format 2 --size 512 test --id admin
root@jck2:~# rbd ls
test

However if I try to map and mount an image i get the following error:
root@jck2:~# rbd map test --id admin
rbd: add failed: (95) Operation not supported

Can anyone tell me why map fails? If someone provides me with some hints on setting up the client I am willing to contribute to ceph-ansible by writing a playbook that configures clients and document the setup for other users.
Thanks in advance

Cornelius

For CentOS7 (kernel 3.10), this is still issue. For now I use the following patch
diff --git a/roles/ceph-common/defaults/main.yml b/roles/ceph-common/defaults/main.yml
index c34a3ce..5411cb1 100644
--- a/roles/ceph-common/defaults/main.yml
+++ b/roles/ceph-common/defaults/main.yml
@@ -105,8 +105,8 @@ ceph_dev_redhat_distro: centos7
 #
 fsid: "{{ cluster_uuid.stdout }}"
 cephx: true
-cephx_require_signatures: true # Kernel RBD does NOT support signatures for Kernels < 3.18!
-cephx_cluster_require_signatures: true
+cephx_require_signatures: false # Kernel RBD does NOT support signatures for Kernels < 3.18!
+cephx_cluster_require_signatures: false
 cephx_service_require_signatures: false
 max_open_files: 131072
 disable_in_memory_logs: true # set this to false while enabling the options below

@rootfs you should probably override this variable from group_vars/all instead.
So your change won't be tracked in git :)
I was wondering if I shouldn't put this to 'false' by default anyway... Feel free to PR the change and I'll merge it. ;)

awesome, will ping you later

@rootfs sure :)

Have you guys had a chance validate client basic operations on centos7.1 with cephx enabled?
I still seeing the same problem, although OS has newer kernel that supposedly works with cephx.
I'd like to make it work properly.
[vagrant@ceph-mon0 ~]$ uname -a
Linux ceph-mon0 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[vagrant@ceph-mon0 ~]$ sudo rbd create --image-format 2 --size 512 test2 --id admin
[vagrant@ceph-mon0 ~]$ sudo rbd map  test2
rbd: sysfs write failed
rbd: map failed: (95) Operation not supported

Aditional question, how to restart services on centos nodes?
systemctl can't work with them. For example on mon0:
[vagrant@ceph-mon0 ~]$ sudo systemctl list-unit-files | grep ceph
<EMAIL_ADDRESS>                 static
<EMAIL_ADDRESS>                        static
<EMAIL_ADDRESS>                         disabled
<EMAIL_ADDRESS>                         enabled
<EMAIL_ADDRESS>                         disabled
<EMAIL_ADDRESS>                     disabled
ceph.target                                 disabled

@rokka-n
For a monitor: systemctl restart<EMAIL_ADDRESS>For an OSD: systemctl restart<EMAIL_ADDRESS>

$ rbd create rbd/test --size 2G --image-feature layering
$rbd info rbd/test
$ rbd map rbd/test
rbd: sysfs write failed
In some cases useful info is found in syslog - try "dmesg | tail" or so.
rbd: map failed: (13) Permission denied

now i am facing this issue. i have applied upper mention suggestions but unfortunately not success.

Want to  setup CEPH cluster integration with OpenNebula 5.2. but we faced some issues on that, after creating CEPH cluster when try to integrating with opennebula frontend node then its not getting up and not showing disk uasble space.
[cephuser@storage1 ~]$ ceph -s
cluster dd8ea244-d2e4-4948-b3b8-ec2ddedeff80
health HEALTH_OK
monmap e1: 3 mons at {storage1=XX:XX:XX:137:6789/0,storage2=XX.XX.XX.138:6789/0,storage3=XX.XX.XX.139:6789/0}
election epoch 70, quorum 0,1,2 storage1,storage2,storage3
osdmap e204: 3 osds: 3 up, 3 in
flags sortbitwise,require_jewel_osds
pgmap v312560: 2112 pgs, 3 pools, 14641 kB data, 24 objects
19914 MB used, 11104 GB / 11123 GB avail
2112 active+clean
[oneadmin@node1 ~]$ cat cephds.conf
NAME = "cephds"
DS_MAD = ceph
TM_MAD = ceph
DISK_TYPE = RBD
POOL_NAME   = data
CEPH_HOST   = storage1-storage2-storage3
CEPH_USER   = oneadmin
CEPH_SECRET = "AQCU1GxYHmbkBxAA6TlD6q+PyMTL+/a+AB+2bg==2dsf"
BRIDGE_LIST = storage1
Plz suggest us.

@subhashchand what are the features enabled on that test image?
Can you run an rbd info <pool>/<image>?


Implement a mongo_grpc_service_template

Like https://github.com/userver-framework/pg_grpc_service_template but with Mongo instead of PostgreSQL
Internal ticket: 8779

Working on it in https://github.com/Vinatorul/mongo_grpc_service_template/pull/1

Many thanks for the great PR!


mac identifer not unique for estimote nearables

its the nearables again :)  it seems they broadcast two different types of packets one a little more often than another but it seems to always use a different mac (i have two stickers near the bluetooth receiver in the screenshot)
not a big deal for me and this ticket can be closed without any work, but wanted to make sure this is at least mentioned here. it seems there is some kind of uuid which is broadcast somewhere, but unsure whether this is standard iBeacon stuff or not.

Hi,
I think you mean the UUID is different, not the MAC.
Correct?
I've come across beacons that broadcast a certain UUID under normal conditions, and when a button is pressed or when say movement is detected (Beacons with 3-axis accelerometer) they broadcasts a different UUID to indicate that this event is taking place.
I would not find that odd, and probably a support ticket with Estimote folks could clarify what you
are seeing, perhaps it is something useful for your scenario.
regards,

This is so long ago I would have to check again :) Will do this weekend.

Hi;
I am running into a similar issue. I have six estimote beacons and they all show the exact same UUID. The screenshot shows 3 of them as an example.
I checked the estimote portal and certainly all six have different UUID(s) and also none of them match what is being picked up in the UWP app.
Any thoughts?
Thanks;
Sujit
-=-


Integrate with SHAP

https://github.com/slundberg/shap

[ ] show feature importance in log
[ ] save other explanations in a pdf/html file:

[ ] beeswarm
[ ] force_plot (currently failing )

Example colab with SHAP.


pf_Ring woes on RHEL

in trying to build with PF-RING 6.0.2 i ran into some odd driver issues. First of all there was the numa stuff, which in RHEL you need numactl-dev in stead of libnuma-dev.
The other issue (don't have the output, but it was earily similar to this: http://www.gossamer-threads.com/lists/ntop/misc/34493) I fixed by going to pf_ring 6.2 (I assume it was the newer drivers in the package that fixed the issue).

0.14 no longer requires building of pfring or linking directly with it


Task casting

This PR implements task preparation for a given task, in the continuation of #2143
Task taxonomy follows 🤗 Transformers's pipelines taxonomy: https://github.com/huggingface/transformers/tree/master/src/transformers/pipelines

cc @abhi1thakur

Looks really nice so far, thanks !
Maybe if a dataset doesn't have a template for a specific task we could try the default template of this task ?

hey @SBrandeis @lhoestq,
i now have a better idea about what you guys are trying to achieve with the task templates and have a few follow-up questions:

how did you envision using DatasetInfo for running evaluation? my understanding is that all dataset_infos.json files are stored in the datasets repo (unlike transformers where each model's weights etc are stored in a dedicated repo).
this suggests the following workflow:

- git clone datasets
- load target dataset to evaluate
- load `dataset_infos.json` for target dataset
- run eval for each task template in `task_templates`
- store metrics as evaluation cards (similar to what is done in `autonlp`)

assuming the above workflow, i see that the current TaskTemplate attributes of task, input_schema, and label_schema still require some wrangling from dataset_infos.json to reproduce additional mappings like label2id that we'd need for e.g. text classification. an alternative would be to instantiate the task template class directly from the JSON with something like

from datasets.tasks import TextClassification
from transformers import AutoModelForSequenceClassification, AutoConfig

tc = TextClassification.from_json("path/to/dataset_infos.json")
# load a model with the desired config
model_ckpt = ...
config = AutoConfig.from_pretrained(model_ckpt, label2id=tc.label2id, id2label=tc.id2label)
model = AutoModelForSequenceClassification.from_pretrained(model_ckpt, config=config)
# run eval ...

perhaps this is what @SBrandeis had in mind with the TaskTemplate.from_dict method?
3. i personally prefer using task_templates over supervised_keys because it encourages the contributor to think in terms of 1 or more tasks. my question here is do we currently use supervised_keys for anything important in the datasets library?

How do you envision using DatasetInfo for running evaluation?

The initial idea was to be able to do something like this:
from datasets import load_dataset
dset = load_dataset("name", task="binary_classification")
# OR
dset = load_dataset("name")
dset = dset.prepare_for-task("binary_classification")

I don't think that's needed if we proceed as mentioned above

supervised_keys are mostly a legacy compatibility thing with TF datasets, not sure it's used for anything right now. I'll let @lhoestq give more details on that

The initial idea was to be able to do something like this:
from datasets import load_dataset
dset = load_dataset("name", task="binary_classification")
# OR
dset = load_dataset("name")
dset = dset.prepare_for_task("binary_classification")

ah that's very elegant! just so i've completely understood, the result would be that the relevant column names of dset would be mapped to e.g. text and label and thus we'd have a uniform schema for the evaluation of all binary_classification tasks?

That's correct! Also, the features need to be appropriately casted
For a classification task for example, we would need to cast the datasets features to something like this:
datasets.Features({
    "text": datasets.Value("string"),
    "label": datasets.ClassLabel(names=[...]),
})

We can ignore supervised_keys (it came from TFDS and we're not using it) and use task_templates

great, thanks a lot for your answers! now it's much clearer what i need to do next 😃

i'm not sure why the benchmarks are getting cancelled - is this expected?

i'm not sure why the benchmarks are getting cancelled - is this expected?

Hmm I don't know. It's certainly unrelated to this PR though. Maybe github has some issues

Something is happening with actions: https://www.githubstatus.com/

hey @lhoestq and @SBrandeis, i've extended the prepare_for_task API along the lines that @lhoestq suggested and added a few new tests to check that we trigger the value errors on incorrect input.
i think this is ready for another review :)

Looks all good thank you :)
Can you also add prepare_for_task in the main_classes.rst file of the documentation ?

Done! I also remembered that I needed to do the same for DatasetDict, so included this as well :)


Documentation for removing an event

Added documentation and example for removing a CasperJS event

Nice -- thank you. Could you adjust indentation to the other code examples in the page?
Apart from that :+1: to merge.
A hint (how) to put this into tearDown like in tests/suites/casper/request.js or tests/suites/casper/open.js would be nice, too. :-)

Thanks a lot.

No worries. Thank you for maintaining CasperJS!


The KL divergence is same as in Variational Auto Encoder ?

Hi,
Your implementation is perfect and so informative. In the paper, they clearly mentioned how they simplified the mutual information constraints into KL divergence w.r.t a Gaussian distribution.  So here we take the KL divergence between two distributions rather than two random variables. I think this is the same technique that they have used in VAE calling variational inference. Is my understanding is correct?
Actually, the difference is in VAE we do not explicitly talk about the information constrain and directly optimizing w.r.t the KL divergence loss. But with the I_c we have to use a Lagrangian multiplier and update the beta adaptive.
Is my understanding is correct?

Hi @shamanez,
Apologies for the late reply. I was laden with too much work and was a bit unwell on the weekend too.
Thank you for the comment and the compliments. Means a lot to me.
First of all, please note that the KL-Divergence cannot be calculated between two different random variables. A lot of students seem to have this confusion which I don't know why. It is a measure of divergence between two distributions over the same random variable. refer kl-divergence-wiki for more information.
You are right about this part that in case of VAE, we are just minimizing the KL-divergence between the predicted gaussians and the unit gaussian, whereas in VGAN, we want the KL-divergence to be specifically equal to the I_c that we set. A lesser KL-Divergence is not allowed too.
Hope this helps.
Best regards,
@akanimax


low 8 bits of channel 3 period are in NR33, not NR13

just a simple copypaste error :)

oh actually the trigger description reference the wrong register too. i'll edit just a bit more..


Refactor output of find to allow for json and normal output.

Rather complicated solution becaused I wanted to retain the streaming
character of the output, which means for json I have to manually add
headers and footers per snapshot scanned + a list around the whole
set.
Closes #869

Codecov Report

Merging #871 into master will decrease coverage by 0.34%.
The diff coverage is 22.95%.

@@            Coverage Diff             @@
##           master     #871      +/-   ##
==========================================
- Coverage   53.56%   53.21%   -0.35%
==========================================
  Files          92       92
  Lines        7565     7616      +51
==========================================
+ Hits         4052     4053       +1
- Misses       2902     2949      +47
- Partials      611      614       +3

Impacted Files
Coverage Δ

src/cmds/restic/cmd_find.go
36.49% <22.95%> (-13.51%)
:x:

src/cmds/restic/lock.go
66.66% <0%> (-2.78%)
:x:

src/restic/archiver/archiver.go
64.74% <0%> (-0.23%)
:x:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fdb9872...81b5f94. Read the comment docs.

Awesome idea, testing included !

FYI: The test fails on windows because (as far as I can see) because either the testfile-hardlink or testfile-symlink are not extracted from the tar.gz and therefore not archived, so the snapshot only contains one file called testfile that the second pattern testfile* matches.
Maybe use the pre-packaged small-repo.tar.gz and search for bar and bar* in snapshot 06c5a25c:
$ restic -r ./repo find 'bar*'
enter password for repository:
Found matching entries in snapshot 6619a8ad
/testdir/bar
Found matching entries in snapshot d187480a
/testdir/bar
Found matching entries in snapshot 2145b769
/testdir/bar
Found matching entries in snapshot 06c5a25c
/testdir/bar
/testdir/bar2

Hmm, I don't think you read that right. Under windows it seems to extract symlinks as real files (either tar or restore), because I get three files back, the normal, the hardlink and the symlink. And since 2 != 3 .. boom
Fixed edgecase by having printJSON handle symlinks now as well.
[{3 e5343e07cd055f251a9f1603c773d7642f3717239a4b8b4dcd9e943da826b58c [{\testdata\0\tests\testfile -rw-rw-rw- 21 2014-08-09 12:14:20 +0000 UTC 0 0} {\testdata\0\tests\testfile-hardlink -rw-rw-rw- 21 2014-08-09 12:14:20 +0000 UTC 0 0} {\testdata\0\tests\testfile-symlink -rw-rw-rw- 21 2014-08-09 12:14:28 +0000 UTC 0 0}]}]

I wonder: shouldn't we just output the raw restic.Node as the hit, complemented by a path field with the absolute path (in addition to the name field)? The output of find --json will be read by a program (e.g. jq), so the caller can decide what fields they are interested in...

Let me figure it out. Last time I tried, Node contained a lot of cruft, so I decided against it. But you are right, it is machine read...


Broken after deploying with my own script.

Whenever I add a new file in /scripts and deploy to heroku, the bot fails to come online again.
If I start with a fresh installation of hubot and dump all my code in the example.coffee file everything works just fine.

My apologies! Looks like it was an issue with Heroku. Restarted my app and it worked. I also tried some different things with and without the keepalive package. Not certain what caused the problem/fix.


How to view dependencies between node_modules?

This didn't show as much info as I was hoping for....
C:\Dev\climate-app>madge .
Processed 2 files (893ms)

app/app.js
server.js

What I want to do is understand dependencies between packages in node_modules. It seems I can't:
C:\Dev\climate-app>madge node_modules
Processed 0 files (723ms)

EDIT: I just found out npm has a built-in way to show a dependency tree of modules: npm list.
I'm also wondering: in a large project, can you suppress transitive dependencies? i.e. if there is a path X->Y->Z, and another path X->Z, suppress display of X->Z?

For grouping paths I wrote a transformer that might be a good starting point:
https://github.com/joebowbeer/repdeps

I have the same question as @qwertie . I suspect I have a circular dependency, but madge doesn't print any node_modules dependencies:
~/data-explorer (mc/histogram-facet): madge ui/src/components/facets/HistogramFacet.js
Processed 2 files (338ms) (8 warnings)

HistogramFacet.css
HistogramFacet.js
  HistogramFacet.css

Have you tried the —include-npm option @mellisachang? It can be terribly slow though.

Yes, that didn't help.
~/data-explorer (mc/histogram-facet): madge --include-npm ui/src/components/facets/HistogramFacet.js
Processed 2 files (336ms) (8 warnings)

HistogramFacet.css
HistogramFacet.js
<EMAIL_ADDRESS>  ../../../node_modules/react/index.js
  HistogramFacet.css

HistogramFacet.js includes:
import React, { Component } from "react";
import { canvas } from "vega-canvas";
import { Handler } from "vega-tooltip";
import * as vl from "vega-lite";
import * as vega from "vega-lib";
import Vega from "react-vega";
import { withStyles } from "@material-ui/core/styles";

import "./HistogramFacet.css";
import * as Style from "libs/style";
import colors from "libs/colors";
import FacetHeader from "components/facets/FacetHeader";

Chrome Developer Tools shows a bunch of things are imported:

Context is, I think there's a circular dependency somewhere and I was hoping madge could find it. (I'm using create-react-app so I can't use circular-dependency-plugin.)


Bump default dev ruby to 2.7.1

While I was doing #248 rvm complained about ruby version. Because CI supports 2.7, I thought it might be a good idea to bump .ruby-version too.

Coverage increased (+0.01%) to 96.837% when pulling f5fe46227161118afeff84a0507f128a3b45a24e on igas:bump-default-ruby into 1c5f92da0b3e7d5e6092f68f699eb99e621d8c0e on devise-security:master.

Coverage increased (+0.01%) to 96.837% when pulling f5fe46227161118afeff84a0507f128a3b45a24e on igas:bump-default-ruby into 1c5f92da0b3e7d5e6092f68f699eb99e621d8c0e on devise-security:master.

Actually, I'm going to have to revert this, the idea behind our maintenance is to keep the development stack linked to the oldest supported version of ruby so we don't try to make use of new language features and introduce compatibility bugs as a result.

@olbrich do you think CI would be too late for feedback loop? I can see a point in it, but even then, it should be 2.3.0 as per gemspec.


Name "Cachable" should be "Cacheable"

I was trying to include Cacheable, but it was not found. Apparently it is mistakenly called Cachable.
File: src/Traits/Cachable

Doh! I will update this with the next breaking update.

Further research suggest this is the correct spelling: https://howtospell.co.uk/droptheerule.php
Keeping Cachable.

I am not sure how you conclude from the text that "it is the correct spelling". Can you clarify?

"cachable" gives "About 125,000 results"
"cacheable" gives "About 905,000 results"

Cachable might be a valid alternative spelling, but the general accepted spelling is clearly "cacheable".

The framework goes the other way:
https://github.com/search?q=repo%3Alaravel%2Fframework Cacheable&type=code


Feature/refactor cache action

Description
Provides CacheAction and CacheActionFactory implementing generalized logic of InMemoryCacheActionFactory.
Splits responsibility into several helper classes and provides unit tests.
Preserves legacy behaviour of InMemoryCacheActionFactory (legacy unit tests are refactored and pass).
Motivation and Context
Closes #192
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

[x] I have read the CONTRIBUTING document.
[x] My change requires a change to the documentation.
[x] I have updated the documentation accordingly.

I hereby agree to the terms of the Knot.x Contributor License Agreement.

PR contains also large refactoring of some Test classes

@marcinus Can you please provide upgrade notes to use CacheActionFactory instead of InMemoryCacheActionFactory.


Disable MCAD by default

https://issues.redhat.com/browse/RHOAIENG-5129

You probably know this, but #491 also accomplishes this.  You'll have the same SDK e2e testing issues with this PR that are blocking #491  (and will need to make similar adjustments in the PR to the e2e tests for this repository to get them to work without AppWrappers).

Heya @dgrove-oss, yep! Just from a risk reduction perspective we need to keep minimal code changes due to some of the other large things that are in motion.

Hi @anishasthana @tedhtchang I tried testing on my OC cluster - glad to re-test/try again tomorrow with extra instructions, but here's what I did so far.

OC 4.14.17 with kustomize, podman, etc.
Installed ODH 2.10.0 (It won't take startingCSV: opendatahub-operator.v2.4.0 anymore which is annoying)
Now I need dsci manually - I applied one, then dsc with CodeFlare Component Removed.

I'm having trouble with model-mesh and kserve...  I tried a few things but throwing up my hands for now on that.

So, ok let's see if I can build the image and install it
4.1  clone this PR
git clone --single-branch --branch add_mcad_config_flag https://github.com/anishasthana/codeflare-operator.git
cd codeflare-operator
4.2 See if I can install or build an image
make install
make deploy
make image-build
4.3 tag and push to quay
podman tag quay.io/project-codeflare/codeflare-operator:v0.0.0-anish quay.io/jbusche/codeflare-operator:pr499-anish
podman push quay.io/jbusche/codeflare-operator:pr499-anish
4.4 oc edit deploy codeflare-operator-manager -n openshift-operators
image: quay.io/jbusche/codeflare-operator:pr499-anish
4.5 check that it's running

oc get pods -n openshift-operators |grep codeflare
codeflare-operator-manager-fdc56c47b-q947k                 1/1     Running   0              14m

I'm leaving it where it is for now... not sure how to resolve the dsc issues or how to make sure KFTO/kueue are up if there's no MCAD.

Superseded by #500.


Normalize JARs produced by MavenJarCreator

Context
As part of https://github.com/gradle/dotcom/issues/8347 we noticed that Performance Tests are not properly cacheable.
One of the reasons was that this test is generating plenty of test JARs and they are not normalized correctly: https://e.grdev.net/c/j35kx4okgjbvo/wmbty2mtvklr2/task-inputs?expanded=WyJra2lxd2V0eGh4NHhvLXRlc3RQcm9qZWN0RmlsZXMtMC0wIiwia2tpcXdldHhoeDR4by10ZXN0UHJvamVjdEZpbGVzLTAtMC0wIiwia2tpcXdldHhoeDR4by10ZXN0cHJvamVjdGZpbGVzIl0#change-kkiqwetxhx4xo-testProjectFiles-0-0-0
Gradle Core Team Checklist

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation
[ ] Recognize contributor in release notes

@bot-gradle test this

OK, I've already triggered ReadyForMerge build for you.

Still not perfect https://e.grdev.net/c/akts3fjxytg4u/agnzlckuva54s/task-inputs?expanded=WyJra2lxd2V0eGh4NHhvLXN0YWJsZWNsYXNzcGF0aCJd#change-kkiqwetxhx4xo-stableClasspath-0-0
@wolfs can we take a look at this together some time this weel?

@bot-gradle test this

OK, I've already triggered ReadyForMerge build for you.

Still not perfect https://e.grdev.net/c/akts3fjxytg4u/agnzlckuva54s/task-inputs?expanded=WyJra2lxd2V0eGh4NHhvLXN0YWJsZWNsYXNzcGF0aCJd#change-kkiqwetxhx4xo-stableClasspath-0-0

There seems to be a source code change between the two builds in internal integ testing.

@bot-gradle test and merge

Your PR is queued because #17053 is being built.

OK, I've already triggered a build for you.


Add translation to SwipeableItemProps

add origin translation node to renderLeadingItem/renderTrailingItem (SwipeableItemProps)

Thanks for the contribution!

oh.. I think "translation" cannot be optional, it's defined by default, can you check it?
translation?: Animated.Node

yes, '?' can be removed

Do you have any plans to release the next version (npm package)?

published v0.0.2


Added missing and little bit fixes in docs/internals-ru

Added missing and little bit fixes

Спасибо


Text created after S2D_Show(window) does not display (Sample included)

#include <simple2d.h>

//Meta
S2D_Window *window;

//Arcade
S2D_Text *title;

void render() {
  if (title) {
    S2D_DrawText(title);
  }
}

void on_key(S2D_Event e) {
  S2D_Close(window);
}

int main() {
  window = S2D_CreateWindow(
    "", 1920, 1080, NULL, render, S2D_RESIZABLE
  );
  window->on_key = on_key;
  window->fps_cap = 120;
  window->viewport.mode = S2D_EXPAND;

  S2D_Show(window);
  title = S2D_CreateText("media/arial.ttf", "Combo.Zone Zap Edition", 20);
  title->x = 127;
  title->y = 740;

  return 0;
}

Needs a media folder with arial.ttf
Expected to see the text display, it doesn't.
If you move the S2D_Show to
  title = S2D_CreateText("media/arial.ttf", "Combo.Zone Zap Edition", 20);
  title->x = 127;
  title->y = 740;
  S2D_Show(window);

the text displays.
I need to dynamically create a S2D_Text after the window is already running.
How do I do this?

I'm realizing that the S2D_Show starts a loop. However, if I initialize a S2D_Text during the update function, it still will not render.

PEBKAC


Added ability to set excluded incoming message headers from copying t…

…o outgoing messages

@bullmastiffo @mzabolotko
Hi guys,
just to notify you - I have made a minor change to the sender and receiver options to support explicit message storage configuration on the endpoint and its propagation to the Rabbit specific defaults. See the commit below https://github.com/SDVentures/Contour/commit/c7cadc07152afef8cde541fa10cbe9633b0ef6ef.
Please do not hesitate to post questions and concerns.


Add test for controllers and services.

Fix failing update method 'cuz of checking order.
Add OrderedRunner and Order for forcing JUnit to run the test in a certain order.
Clear database after tests run.

@all-contributors please add @afgalvan for tests


Module tests unnecessary strict mode

As module code is always treated as strict mode code (https://tc39.github.io/ecma262/#sec-strict-mode-code) it is unnecessary to emit strict and non-strict versions of tests that are to be run as module code.
This results in longer running times and doubling the failure/pass counts for these tests for no purpose.
I think this could be updated by revising create-scenarios.js to check for test.attrs.flags.module.

I'm in favor of this! Do you want to prepare a PR?


Document example suggestion

Version
v3.1.1
Expected Behavior
First of all thanks to make this wonderful library.
Reason that I create this issue is I have waste much time the Internationalization (i18n) is doesn't work well.
It's my fault. I only saw the example and mistakenly thought that it didn't work well.
After seeing this document I found that I don't import source file of i18n.
If it's not rude, can you put the phrase of the import file in the example as well? or can I PR about this?
like this

@HTMLhead
I agree with you! Can you write the PR for this?


access to portal information database

IITC seems to be the API for accessing portals in an easy way. Is it possible to query the db of all known portals and get info like the image, name, latlong, etc?

You can access the info for any portal that has been retrieved through the interface. There's also the cache option, which maintains the info for a number of previously viewed portals. This is definitely NOT the same as a database of all known portals! :)

Cool, thank you. Follow up: is there a function to display ALL portals inside a map view, rather than just linked ones over a certain length?

Did you look at the "show list of portals" plug-in?

The portals list plugin will only fire when you've zoomed in far enough to show portals rather than links.

Is there a way to access the cashed portal information for the currently displayed portals. I am interested in being able to look at AP, MOD, Links, etc. But I don't want to query the server. Is this possible?

@billplaysonline Look at the existing plugins that do use such info. You will still be limited to requiring the close zoom level for much of the info.

Without Ordinary provides good advice.  For example, take a look at
portals-list (
https://static.iitc.me/build/release/plugins/portals-list.user.js) as a
starting point.  You get the truncated level, team, health, num resonators,
num links, fields, and AP.  Mods are in the portal details which *typically
*require you to click on the portal or, indirectly, via link as
portals-list does it. You've already written about not wanting to query the
server; thus, you seem to want a solution that keeps in the accepted spirit
of the game where portal details require a human, or perhaps a trained frog
😀, to click on something.
If you continue pursuing your own solution further, you'll be getting into
objects like window.portals, window.links, window.resonators and
window.fields.  These are accessible from your browser's console - thus,
you can inspect them to understand their content.
I hope this is helpful!
On Fri, Dec 16, 2016 at 12:17 PM, Without Ordinary<EMAIL_ADDRESS>
wrote:

@billplaysonline https://github.com/billplaysonline Look at the
existing plugins that do use such info. You will still be limited to
requiring the close zoom level for much of the info.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/iitc-project/ingress-intel-total-conversion/issues/1116#issuecomment-267686374,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEGbQU_9xyJ-4vpDfZ6zyj4Y88WpEsHmks5rIvHfgaJpZM4JEBLi
.

While examining the wonderful information supplied I ran across what I would call a nuisance bug.
The script works fine when at the proper portal layer. If not it just doesn't respond on the desktop and on mobile it brings up a blank popup.
Anyway I added
//fill the listPortals array with portals avaliable on the map (level filtered portals will not appear in the table)
window.plugin.portalslist.getPortals = function() {
//filter : 0 = All, 1 = Neutral, 2 = Res, 3 = Enl, -x = all but x
var retval=false;
var displayBounds = map.getBounds();
//*******************************************************************************************
//BS Below added by billplaysonline AKA William Schlesinger<EMAIL_ADDRESS>// This fixes the problem where if you click on get portals list and you are zoomed out too far it just dies.
// Now it displays a message "nothing to show".
var zoom = map.getZoom();
zoom = getDataZoomForMapZoom(zoom);
var tileParams = getMapZoomTileParameters(zoom);
if (tileParams.hasPortals === false) {
return retval;
}
//BS end of change
window.plugin.portalslist.listPortals = [];
What is the proper way for me to  have this done?
I usually don't try and fix bugs, Just locate them and report them.
but I found this in another script.

When the map is zoomed out all the way, 'map.getBounds()' is returning an invalid southwest longitude (e.g. -300.9375).  We expect values -180 to +180.

@hayeswise
http://stackoverflow.com/questions/18063278/leaflet-getbounds-returning-longitudes-greater-than-180
event.latlng.wrap().lng should help

Thanks!  It did help.
My getPortalsInMapBounds function now does the following:
displayBounds = window.map.getBounds();
displayBounds.getSouthWest().wrap();
displayBounds.getNorthEast().wrap();

displayBounds.contains(portal.getLatLng()) now works as expected.
The mapZoomHasPortals() function in your fix  #1169 is still useful to handle the case of the portal data not being there when the map is zoomed out to far (other readers: see billplaysonline post above with the code).
I made the following small change in the window.plugin.portalslist.fields.forEach(function (field, i) { loop that gathers the portal data for each row to make it more robust:
var value = field.value(portal);
if (typeof value === 'undefined') {
   value = "[unknown]"; // or set to blank
}

Thus, if the getPortalsInMapBounds check is not used, and you are sufficiently zoomed out, you'll get a portal list with all blank names and "[unknown]" for for various portal values.

I don't believe there is anything actionable here? Can this issue be closed?

Has the new portal-list been released? If so it is OK by me

@billplaysonline I think your issue is actually bug #1161 ?

@Nhamer I think you are correct.


Unknow Hash Algorithm OID

After "cloning the certification";
[*] Spoofing arp replies...
[*] Turning on IP forwarding...
[*] Set iptables rules for SYN packets...
[*] Waiting for a SYN packet to the original destination...
[+] Got it! Original destination is <IP_ADDRESS>
[*] Clone the x509 certificate of the original destination...
Unknow Hash Algorithm OID:
[*] Cleaning up...
[*] Done.

Apparently older versions of openssl asn1parse don't support output to stdout with -out -. 8cb1cf1e fixes it.


Error: Initializer of instance member variable 'x' cannot reference identifier 'y' declared in the constructor.

I got this error:

Initializer of instance member variable 'objectWithFunc' cannot reference identifier 'param' declared in the constructor.

for this piece of code:
interface ITest {
    foo: string;
    bar: string;
}

var param: ITest = null;

class Test {

    constructor(param: ITest) {
    }

    method() {
        console.log(param.foo);
    }

    private objectWithFunc = {
        func: () => {
            console.log(param.foo);
        }
    }
}

When I change parameter name in constructor to other name (for example constructor(param2: ITest)) it compiles.
This behaviour was surprising for me. Is it a bug?

I'm using Typescript 1.4 with Visual Studio Community 2013.

This is by design. Property initialization happens as part of constructying the object. the compiler will generate it as part of the constructor, that means that your constructor parameter "param" will shadow the variable "param" at runtime. this error message is letting you know that this is going to happen.
so the following ts code:
var param = null;
class Test {
    constructor(param) {
       //constructor body
    }
    private property = param; // initialization
}

will be transformed to:
var param = null;
var Test = (function () {
    function Test(param) {
        this.property = param; // initialization
        //constructor body
    }
    return Test;
})();


Falco not having enough priority and paging Phoenix

Phoenix gets paged several times a day because falco-app can't be scheduled in the MC nodes.
There are a few things to say:

there is no reason why phoenix should get paged for this. The cluster is 100% healthy. Thus as soon as team security has an oncall rotation we ask you to take over this alert.
if falco strictly has to be running in all nodes, then let's raise its priorityClass

The SIG discussed. Because the pods can't schedule, this is more a symptom of the MCs being generally out of resources and not a problem with falco. But we definitely want to make it nicer to oncallers.
Action items:

Zach will bump the priorityclass of falco
Some changes to node pool scaling are rolling out soon. We suspect it might help because prometheus has a history of forcing other pods off of nodes, often because it gets stuck on the only remaining node in an availability zone. The new behavior would make the scheduler aware of the need for more resources in a given AZ, so we will see if helps :crossed_fingers:

If not, we may need to increase node sizes or revisit https://github.com/giantswarm/giantswarm/issues/14269

Released priority class for Falco in v0.3.1 https://github.com/giantswarm/falco-app/pull/88

Closing this as the quick fix has been released. If the problem persists or moves to a different component, we'll have to discuss increasing node sizes or revisiting our prometheus sharding


Genesis requires Node.js 6 or later but it's not mentioned anywhere

Genesis requires Node.js 6 or later but the requirement is not mentioned anywhere.

Genesis Replicator requires Node.js 6, and this actually mentioned in the documentation as the last paragraph before the Limitations section.
If you launch Genesis itself with vagrant up, it installs Node.js 6 on the virtual machine that hosts the replicators (and other components of the system). In this case the only requirements are Vagrant and VirtualBox.


Unsupported map version with <IP_ADDRESS> worlds

Same

It looks like the fix has been merged.  Just need someone to release it.

Binaries tomorrow.


Gather detailed PKI requirements

Is Vault the reference PKI CLM? Another type we should support?
Should we check for existing cert before issuing new one? IE, are certs cached?
Are we simply issuing cert from a known intermediate? or do we need to setup a chain (eg, an intermediate for the owner if it doesn't exist).
Are we setting up roles in Vault? (I think probably not)

In our last meeting, we decided caching was a premature optimization, and to hold off on this for now.


19043.1202

it's work
rdpwrap.zip

Thank you, it's work also for 19043.1165

Amazing work! Why isn't this in autoupdate? Thanks a lot!

Thanks, I can confirm this works for me too.  Autoupdate didn't work.
For Reference:
net stop termservice
Copy over file
net start termservice

Hi everyone,
This ini file works fine in 10.0.19041.1202.
Thanks a lot

This file also works fine in 19043.1100.

[10.0.19041.1202-SLInit]
; no x86 section

bInitialized.x64      =107108
bServerSku.x64        =10710C
lMaxUserSessions.x64  =107110
bAppServerAllowed.x64 =107118
bRemoteConnAllowed.x64=107120
bMultimonAllowed.x64  =107124
ulMaxDebugSessions.x64=107128
bFUSEnabled.x64       =10712C

[10.0.19041.1202]
; no x86 section

LocalOnlyPatch.x64    =1
LocalOnlyOffset.x64   =89D81
LocalOnlyCode.x64     =jmpshort
SingleUserPatch.x64   =1
SingleUserOffset.x64  =0D139
SingleUserCode.x64    =Zero
DefPolicyPatch.x64    =1
DefPolicyOffset.x64   =19105
DefPolicyCode.x64     =CDefPolicy_Query_eax_rcx
SLInitHook.x64        =1
SLInitOffset.x64      =1E98C
SLInitFunc.x64        =New_CSLQuery_Initialize

And now please for x86 :-)

Nice work! works fine on my machine.