MicroBitBLEManager - increase MAXIMUM_BONDS from 4 to 15

See https://github.com/lancaster-university/codal-microbit-v2/issues/289.
I can't think what I can do to test this.

Yes please, let's make it a built time config option 👍
We should aim to have all flag defaults in MicroBitConfig.h, so let's move it there (looks like it's already identified as a BLE config not ported there yet, I think it's fine to start with this one, even if we don't have the others there yet).
For the DAL vs CODAL config in MakeCode, I thought they are separated?
{
    "yotta": {
        "config": {
            "codal": {
                "MICROBIT_BLE_MAXIMUM_BONDS": 20,
            },
            "microbit-dal": {
            }
        }
    }
}

@microbit-carlos

For the DAL vs CODAL config in MakeCode, I thought they are separated?
Ah I did not know that!

MakeCode project with settings edited to choose 15 bonds and create utility service
https://makecode.microbit.org/_KLihp9h8PXiu

Thanks @carlosperate  @martinwork
Looks good to me! This shouldn't change any default behaviour, but it does feel like we should see what happens if we set a value other than 4 :)
As I recall, SoftDevice stores its bond data in a dedicate page of FLASH... so my guess is that we're good up until the point that page fills up.
I'm happy for this to go in as it is low risk, but we should work out what the hard maximum is in due course, and ensure we're not creating undefined behviour by that page overflowing into other areas of FLASH etc.

Yeah, the plan is to be able to expose this a build-time config option, so that we can create a MakeCode project and test if we can store 30 bonds:

https://github.com/lancaster-university/codal-microbit-v2/issues/289

As we don't have that many devices around, we can get some help testing from teachers and other volunteers, and for that having a MakeCode project to share with them would be quite handy.

cool - thanks carlos. Sounds good!

The bonds are stored by the Nordic SDK in it's FDS file system, for which we have reserved the minimum 2 pages.
https://github.com/martinwork/codal-microbit-nrf5sdk/blob/master/sdk_config/sdk_config.h#L967
I calculated that 20 is the absolute most that might fit without changing the memory map, but 20 might trigger garbage collection, so I suggested 15 as a safer bet. I would guess that exceeding the limit has been tested for 5 devices, but the FDS has never been filled.


Call skipChildren for nested object

What changes were proposed in this pull request?
In jsonObjectKeys, we should call skipChildren() properly for nested object.
Why are the changes needed?
Currently if jsonParser.currentName() is null, the loop is terminated.
We should place current name check inside the loop.
Does this PR introduce any user-facing change?
No.
How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?
No.

cc @panbingkun @MaxGekk

Test suite seems to pass - there was one error w.r.t. downloading which is unrelated.
[error] lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] https://maven-central.storage-download.googleapis.com/maven2/org/codehaus/jackson/jackson-jaxrs/1.9.2/jackson-jaxrs-1.9.2.jar: download error: Caught java.io.IOException (Server returned HTTP response code: 503 for URL: https://maven-central.storage-download.googleapis.com/maven2/org/codehaus/jackson/jackson-jaxrs/1.9.2/jackson-jaxrs-1.9.2.jar) while downloading https://maven-central.storage-download.googleapis.com/maven2/org/codehaus/jackson/jackson-jaxrs/1.9.2/jackson-jaxrs-1.9.2.jar
[error]
[error]         at lmcoursier.internal.shaded.coursier.Artifacts$.$anonfun$fetchArtifacts$9(Artifacts.scala:365)

Do we have corresponding cases? You can use UT to cover it.

In which directory should I put the following test ?
import static org.junit.Assert.*;
import org.apache.spark.unsafe.types.UTF8String;
import org.apache.spark.sql.catalyst.util.GenericArrayData;
import org.junit.Test;
import java.io.IOException;

public class JsonParserTest {

    @Test
    public void testJsonObjectKeysWithNullAndNonNullKeys() throws IOException {
        // JSON string that has a sequence where the parser will encounter null, then non-null keys
        String jsonString = "{\"key1\":null, \"key2\":{\"nestedKey1\":\"value1\"}, \"key3\":\"value2\"}";

        UTF8String json = UTF8String.fromString(jsonString);

        GenericArrayData result = JsonExpressionUtils.jsonObjectKeys(json);

        UTF8String[] expectedKeys = {
            UTF8String.fromString("key1"),
            UTF8String.fromString("key2"),
            UTF8String.fromString("key3")
        };

        assertNotNull("Result should not be null", result);
        assertEquals("Length of keys array should be 3", 3, result.numElements());
        assertArrayEquals("Keys should match expected values", expectedKeys, (UTF8String[]) result.array());
    }
}

@panbingkun @MaxGekk
Please take another look.

The following was not related to this PR
[info] - SPARK-43923: commands send events ((streaming_query_command {
[info]   query_id {
[info]     id: "89ea6117-1f45-4c03-ae27-f47c6aded093"
[info]     run_id: "89ea6117-1f45-4c03-ae27-f47c6aded093"
[info]   }
[info]   stop: true
[info] }
[info] ,None)) *** FAILED *** (10 seconds, 25 milliseconds)
[info]   The code passed to eventually never returned normally. Attempted 661 times over 10.00008105 seconds. Last failure message: VerifyEvents.this.listener.executeHolder.isDefined was false No events have been posted in Timeout(Span(10, Seconds)). (SparkConnectServiceSuite.scala:878)
[info]   org.scalatest.exceptions.TestFailedDueToTimeoutException:
[info]   at org.scalatest.enablers.Retrying$$anon$4.tryTryAgain$2(Retrying.scala:219)
[info]   at org.scalatest.enablers.Retrying$$anon$4.retry(Retrying.scala:226)
[info]   at org.scalatest.concurrent.Eventually.eventually(Eventually.scala:348)
[info]   at org.scalatest.concurrent.Eventually.eventually$(Eventually.scala:347)
[info]   at org.scalatest.concurrent.Eventually$.eventually(Eventually.scala:457)
[info]   at org.apache.spark.sql.connect.planner.SparkConnectServiceSuite$VerifyEvents.executeHolder(SparkConnectServiceSuite.scala:878)
[info]   at org.apache.spark.sql.connect.planner.SparkConnectServiceSuite$VerifyEvents.onCompleted(SparkConnectServiceSuite.scala:898)
[info]   at org.apache.spark.sql.connect.planner.SparkConnectServiceSuite.$anonfun$new$27(SparkConnectServiceSuite.scala:536)
[info]   at


LSP and Snippet issues

I have really been trying to read through the documentation and figure this out, but I am lost. Neither the LSP source nor snippets are working. I saw that some other people were having issues with the sumneko Lua LSP, but neither of these features work in python either.
Here is my config:
local cmp = require("cmp")
local luasnip = require("luasnip")
local lspkind = require('lspkind')

local has_words_before = function()
    if vim.api.nvim_buf_get_option(0, "buftype") == "prompt" then
        return false
    end
    local line, col = unpack(vim.api.nvim_win_get_cursor(0))
    return col ~= 0 and vim.api.nvim_buf_get_lines(0, line - 1, line, true)[1]:sub(col, col):match("%s") == nil
end

local feedkey = function(key)
    vim.api.nvim_feedkeys(vim.api.nvim_replace_termcodes(key, true, true, true), "n", true)
end

cmp.setup({
    snippet = {
        expand = function(args)
            luasnip.lsp_expand(args.body)
        end,
    },
    completion = {
        completeopt = "menu,menuone,noselect",
        get_trigger_characters = function(trigger_characters)
            return vim.tbl_filter(function(char)
                return char ~= ' '
            end, trigger_characters)
        end,
    },
    formatting = {
        format = function(entry, vim_item)
            vim_item.kind = lspkind.presets.default[vim_item.kind] .. " " .. vim_item.kind
             -- set a name for each source
            vim_item.menu = ({
                spell = "[Spell]",
                buffer = "[Buffer]",
                calc = "[Calc]",
                emoji = "[Emoji]",
                nvim_lsp = "[LSP]",
                path = "[Path]",
                look = "[Look]",
                treesitter = "[treesitter]",
                luasnip = "[LuaSnip]",
                nvim_lua = "[Lua]",
                latex_symbols = "[Latex]",
                })[entry.source.name]
            return vim_item
        end
    },
    mapping = {
        ["<Tab>"] = cmp.mapping(function(fallback)
            if vim.fn.pumvisible() == 1 then
                feedkey("<C-n>")
            elseif luasnip.expand_or_jumpable() then
                luasnip.expand_or_jump()
            elseif has_words_before() then
                cmp.complete()
            else
                fallback() -- The fallback function sends a already mapped key. In this case, it's probably `<Tab>`.
            end
        end, { "i", "s" }),

        ["<S-Tab>"] = cmp.mapping(function(fallback)
            if vim.fn.pumvisible() == 1 then
                feedkey("<C-p>")
            elseif luasnip.jumpable(-1) then
                luasnip.jump(-1)
            else
                fallback()
            end
        end, { "i", "s" }),
    },
    sources = {
        { name = 'nvim-lsp' },
        { name = 'luasnip' },
        --{ name = 'treesitter' },
        { name = 'nvim_lua' },
        --{ name = 'spell' },
        { name = 'buffer' },
        --{ name = 'emoji' },
        --{ name = 'path' },
        --{ name = 'look' },
        --{ name = 'latex_symbols'}
    },
})

for

-- The nvim-cmp almost supports LSP's capabilities so You should advertise it to LSP servers..
local capabilities = vim.lsp.protocol.make_client_capabilities()
capabilities = require('cmp_nvim_lsp').update_capabilities(capabilities)

-- ADD OTHERS LATER
require'lspconfig'.lua.setup {
    capabilities = capabilities,
}

Apologies for the mess. As you can see, it only shows the buffer source.

Any help is greatly appreciated :)
Edit: I already have made sure everything is installed in Packer

Could you paste the :LspInfo output?

Language client log: /home/vm/.cache/nvim/lsp.log
 Detected filetype:   lua

 1 client(s) attached to this buffer:

 Client: lua (id: 1, pid: 3210, bufnr: [1, 1, 1])
 	filetypes:       lua
 	autostart:       true
 	root directory:  /home/vm/.config/nvim/lua/config
 	cmd:             /home/vm/.local/share/nvim/lspinstall/lua/./sumneko-lua-language-server

 Configured servers list: html, typescript, diagnosticls, java, python, cpp, lua, bash, dockerfile, latex, json, css, clojure, php, vim, rust, go

Ah. OK. I've undetstood.
You should specify the source name nvim_lsp instead of nvim-lsp.

Oh. I'm dumb. My bad. Keep up the amazing work, I loved compe and look forward to using this :D

Oh wait, still having issues. LSP and snippets are still not showing up in the completion window.

@AndOrangutan So you defined the your own snippet, right?

Figured it out, those last couple lines in my config weren't needed and were the last issue, works as it should now. Thank You :D


Adds confirmationStrategy to confirmTransaction

Update SendOneLamportToRandomAddress to use the new confirmationStrategy in confirmTransaction. This replaces the current deprecated function call.

Thanks!


Allow recreating entities for PurView registry

Description

Resolves #675 partly, we still do not allow updating on any existing features and sources, but we do allow re-creating features and sources as long as the definitions keep the same, and we allow adding new entities into existing containers such as Anchor and Project.
Does this PR introduce any user-facing changes?

[x] No. You can skip the rest of this section.
[ ] Yes. Make sure to clarify your proposed changes.

@windoze - From end user perspective, if I run the sample notebook two times against Purview backend, it should work, since feature definition will remain same?
Can you add more examples of what is allowed and what is not in terms of Purview updates?

@windoze - From end user perspective, if I run the sample notebook two times against Purview backend, it should work, since feature definition will remain same?
Can you add more examples of what is allowed and what is not in terms of Purview updates?

The current design is: the recreating should succeed as long as all attributes remain exactly the same as the old one, and the returned id also remains the same. All other cases will trigger a 402/409 conflict error. Please report if you see any other behavior as they should be bugs.
We added this constraint for future compatibility since we still didn't figure out what should be the most appropriate behavior to update the existing entities.


what is the base model you used with object detection?

as you developed an object detection on coral board but my question is which is your base model like mobilenetv2, SSD, etc. in your case what it is? you're using official supported base model from google dev board team or something else other than that?

I believe the model is mobilenetV2 with SSDLite.

Yes. @lcycoding is correct.


reserve epsilon feature

Add feature to allow the data depositor or analyst to "reserve epsilon" for later releases.
e.g apportioning only part of the total budget to the current release

Closing. Epsilon is allotted when creating/assigning AnalysisPlan objects


Writer.Flush: properly compute line count for buffers longer than 2 lines

Fix #32

Hello @gosuri , thanks for this nice library. What do you think of this PR? It fixes a bug for me and seems to work well to me.
thanks.

Hi @fkaleo
To let you know, I've made a fork of this project as it hasn't been updated for years. Here is the link : https://github.com/Adaendra/uilive/tree/master
In the latest version by my side, I've included your MR.


Great github action for release

See https://github.com/johanhaleby/aspectj-maven-plugin/pull/3/files

https://github.com/johanhaleby/aspectj-maven-plugin/pull/3


Removed unused variables and replaced var declations with let and const where appropriate.

resolves #14
Also contained in this PR:

necessary spacing has been added to the code to make it more readable

@iamziike please resolve conflicts


Add cop metadata

This is the foundation to support the planned RuboCop behaviors (currently only Safe and SafeAutocorrect affect the actual behavior).
As far as I know we don't have any cops that are unsafe by design, so I've just added the versions. Now we can start also mark cops as Safe: false or SafeAutocorrect: false

Before submitting the PR make sure the following are checked:

[x] Feature branch is up-to-date with master (if not - rebase it).
[x] Squashed related commits together.
[x] ~Added tests.~
[x] ~Added an entry to the changelog if the new code introduces user-observable changes.~
[x] The build (bundle exec rake) passes (be sure to run this locally, since it may produce updated documentation that you will need to commit).

I’m not sure how to review this PR.

Basically this just follows on what was done in Rubocop, so I guess someone else from @rubocop-hq/rubocop-core can take a look.
Btw, we can use Safe and SafeAutocorrect without importing the versions, they are independent ;)

Looks good to me! Great work!

Ah, could you rebase please, to get the 2.6 builds running?

@Darhazer I think forgot to merge this. :-)

As far as I know we don't have any cops that are unsafe by design

I thought we had a few but I'd have to re-examine the cops to jog my memory.

When this lands, we should add RSpec/DescribedClass to the list of unsafe autocorrects: https://github.com/rubocop-hq/rubocop-rspec/issues/795.
Also, i think it might be safe if we skipped blocks by default (which , btw, maybe we should consider making the default), but it's interesting because I wonder how many cops might be safe with certain options but not others. I wonder if there should ever be a system for safe under certain conditions (though I suspect that would add too much complexity).

We should add RSpec/PredicateMatcher to the list of unsafe autocorrects as well.

@Darhazer do you need some help on that one?
I've heard on the grapevine that RuboCop 1.0 may be released this week! 🎉

@pirj I'm pretty sure he won't mind help. :-)

 1.0-ready ✔

Thank you @pirj
Now I can't approve it as I'm still the author :joy:
From now on we should introduce the new cops as pending, maybe even those that are currently not released?

I think that's a good idea.

I'll merge while @bquorning is not looking :D

From now on we should introduce the new cops as pending, maybe even those that are currently not released?

👍
Or we can do a 2.0 release, bump the dependency to rubocop 1.0, and start following the same versioning policy. Probably with a reduced release cycle, say, 6 months.


adding some structure, data and readme updates

this one adds the /data directory w/ two sub-dirs (county and uauc), and a processing directory with the processing code.  i added some readme's, but still have some work to do on the readme's.  figured it was a good place to start.

Merging. One quick thought in #2, for later.


GitHub Setup

Set up Project for GitHub.

Resolved in https://github.com/RommelTJ/LearningTypeScript/pull/2


Error: Instances [[i-0804c5]] not in a valid state for account 28878

Do you know what could be causing this error?
Error: Instances [[i-0804c5]] not in a valid state for account 28878

likely the instance is still booting up

It's a hilariously long cycle between the question (September 2022), the first answer (October 2023) and now my answer (November 2024).
One of the reasons that this can happen is if the Systems Manager Agent is not running on the instance. I have a fleet of Windows instances and every so often one of them will get a full hard disk. The first sign we get that the disk is full is usually that Systems Manager agent stops responding. Often a reboot will fix that. If your instance is Linux, check that the system manager agent is running/communicating. An easy way to do that is to go to the EC2 console, click on the instance, then click 'Connect'. If the agent is not communicating properly, you'll see an error like:

The SSM Agent was unable to connect to a Systems Manager endpoint to register itself with the service.

Another reason for this is if you screw up and ask SSM to send_command() to an instance id that doesn't exist in that region. I've goofed that up a couple times where I get a list of instances from one region, then call send_command() in the wrong region with the instance IDs.


Placement of the MessageComposerView

When having a UITableViewController within a UITabBarController, the MessageComposerView gets below the bottom bar. There should be a way to force it further up.

I recently had the same issue. I add an NSString in both textview delegate methods (beingEditing and endEditing) and I set an IF ELSE (Not IF, ELSE IF) on the NSString set in the beginEditing method. I got rid of the _keyboardHeight instance and change it to 49(height of TabBar). This is what it looks like
if ([self.heightNeeded isEqualToString:@"Lower"]) {
            frame.origin.y = ([self currentScreenSize].height - [self currentKeyboardHeight]) - frame.size.height;
 } else {
            frame.origin.y = ([self currentScreenSize].height - [self currentKeyboardHeight]) - frame.size.height - 49;
 }

This i put in the "layoutSubview" method This method get called initially when loading and when the keyboard comes up and goes down.

Could you not initialize the MessageComposerView via initWithKeyboardOffset:49 andMaxHeight:CGFLOAT_MAX? This should provide you with the desired result without requiring you to edit the lib manually

@oseparovic it starts off initially above the tabbar but when the keyboard goes up, that offset stays the same and the input view is mantains that offset above the keyboard. So there is a huge space.

Ah I see. Then initWithKeyboardOffset:andMaxHeight is definitely not the right function for the job. Perhaps I need to add another initializer for that scenario. Please contact me via my contact form here http://www.thegameengine.org/contact/ and I'll email you back asap.

I mean what ever I am doing here seems to be working. But I have also found some more complex messagingviewcontrollers like SOMessagingViewController and JSQMessagingViewController. So I am going to give those a shot

On Feb 11, 2015, at 2:40 PM, oseparovic<EMAIL_ADDRESS>wrote:
Ah I see. Then initWithKeyboardOffset:andMaxHeight is definitely not the right function for the job. Perhaps I need to add another initializer for that scenario. Please contact me via my contact form here http://www.thegameengine.org/contact/ http://www.thegameengine.org/contact/ and I'll email you back asap.
—
Reply to this email directly or view it on GitHub https://github.com/oseparovic/MessageComposerView/issues/1#issuecomment-73949248.

I ran into this same problem trying to use this over a tab bar, so I set the keyboard offset to 50 and then I went into your .m file and change the keyboard height to self.keyboardHeight - 50. I'm not sure that's the best way to do it, but it works.

(CGFloat)currentKeyboardHeight {
if ([self.messageTextView isFirstResponder]) {
return self.keyboardHeight - 50;
} else {
return 0;
}
}

That sounds pretty reasonable. I'm still trying to find a good way to determine if the parent is a UITabBarController from within the MessageComposerView but it's tricky. Thanks for your suggestion


[TESTING] mobile/ci: Fix flaky start Android emulator step

This is an attempt to fix multiple timeout failures when starting the Android emulator by doing the following:

Update the system image to Android 30 instead of 29.
Use a specific hardware profile, i.e. Pixel 4 instead of the default hardware profile since the default hardware profile only has 128MB.

This also adds some logging when starting the emulator instead of redirecting everything to /dev/null to make debugging easier.
Risk Level: low
Testing: CI
Docs Changes: n/a
Release Notes: n/a
Platform Specific Features: n/a

@fredyw i wondered about this previously - im not sure if increasing the timeout is helpful or just slows things down
like when its successful (eg here https://github.com/envoyproxy/envoy/actions/runs/6576983948/job/17870664914) it only takes 4-5 mins to start the emulator, so not sure if its timing out or just hanging for some other reason

@fredyw i wondered about this previously - im not sure if increasing the timeout is helpful or just slows things down
like when its successful (eg here https://github.com/envoyproxy/envoy/actions/runs/6576983948/job/17870664914) it only takes 4-5 mins to start the emulator, so not sure if its timing out or just hanging for some other reason

Yeah, I'm only experimenting with this. Increasing the timeout didn't seem to fix it, but at least now we have some logging, which can hopefully help us to debug this issue.

/retest

/retest

/retest

/retest

/retest

/retest

/retest

/assign @abeyad @RyanTheOptimist

/assign @phlax

@fredyw would you mind pushing an empty commit or similar so i can see a fresh ci run

@fredyw would you mind pushing an empty commit or similar so i can see a fresh ci run

Done. Re-running the CI now.

lgtm, thanks @fredyw
now we have decent logging im wondering about increasing the timeout - maybe we should increase some - seems it can take ~10 mins - and at least with this we will be able to see the output
also wondering about the nil device and whether that is causing startup issues and/or can be resolve

Good idea on increasing the timeout. I noticed that, too. I will send a PR to increase it to 15 mins. As for the nil device, it's normal. It will stop once the emulator is fully booted up as in https://github.com/envoyproxy/envoy/blob/dc841d18fd45ff8043dcc0d3bb9e7f8e84768d55/mobile/ci/start_android_emulator.sh#L13

hmm maybe we should supress that output, its not very helpful


Manager is sometimes started after demotion

If a certificate renewal runs right before a demotion, it causes the manager to restart instead of shutting it down. After spec change role is changed to worker but if there is already a renewal going on, next downloaded certificate will be a manager one. This will change the role back to "manager" and make the manager start again.
This causesTestApiSwarmPromoteDemote https://github.com/docker/docker/blob/1225e3e6216b0388ed5f3985d7263efdb6e22043/integration-cli/docker_api_swarm_test.go#L176 to sometimes fail.
https://jenkins.dockerproject.org/job/Docker Master/label=rhel && devicemapper/6520/changes
https://jenkins.dockerproject.org/job/Docker Master/label=centos && devicemapper/6477/console
2016-08-16T15:31:10.451587012Z Certificate renewal starts in deamon2 https://gist.github.com/tonistiigi/13734a6e518a107673eaae7712eb1095#file-daemon2-d4662005-log-L140
2016-08-16T15:31:11.881208322Z Second daemon is demoted https://gist.github.com/tonistiigi/13734a6e518a107673eaae7712eb1095#file-daemon1-d89581834-log-L202
2016-08-16T15:31:11.940996098Z Manager cert is downloaded in daemon2. Likely that the demotion was already triggered before that. https://gist.github.com/tonistiigi/13734a6e518a107673eaae7712eb1095#file-daemon2-d4662005-log-L158
2016-08-16T15:31:11.950649719Z Second daemon has shut down its manager https://gist.github.com/tonistiigi/13734a6e518a107673eaae7712eb1095#file-daemon2-d4662005-log-L161
2016-08-16T15:31:11.962683845Z New manager is started https://gist.github.com/tonistiigi/13734a6e518a107673eaae7712eb1095#file-daemon2-d4662005-log-L169
2016-08-16T15:31:13.887635695Z Second daemon downloads the worker certificate https://gist.github.com/tonistiigi/13734a6e518a107673eaae7712eb1095#file-daemon2-d4662005-log-L184
@LK4D4

@tonistiigi do you think it should be second cert renewal with new role?

@LK4D4 I'm not sure why the manager didn't shut down again after second renewal https://gist.github.com/tonistiigi/13734a6e518a107673eaae7712eb1095#file-daemon2-d4662005-log-L184 after it got the worker cert. Maybe another issue.
This code originally only switched the node role after a certificate with a new role was downloaded. But this had some issues if fast promote/demote cycles if I remember correctly. If we can't bring back this logic then maybe we can cancel all current renewals when we get a signal for demotion.

Do we still have bugs on current upstream? anyone can give a confirm?

@xiaods I think this issue is still valid.

Closing now that #1992 was merged.


Update bundle

Not sure if this is helpful, feel free to ignore. Let see if CI passes anyway.

Cool. Merging this should automatically close all the superseded PRs.

Damn man.


Engine API: add getPayloadBodies method

Proposal

Adds engine_getPayloadBodies method to the Engine API
Given the block_hash values the method returns ExecutionPayloadBody objects which consist of transactions list of the corresponding payloads
The logic of the new method maps on the logic of existing GetBlockBodies message in ETH protocol -- this is the crux of this proposal

Concerns
The main concern is that retrieval of the bodies by hashes doesn't allow for utilizing linearity of block data as if the request was made by block numbers. Utilizing the linearity property opens up an opportunity for more optimal way of accessing this information, especially it makes sense for the finalized part of the blockchain.
See https://github.com/ethereum/execution-apis/issues/137 and discussion thread for more details
Implementations

Nethermind, GetPayloadBodiesV1Handler.cs#L40

Would it make more sense for the interface here to be getPayloadBodies(hash) -> body and lean into JSON-RPC batching instead of enshrine the batching? Otherwise, looks good to me.

Good question. I don't know how convenient JSON-RPC batching interface is across different client libraries. I think that as it's straightforward for EL clients to implement this proposal (and it is because of already existing logic) we can accept batching to be enshrined.

Would it make more sense for the interface here to be getPayloadBodies(hash) -> body and lean into JSON-RPC batching instead of enshrine the batching? Otherwise, looks good to me.

Valuable input on that from Discord https://discord.com/channels/595666850260713488/692062809701482577/931309418879012945:

My preference is definitely for adding the batching logic in the method itself rather than relying on JSONRPC batch support.  At fair number of clients/libraries don't have full support for JSONRPC batch requests (including IIRC github.com/ethereum/go-ethereum/ethclient).

Batching enshrined in a method semantics is preferable.

Is there any concern about JSON ser/deser overhead? If I'm following correctly it's sending what was previously p2p traffic over json. I'd guess that the overhead will be significant and may become the bottleneck for sync.

Is there any concern about JSON ser/deser overhead? If I'm following correctly it's sending what was previously p2p traffic over json. I'd guess that the overhead will be significant and may become the bottleneck for sync.

Each transaction is still in raw format:
Array of transaction objects, each object is a byte list (DATA)
So I think the ser/deser overhead is pretty minimal.

When EL clients are snap syncing and CL clients are up to head optimistically. Will the EL clients respond to requests of getPayloadBodies?

We're seeing issues with ELs timing out under a flood of eth_getBlockByHash requests, so I think it would be great to prioritise this standard if we can find the bandwidth :pray:

We're seeing issues with ELs timing out under a flood of eth_getBlockByHash requests, so I think it would be great to prioritise this standard if we can find the bandwidth 🙏

Agree. We have some users that reporting this problem.

Closing in favour of https://github.com/ethereum/execution-apis/pull/218


feat: Implemented so that ClusterManager#setAlgorithm can be use.

Existing problems

ClusterManager#setAlgorithm is not available.
clusterContent and clusterItemContent are not used when clusterRenderer is specified.

As described in the Issue, we thought it would be less confusing for users to separate the low-level API and the high-level API.
Clustering Composable Function that takes a clusterManager as an argument, memberClusterRenderer and memberClusterManager are implemented and make public.
The existing Clustering Composable Function calls the low-level Clustering Composable Function.

Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

[x] Make sure to open a GitHub issue as a bug/feature request before writing your code!  That way we can discuss the change, evaluate designs, and agree on the general idea
[x] Ensure the tests and linter pass
[x] Code coverage does not decrease (if any source code was changed)
[ ] Appropriate docs were updated (if necessary)

Fixes #420 🦕

There are 10 icons set, but 11 are displayed at main branch. I'm looking into whether the issues are related.

Even if you branch Clustering with if, the marker remains displayed. This issue may be related.

fixed.

java.lang.IllegalStateException: Maps API key not specified
https://github.com/googlemaps/android-maps-compose/actions/runs/6847027102/job/18629617389?pr=421

The test was successful in my environment. I didn't know the cause.

This will also help us customize the algorithm for developing and testing fixes to #380


fix: Markdown anchor links in package readMe.md broken #2712

Fixes issue #2712 by replacing all href="# occurrences generated by marker with href="${req.headers.referer}#.
Can't be fixed using baseUrl option from marker as it does not allow a base-URL to be added to an URL starting with #.

Sorry PR on wrong branch


Implement optionalExpand feature

See #26
This is particularly nasty. We don't really have a choice other than passing these parameters in with each of MoatDatas records.
We really need a Reader in MoatM to handle passing around the Options. Additionally, a Reader in the pretty swift module would help as well.

Looks like a ton of other changes are included in this PR, making it hard to review. Maybe a rebate is in order?

Yeah, this one is based off of #28, we'll need to merge that before this one. I did this because rebasing these tests are pretty annoying each time for each feature

Makes sense

Oof, now I remember why I never fully implemented this. Lots of complexity for something that may see little use.
I'll leave it up to you if you think we should move forward with including it.

I think it is worth doing but I think it highlights some really nice refactoring we could do to clean future features like this up.

MoatM needs Reader Options
Each record in MoatData should carry a GenerationOptions field to contain things like Annotations, optionalExpand, etc

i.e.
data MoatData
  = MoatStruct { structExpandOptional :: Bool
                        , structAnnotations :: Annotations
                        , structData... :: Data...
                        }
  ...

becomes
data MoatData
  = MoatStruct { structGenerationOptions :: GenerationOptions
                        , structData... :: Data...
                        }
  ...

I think this would allows us to pull out the TH into a separate function and clean the code up quite a bit.

With a common type for all fields, we can also add a Reader GenerativeOptions to the pretty code too

I think these three things would make new generative features quite a bit easier.
The other option was allowing one to change the generative options at generation time, however I prefer the former because the decision is made a compile time and the generative options can be chosen for each of your types. e.g. Maybe you only want to optionalExpand for one specific type for some reason.

I think I'll leave this one open as a reminder to do this refactoring


i2ctool dev command now uses an empty write header to scan the I2C bus

Summary
The i2ctool dev command will now use an empty 'write' header to scan the I2C bus by default (if no register is provided), which results in the detection of more I2C devices by default (including the SHT41 and MS5611).
This PR closes #994.
Impact
This should save some developers time debugging or wondering why their devices didn't show up (and trying to wrongly diagnose a hardware issue like myself :laughing:).

 Additionally, the general call address of 00 does not appear anymore, which is good as it is not a real device.

@linguini1 did you test i2ctools on Linux and look at logic analyzer? Let's confirm Linux also do this way.

I won't have a change to access a logic analyzer for a little while. I had a look at the description for the Linux I2C tools here:
https://github.com/MLAB-project/i2c-tools/blob/master/tools%2Fi2cdetect.8
It seems they either use a single byte read (as was the previous implementation) or a write command, depending on the address by default. Although they do warn that scanning the bus can affect the I2C devices since there is no actual standard for scanning in the I2C protocol.

@linguini1 did you test i2ctools on Linux and look at logic analyzer? Let's confirm Linux also do this way.

I won't have a change to access a logic analyzer for a little while. I had a look at the description for the Linux I2C tools here:
https://github.com/MLAB-project/i2c-tools/blob/master/tools%2Fi2cdetect.8
It seems they either use a single byte read (as was the previous implementation) or a write command, depending on the address by default. Although they do warn that scanning the bus can affect the I2C devices since there is no actual standard for scanning in the I2C protocol.
Perhaps a good compromise would be trying to read one byte from the address, and if that fails, attempt to send a 0 byte write?

I'm curious to know if Linux i2ctool is able to detect these sensors.
@linguini1 maybe to avoid David's worryings (in fact it could break I2C for arch that don't work when msgc is 0 or msg is NULL), I suggest you implementing your suggestion as special search case (like -z for zero byte reading).

@linguini1I suggest you implementing your suggestion as special search case (like -z for zero byte reading).

Do you think it would be worth while for the i2ctool to have some #ifdef to check the architecture and disable the -z functionality for architectures which do not support the 0 byte write?

@linguini1I suggest you implementing your suggestion as special search case (like -z for zero byte reading).

Do you think it would be worth while for the i2ctool to have some #ifdef to check the architecture and disable the -z functionality for architectures which do not support the 0 byte write?

verbose message in either situation could be helpful..?

@linguini1I suggest you implementing your suggestion as special search case (like -z for zero byte reading).

Do you think it would be worth while for the i2ctool to have some #ifdef to check the architecture and disable the -z functionality for architectures which do not support the 0 byte write?

I don't think we can check the architecture now, because we don't know which arch will fail and which will work. My suggestion for now is adding the "-z" and maybe include a suggestion just before the scan start:
If your device didn't appear in the bus, try to use the -z option

@davids5 As far as I can tell, this should be safe. A 0 byte write means that any sensors which allow a write with no register address first specified should not perform the write regardless, since there is no data sent to be written. A device should see the stop condition and discard the write request, but I suppose it is possible for a poorly designed device to not handle this case.
Architectures which do not support a 0 byte write through their I2C driver will return some error, which will cause any device to just not appear. This is fine since with the addition of the -z option, the default behaviour is still a single byte read. The -z is a fallback for the user to check in case they do not see their expected device appearing in the scan.
Please let me know if there's anything I've overlooked!

@davids5 the patch look fine since zero write is guarded by option in the new change.


Update ProjectReactor dependency to 3.x

Will be implemented in https://github.com/redisson/redisson/issues/958


[Feature] Add hudi connector

Search before asking

[X] I had searched in the feature and found no similar feature requirement.

Description
apache hudi is a very popular data lake solution，it has a official release flink connector, it streamx could integration it internally, it will make streamx api fit much more scenario.
Usage Scenario
No response
Related issues
No response
Are you willing to submit a PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

do you want to contribute this PR? welcome !😁

Do streampark still need the hudi connector of datastream api version?
At the FFW2022 conference, I saw that Streammark wanted to integrate existing connectors to achieve data integration in its future planning.

Welcome to propose again


Age restricted error being thrown when trying to access the streams of videos that aren't age restricted

Description
Trying to access certain videos through Pytubefix 3.0.0 throws an age restricted error, despite the video not actually being age restricted. This might be due to the user agent, which has been changed to "ANDROID_MUSIC" in the latest version of innertube.py. I fixed this temporarily by setting the InnerTube user agent to "IOS", but I'm not sure how the functionality of the library itself is impacted by this change as I'm only using it for audio downloads (mostly through itag 140).
To Reproduce
Steps to reproduce the behavior:
Try to access the streams on one of the following videos:
https://www.youtube.com/watch?v=jDwVkXVHIqg&t=25s
https://www.youtube.com/watch?v=CP9tRsE5uaQ&t=276s
Expected behavior
We should be able to access any of the videos above without authentication as they aren't age restricted in any way.
Desktop (please complete the following information):

OS: Debian 12
Python Version 3.11
Pytubefix Version 3.0.0

Some my audios also not downloading.
Silent Video Not Available.mp3 file is downloaded when:
stream = ytsong.streams.get_audio_only()
stream.download(mp3=True)

Some my audios also not downloading.
Silent Video Not Available.mp3 file is downloaded when: stream = ytsong.streams.get_audio_only() stream.download(mp3=True)

For when Youtube send a fake audio that is silent, to fix you have to upgrade pytubefix to version 3.0.0 final release.
I am also facing problems to download some audio files (itag = 140) with pytubefix 3.0.0 final release.
It is raising the Exception of "Unavailable video". I am investigating and will post more details later.

I did an investigation on the Exception of "Unavailable video" I was getting as mention in my previous comment
In this particular case it is not an error, but the channel owner did set some geographical restriction for the video, that is available only for viewers of a particular country, and pytubefix reports as "unavailable" when the download is requested.

Description Trying to access certain videos through Pytubefix 3.0.0 throws an age restricted error, despite the video not actually being age restricted. This might be due to the user agent, which has been changed to "ANDROID_MUSIC" in the latest version of innertube.py. I fixed this temporarily by setting the InnerTube user agent to "IOS", but I'm not sure how the functionality of the library itself is impacted by this change as I'm only using it for audio downloads (mostly through itag 140).
To Reproduce Steps to reproduce the behavior: Try to access the streams on one of the following videos: https://www.youtube.com/watch?v=jDwVkXVHIqg&t=25s https://www.youtube.com/watch?v=CP9tRsE5uaQ&t=276s
Expected behavior We should be able to access any of the videos above without authentication as they aren't age restricted in any way.
Desktop (please complete the following information):
* OS: Debian 12

* Python Version 3.11

* Pytubefix Version 3.0.0

I tested the two videos you mention and got the same "age restricted" issue.
Tested on Windows with python 3.9 and pytubefix 3.0.0.

I did an investigation on the Exception of "Unavailable video" I was getting as mention in my previous comment In this particular case it is not an error, but the channel owner did set some geographical restriction for the video, that is available only for viewers of a particular country, and pytubefix 3.0.0 reports as "unavailable" when the download is requested.

My update:

Link https://youtube.com/watch?v=-lPN2sh6C5E
I upgraded the package to latest, and I also download by itag 140 now.
Download is done successfully, but still there is no sound.
Running script locally on a Mac (country Kazakhstan)
Video with sound available in browser (Chrome Incognito mode)
I tried to download video itag==18 (not sound), the picture on video says that it's not available on my appliaction

Link https://youtube.com/watch?v=-lPN2sh6C5E

I upgraded the package to latest, and I also download by itag 140 now.

Download is done successfully, but still there is no sound.

I just tested your video link.
I could download the audio with itag 140 without any issue.
The file has the correct sound it is expected to have (not silent).

Started working now on my remote VPS.
Locally and via VPN - it doesn't. So probably some issues related to location/ip from Youtube.

update to latest version -> pytubefix==4.0.0

The videos I mentioned in the original post are now downloading correctly as of 4.0.0. Moving back to using the Android client as default seems to have solved the issue.

The videos I mentioned in the original post are now downloading correctly as of 4.0.0. Moving back to using the Android client as default seems to have solved the issue.

I put it in ANDROID_MUSIC temporarily just so the library wouldn't be stuck 100%, it's not true the problem is in the parameter passed by the ANDROID client, which was causing nothing to work, every time this happens we will have to wait to find the parameter to be passed .


Do not prompt to save as admin for a file after authenticated once

VS Code has "Retry as Admin" feature when trying to save files with insufficient access permissions. It was discussed in http://github.com/Microsoft/vscode/issues/1614.
The problem is that this access elevation after first retry as admin is not remembered for some time and I am getting password prompt all the time. Comparing to this sudo has timestamp_timeout which controls how sudo expires.

Please lobby for this over at https://github.com/jorangreef/sudo-prompt/commits/master

If your file has name myfile.py
than you would type
sudo chmod 777 myfile.py - not save thought but works


Provide a way to set "One Time" as an interval in URL params

https://opencollective.com/embed/papertree-bedstuy/contribute/custom-43382?interval=one-time should force the step details to pre-select "One Time" contribution.

Resolved by https://github.com/opencollective/opencollective-frontend/pull/8508 + https://github.com/opencollective/documentation/pull/138

Did this change also include removing the "platformContribution" parameter?

@rudyfraser Not related to this specific issue, but yes https://github.com/opencollective/opencollective-frontend/pull/8508 took care of removing the platformContribution parameter that was marked as deprecated and planned for deletion since long ago.


Re-send

Nice script indeed!
Feature request - Re-send email
Allow the user to modify email config(without seeing the matches) and resend the info from "secret-santa-email-record.txt"
I had an issue now where two emails wasn't received, so this could help this issue by simply changing to different email accounts

That's a really good idea, thanks! I'll definitely implement that.


Fail to build CUDA suport

tried to build tf OD with GPU support on a cloud instance by running
#!/usr/bin/env bash
docker-compose -f docker-compose.yml
-f docker-compose.override.yml
-f ./components/cuda/docker-compose.cuda.yml
-f ./components/tf_annotation/docker-compose.tf_annotation.yml
up -d --build
got error and here's the following message
Step 22/48 : RUN if [ "$CUDA_SUPPORT" = "yes" ]; then         /tmp/components/cuda/install.sh;     fi
---> Running in 2ba0c7259a15
Executing: /tmp/tmp.E9gKbYvfQy/gpg.1.sh --fetch-keys
http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub
gpgkeys: no key data found for http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub
gpg: no valid OpenPGP data found.
gpg: Total number processed: 0
gpg: keyserver communications error: key not found
gpg: keyserver communications error: bad public key
gpg: WARNING: unable to fetch URI http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub: bad public key
Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [109 kB]
Get:2 http://archive.ubuntu.com/ubuntu xenial InRelease [247 kB]
Get:3 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [853 kB]
Get:4 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB]
Get:5 http://archive.ubuntu.com/ubuntu xenial-backports InRelease [107 kB]
Get:6 http://security.ubuntu.com/ubuntu xenial-security/restricted amd64 Packages [12.7 kB]
Get:7 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [557 kB]
Get:8 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages [1558 kB]
Get:9 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 Packages [6113 B]
Get:10 http://archive.ubuntu.com/ubuntu xenial/restricted amd64 Packages [14.1 kB]Get:11 http://archive.ubuntu.com/ubuntu xenial/universe amd64 Packages [9827 kB]
Get:12 http://archive.ubuntu.com/ubuntu xenial/multiverse amd64 Packages [176 kB]
Get:13 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [1248 kB]
Get:14 http://archive.ubuntu.com/ubuntu xenial-updates/restricted amd64 Packages [13.1 kB]
Get:15 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [968 kB]
Get:16 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 Packages [19.1 kB]
Get:17 http://archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages [7942 B]
Get:18 http://archive.ubuntu.com/ubuntu xenial-backports/universe amd64 Packages [8532 B]
Fetched 15.8 MB in 12s (1296 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package cuda-cudart-9-0
E: Unable to locate package cuda-libraries-9-0
E: Unable to locate package libnccl2
E: Unable to locate package libcudnn7
ERROR: Service 'cvat' failed to build: The command '/bin/sh -c if [ "$CUDA_SUPPORT" = "yes" ]; then         /tmp/components/cuda/install.sh;     fi' returned a non-zero code: 100

@shuait , does the command work locally on your machine?

@shuait
Sometimes NVIDIA has problems with keys. Is the problem still observed?

I have same problem in my own machine.
OS: Ubuntu 16.04

@WeiyeShao
Still have the problem?

problem still exists... I can download the public key from my browser however.

Hi @shuait , we don't see the issue on our side. Could you please help us to investigate? We are not able to resolve such kind of problems remotely. It can be a problem with proxy or some settings of your computer. Just try to repeat the same actions on your host and see if you can reproduce the issue.

Switched to another cluster in Singapore and the problem resolved by itself. Seems to be a firewall related issue.

@shuait , thanks for the note. I will close the issue for now.


Feature(Examples): Aws DynamoDb Authentication Example

Links https://fivetran.height.app/T-784575
Description
Add example to authenticate to Aws using IAM role credentials. This example will fail if no credentials provided.
Also folder changes:

getting started - examples
common patters
database sources - worked examples

Testing
Successfully fetched records from DynamoDb with local testing.

Update docs as well. so that links do not break

Update docs as well. so that links do not break

https://github.com/fivetran/engineering/pull/192607 @fivetran-rishabhghosh

@fivetran-rishabhghosh other PR is merged. Pls approve soon, so I can keep it ready for merge.

Hi @fivetran-rishabhghosh tester changes were merged here and here which should fix you're issue. Pls release them. Let's merge this PR else links in docs won't work.


Dashboard: Integrating amp-animations into a generated amp-story

Summary
This PR updates the amp-story generation logic to include AMP animations if the edit-story feature flag enableAnimation is set to true.
To-do
I had to skip a couple of the new tests I wrote because they're using toBeValidAMP to assert that the generated story is valid, but the validation check fails because we're using amp-animation tags in our generated html.  The stories do animation correctly so I'm not sure if we should be using another tag or if the toBeValidAMP logic needs to be updated (since amp-animations in amp-stories is a new feature).  Once we figure out what's what, I can go back and update these tests (and our logic) so everything passes.
User-facing changes
Technically nothing should change, a generated story without animations should generate the same, but because this touches story generation logic this should definitely be QA'd.
Testing Instructions
Testing stories without animations
1.) Edit an existing story, or make a brand new story (and add elements to it).
2.) Click the "Preview" button.
3.) This should open up a window with the generated story and the pages should look just how they looked in the editor.
Testing stories with animations
1.) Go to Story_Post_type.php and set enableAnimation to true.
2.) Go to the animation UI tool (url: /wp-admin/edit.php?post_type=web-story&page=stories-dashboard#/story-anim-tool).
3.) Follow the instructions found here to add an animation to an existing story in your repo.
4.) Go to the dashboard and find that story you edited and open it up in the editor.
5.) Click on "Preview".
6.) The elements you applied animations too should animate appropriately in the preview window.

This PR address this issue
Screenshots
Simple story generated with enableAnimations set to false:

Same story generated with enableAnimations set to true:

@pbakaus Alrighty, I updated the code so the keyframes are embedded with each amp-story-animation tag and we no longer render amp-animation tags.  This PR touches the generation logic for amp-stories so please feel free to give it a full review to make sure I didn't miss anything.

@mariano-formidable nice, this looks good at first glance! No bandwidth to do a full review, but I think @dvoytenko would probably catch everything.

@dvoytenko I believe I addressed all your comments and the code is updated and ready for a re-review.  Thanks


Request never completes - Node async bug

Describe the bug
This is an intermittent bug that is related to an async timing issue inherent in Node.
When we make a call to the same FileMaker file in short succession, one of the requests never finishes. This is due to a problem with a promise never resolving or rejecting.
Expected behavior
Request should return with the results of the FileMaker server. The promise should either resolve or reject.
Code Examples
I tracked the bug down to the watcher method on the agent model.
if (!global.FMS_API_CLIENT.WATCHERS[this.global]) {
      const WATCHER = setTimeout(
        function watch() {

What happens is on the second request, the watch with this.global uuid actually exists so it doesn't create a new watcher.
However, the problem is that later on in the watcher, this.pending.length is 0 meaning the watcher is destroyed and the second request never completes.
if (this.queue.length === 0 && this.pending.length === 0) {
            clearTimeout(global.FMS_API_CLIENT.WATCHERS[this.global]);
            delete global.FMS_API_CLIENT.WATCHERS[this.global];
          } else {
            setTimeout(watch.bind(this), this.delay);
          }

Therefore this.resolve() is never called and the promise is left in an unresolved state.
What is being hard to track down is how the watcher can exist if it appears that the second request is not in the pending array. The only explanation is that there is some weird timing issue happening in the Node event loop where things don't line up the way they should.
Tests
Working on this now. Hard to reproduce since it's timing based and therefore intermittent.
Additional context
I'm wondering if there is a better way to deal with the request in the queue/pending than a setTimeout that gets renewed or torn down? Did you every look at the possibility of using Node's event emitter system instead?
One potential solution would be to make the watcher unique for each request
global.FMS_API_CLIENT.WATCHERS[`${this.global}-${some other unique identifier from the request}`] = WATCHER;

Would have to figure out what the unique identifier would be and if there are other repercussions of having individual watchers for each request. Probably won't even need that this.global for the watcher. That seems to be there to reuse the watcher but I'm actually not sure what situation that would ever happen, except for the one I'm experiencing that is a bug.

So looks like I got to the bottom of the issue finally.
What was actually happening was that 2 separate clients for the same FileMaker file were being created and somehow they were conflicting with each other.
A little more background in case it helps anyone:
Normally you would initialize a database connection when the app first runs. Then all requests can count on that database connection being open. However, due to the FileMaker API design, in our case we need to initialize the database using fms-api-client on first request. This is because we have multiple users and we need to login to the FileMaker file with that specific username. If we were to initialize when the app started we'd have a huge number of clients initialized, one for each user. Then because we have multiple files we'd need to do a new client for each user * each file. That gets exponentially huge.
Our idea was to build Express middleware that would check the fms-api-client marpat store to see if there is a client already made for that user/database, and if not, initialize one.
// Check if this client already exists
  const existingClient = await Filemaker.findOne({
    name: `${user}-${database}`,
  });

Conceptually it worked great, however every so often we would get a request that never finished. The issue was that if 2 request came in from the same user for the same file, it would create 2 clients. This was due to the first client not being created when the 2nd request was processed so it thought it needed to create another client.
The answer was a little complicated and required storing each client in the express app.locals variable which would happen instantaneously. Of course it would start a promise so we needed to await any promises before allowing the request to proceed.
Anyway, if anyone is interested in the solution I could DM since we're working on a closed source project.
Hope this helps someone since it was a couple days of troubleshooting to solve!


Feature Proposal: Middleware API for event handlers

Hi :smile:
First, I love the probot framework, thanks to the whole team for developing such an awesome framework and making it opensource!
Feature Proposal
Recently I build a probot app for my team for posting pull request notifications to our team's slack channel. In the process, I wanted to see if I could have middleware for github event handlers which allows defining event/context related transformations at an app level scale and not per individual event handler. Less code duplication and less errors. I'm proposing a middleware implementation for probot event handlers. The middleware API I'm proposing works similar to the redux middleware API.
Middleware API
The following shows the event handler middleware api.
next : the next middleware in the chain. The last next is going to be the original event handler. E.g) app.on(event, handler)
context: the context for the current github event being processed
next => context => {
  // do something with `context` or `context.payload`.
  // forward context to the next middleware in the chain
  return next(context);
};

// async middleware
next => async context => {
  // fetch some data from another api.
  const data = await fetchData();
  // forward context to the next middleware in the chain
  return next({
    ...context,
    payload: {
      ...context.payload,
      ...data
    }
  });
};

I'll start with some middleware examples, followed by how I implemented it for my probot application.
To begin, assume I have a probot application subscribed to pull_request events from github.
Middleware 1 - Metadata
A middleware can transform the context.payload object and include parsed metadata. Similar to probot-metadata. However, compared to probot-metadata, reduces the code duplication involved in having each event handler parse the metadata. It's all defined in a single place now.
const parseMetadataFromBody = require('../utils/parse-metadata-from-body');

const parseMetadata = next => (context) => {
  if (!context.payload) {
    return next(context);
  }

  const { payload } = context;

  if (!payload.pull_request) {
    return next(context);
  }

  const {
    pull_request: { body },
  } = payload;

  const { metadata, bodyWithoutMetadata } = parseMetadataFromBody(body);

  return next({
    ...context,
    payload: {
      ...context.payload,
      pull_request: {
        ...context.payload.pull_request,
        body: bodyWithoutMetadata,
      },
      myApp: {
        metadata,
      },
    },
  });
};

module.exports = parseMetadata;

Middleware 2 - Error Reporting
Another middleware implemented was one which can capture any errors and log them. No more need to capture errors at the event handler level. Any error thrown by an event handler will be captured by this middleware. This middleware also is asynchronous!
const slack = require('../notification-clients/slack');

const errorReporting = next => async (context) => {
  try {
    // capture errors thrown by any middleware down the chain
    // and event handlers
    const result = await next(context);
    return result;
  } catch (error) {
    await slack.dumpLogs([`${error.message}\n${error.stack}`]);
    return 'Failed to handle event';
  }
};
module.exports = errorReporting;

Middleware 3 - Appending Additional Pull Request Information
The following middleware will detect pull_request events and automatically append additional information returned by the github api for that pull request and also the files included in the pull request. Now any event handler will have all this information immediately!
const path = require('path');
const addPullRequestInfo = next => async (context) => {
  const { payload } = context;
  if (payload.pull_request && payload.pull_request.number) {
    const {
     pull_request: { number },
      repository: {
        name,
        owner: { login },
      },
    } = payload;

    const { data: pullRequestInfo } = await context.github.pullRequests.get({
      owner: login,
      repo: name,
      number,
    });

   // for example purposes
    const { data: files } = await context.github.pullRequests.getFiles({
      owner: login,
      repo: name,
      number,
      per_page: 100,
      page: 1,
    });

    const { fileTypes, fileTypesPretty } = getFileTypes(files);

    return next({
      ...context,
      payload: {
        ...context.payload,
        pull_request: {
          ...pullRequestInfo,
          files,
        },
      },
    });
  }

  return next(context);
};

module.exports = addPullRequestInfo;

The possibilities are endless. Applying these transformations and data retrieval at the application scale removes a lot of cruft away from individual event handlers.
Implementation Of The Middleware API
Currently, here's how I implemented the middleware api in my probot application. It takes the app object and patches app.on to add the middleware functionality.
const compose = require('./utils/compose');

const applyEventMiddleware = app => (middlewares) => {
  const originalAppOn = app.on.bind(app);
  app.on = (event, handler) => {
    // build chain. a->b->c
    const handlerWithMiddlewares = compose(...middlewares)(handler);
    originalAppOn(event, handlerWithMiddlewares);
  };
};

module.exports = applyEventMiddleware;

Composition of the middleware chain
// ./utils/compose
const compose = (...functions) => {
  if (functions.length === 0) {
    return arg => arg;
  }

  if (functions.length === 1) {
    return functions[0];
  }

  return functions.reduce((a, b) => (...args) => a(b(...args)));
};

module.exports = compose;

If we could have the middleware api integrated into Probot, it would definitely clean this patching up.
Hence, I'm proposing a middleware API for event handlers. This reduces code duplication and errors by defining necessary transforms and data retrieval operations at the application level and not the event handler level.
If this proposal is approved, I would be happy to help contribute for it. Thank you! 😄

Hey @gr2m , sorry for the late response 😄 . I'm confused as to how that would be incorporated into probot from a user perspective? Would we forward middleware definitions in a probot application over to @octokit/webhooks when probot internally registers the webhook? (e.g, app.on('event'))

In Probot, you’ll probably do something like this:
app.hook.error('webhook', async (error, event) => {
  await slack.dumpLogs([`${error.message}\n${error.stack}`]);
  return 'Failed to handle event';
})

hmm, I don't see .hook defined anywhere in the documentation https://probot.github.io/api/latest/classes/application.html. Is .hook available yet?

Probot has been using @octokit/webhooks for a couple years now. There isn't however a user facing way to append data using the transform option


Prevent YOURLS from accessing the long URL to shorten (fetch the page to get its title) #1206

Technical details regarding my environment

YOURLS version: 1.7.2
PHP version: 5.5

I have the same question as has been asked in #1206 however, the links provided there do not work anymore and I cannot read the comments.
My use case is that I am generating a long URL for the user, that will trigger an action when visited. Then I will shorten the URL to make it look better, then send it to the user.
If YOURLS visits the url to get the title, it triggers the action.
Therefore I would like to ask the same question:
"How can I disable fetching of the page from the long URL"
thanks!

Duplicate of #1966

Thank you!


Success measures for Public Programme

Users click through to pages

page views for campaign pages
avg page view per visit for campaign pages

Programme landing page is used (ie. people view pages after viewing landing page)

Data studio: create segment user views Programme landing page during session; data studio:
= page views for campaign pages filtered by segment
less programme landing page views
plus page views for content able to view directly on programme landing page

People are finding pages direct and social media campaigns are working

traffic source analysis of users who view campaign pages (create segment for users who view any campaign page)

Content is being read/played

page views or  avg time on page per campaign page, filtered by page contains campaign UTM

Listing of urls and accounts for monitoring usage:
https://docs.google.com/spreadsheets/d/1yIaLeCk9wgjSlOR0nSLQuuFowPD2gqUSsCksl9cFeqQ/edit#gid=0

Listing of urls and accounts for monitoring usage:
https://docs.google.com/spreadsheets/d/1yIaLeCk9wgjSlOR0nSLQuuFowPD2gqUSsCksl9cFeqQ/edit#gid=0

@jennpb @dannybirchall following discussion on email,
Users click through from list pages to season content

GA content group, "Seasons" for list pages: page views
Data studio: page views for season non-list pages

Season landing page is used

GA landing page: page views

Content is being read/played

Data studio: avg time on page, total time on page for season non-list pages
GA4, Soundcloud and Libsyn: total time, avg time video, podcast plays

People are finding pages direct and social media campaigns are working

GA content group, "Seasons" for list pages: traffic source analysis

@jennpb @dannybirchall following discussion on email,
Users click through from list pages to season content

GA content group, "Seasons" for list pages: page views
Data studio: page views for season non-list pages

Season landing page is used

GA landing page: page views

Content is being read/played

Data studio: avg time on page, total time on page for season non-list pages
GA4, Soundcloud and Libsyn: total time, avg time video, podcast plays

People are finding pages direct and social media campaigns are working

GA content group, "Seasons" for list pages: traffic source analysis

Data for audio and video is limited by source reporting and what I am able to access. Revised measures here:
Content is being read
Data studio: avg time on page, total time on page for season non-list pages
Content is being played
Libsyn total podcast plays
Soundcloud total audio plays
GA4 total video starts
GA4 total video completes

Here is the analysis summary and asset listing I sent to Elayne Hodgson today. The spreadsheet also contains the links I used.
Analysis of Season 16Dec-31Jan2021.xlsx


CLOUD-58111 update warmup images on Amazon for HDP 2.4.3

@biharitomi

Refer to this link for build results (access rights to CI server needed):
http://ci.sequenceiq.com//job/mock-integration-test/137/

Refer to this link for build results (access rights to CI server needed):
http://ci.sequenceiq.com//job/cloudbreak-pull-request/2646/


Issue with outputting Win32 versu Win64 apps

electron-packager . ${APP_NAME} --platform=win32 --arch=ia32 --version=${ELECTRON_VERSION} --out=${OUTPUT_DIR}

electron-packager . ${APP_NAME} --platform=win32 --arch=x64 --version=${ELECTRON_VERSION} --out=${OUTPUT_DIR}

Both of these calls export the same root folder ${APP_NAME}-win32
Shouldn't it be expected that the Win64 version would have 64 as a suffix?

we're using the node nomenclature for platform and architecture, so 64 bit windows would be win32-x64. I think it would be a good idea to include architecture in output directory names.
Related discussion: https://github.com/maxogden/electron-packager/issues/38

Thank you @jden


fix: change redirect to static URL for cypress test

Description
Change redirect location in Cypress test case

Testing required outside of automated testing?

[x] Not Applicable

Screenshots (if appropriate):

[ ] Not Applicable

Rollback / Rollforward Procedure

[ ] Roll Forward
[ ] Roll Back

Reviewer Checklist

[ ] Description of Change
[ ] Description of outside testing if applicable.
[ ] Description of Roll Forward / Backward Procedure
[ ] Documentation updated for Change

:tada: This PR is included in version 4.7.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Connector reads first file but not new files

Hello,
I'm trying to read .csv files from an input directory. The first file I drop there works just fine and is sent to the output topic, and reaches the status COMPLETED in the "connect-file-pulse-status" topic. Any other files I drop there are not read, since it seems that the task is never finished, even though it has finished the file. However, if I restart the connector, it reads all those existing files normally, but does not read new files again.
kafka-connect.log:
2021-02-02 16:00:56,261 INFO [FileSystemMonitorThread] Scheduled files still being processed: 2. Skip directory scan while waiting for tasks completion (io.streamthoughts.kafka.connect.filepulse.scanner.LocalFileSystemScanner)
2021-02-02 16:00:56,261 INFO [FileSystemMonitorThread] Waiting 10000 ms to scan for new files. (io.streamthoughts.kafka.connect.filepulse.source.FileSystemMonitorThread)
2021-02-02 16:00:58,629 INFO [SourceTaskOffsetCommitter-1] WorkerSourceTask{id=oss.probe.2g-adr.source.file-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask)
2021-02-02 16:00:58,629 INFO [SourceTaskOffsetCommitter-1] WorkerSourceTask{id=oss.probe.2g-adr.source.file-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask)

this repeats every 10 seconds according to the "fs.scan.interval.ms"
Connector config:
CREATE SOURCE CONNECTOR "oss.source.file" WITH (
    "connector.class"= 'io.streamthoughts.kafka.connect.filepulse.source.FilePulseSourceConnector',
    "tasks.max"= 1,
    "fs.scan.directory.path"= '/data/adr/input/',
    "fs.scan.interval.ms"= 10000,
    "fs.scan.filters"= 'io.streamthoughts.kafka.connect.filepulse.scanner.local.filter.RegexFileListFilter',

    "file.filter.regex.pattern"= '^(.*?)',
    "topic"= 'oss.raw',
    "skip.headers"= 1,
    "read.max.wait.ms"=1000,

    "internal.kafka.reporter.bootstrap.servers"= 'broker:29092',
    "internal.kafka.reporter.topic"= 'connect-file-pulse-status',

    "fs.cleanup.policy.class"= 'io.streamthoughts.kafka.connect.filepulse.clean.MoveCleanupPolicy',
    "cleaner.output.failed.path"= '/data/adr/error',
    "cleaner.output.succeed.path"= '/data/adr/succeed',

    "filters"= 'Row',
    "filters.Row.extractColumnName"= 'headers',
    "filters.Row.separator"= ';',
    "filters.Row.trimColumn"= true,
    "filters.Row.type"= 'io.streamthoughts.kafka.connect.filepulse.filter.DelimitedRowFilter');

I'm using Docker 19.03.13 and Windows 10, and the confluent platform community edition image

Hi @Joao-Garcia-Bras, can you give the vesion of the FilePulseSourceConnector you are using ? Also, a task is considered to be finished once all scheduled files has been cleanup. So the final status should be CLEANED. Did you see any thrown exception ?

Hi @fhussonnois , I managed to get this working, the connector config was missing the "internal.kafka.reporter.id" property. I think it would be a nice feature to make the connector throw an exception for this, since many of the "high" importance properties fail fast and specify the missing config.

Hi @Joao-Garcia-Bras, the internal.kafka.reporter.id is deprecated since Connect File Pulse 1.3.x. FilePulse uses the connector name directly as a reporter id. Which version are you using ?

Hi @fhussonnois , I am using the kafka-connect-file-pulse:latest image from dockerhub, which in the files says version 1.6.1.
There was no exception, just the log message I pasted above :(

Hi @Joao-Garcia-Bras, I think  internal.kafka.reporter.id is not related to your issue.
I've failed to reproduce your issue, everything is working as expected using the Docker image.
Here is my test :
Docker
$ wget https://raw.githubusercontent.com/streamthoughts/kafka-connect-file-pulse/master/docker-compose.yml
$ docker-compose up -d

Init directories

Config :
$ curl \
    -i -X PUT -H "Accept:application/json" \
    -H  "Content-Type:application/json" http://localhost:8083/connectors/issue-github/config \
    -d '{
        "connector.class":"io.streamthoughts.kafka.connect.filepulse.source.FilePulseSourceConnector",
        "fs.scan.directory.path":"/data/adr/input/",
        "fs.scan.interval.ms":"10000",
        "fs.scan.filters":"io.streamthoughts.kafka.connect.filepulse.scanner.local.filter.RegexFileListFilter",
        "file.filter.regex.pattern":"^(.*?)",
        "task.reader.class": "io.streamthoughts.kafka.connect.filepulse.reader.RowFileInputReader",
        "read.max.wait.ms":"1000",
        "skip.headers": "1",
        "topic":"oss.raw",
        "internal.kafka.reporter.bootstrap.servers": "broker:29092",
        "internal.kafka.reporter.topic":"connect-file-pulse-status",
        "fs.cleanup.policy.class": "io.streamthoughts.kafka.connect.filepulse.clean.MoveCleanupPolicy",
        "cleaner.output.failed.path": "/data/adr/error",
        "cleaner.output.succeed.path": "/data/adr/succeed",
        "tasks.max": "1",
        "filters" : "Row",
        "filters.Row.extractColumnName" : "headers",
        "filters.Row.separator" : ";",
        "filters.Row.type" : "io.streamthoughts.kafka.connect.filepulse.filter.DelimitedRowFilter"
    }'

@fhussonnois we actually facing the same issue with version 2.0.0 in our production environment on a few of the connector instances.
For example I have two connector instances which are configured (almost) identical and the csv files are also pretty much the same. They containing two integer fields. They even running on the same connect node. one is working fine the second one read all the data but not moving the file.
You can see the logs here:
[2021-08-12 14:48:05,697] INFO Closing iterator for: [uri=file:/data/mos/supplier_cleaned/cusomter_2/stockinfo_cusomter_2_20210812_151527.csv, name='stockinfo_cusomter_2_20210812_151527.csv', contentLength=3545983, lastModified=1628774135697, contentDigest=[digest=3751420300, algorithm='CRC32'], userDefinedMetadata={system.inode=-9079213280630669312, system.hostname=prodKafkaConnect000004}]  (io.streamthoughts.kafka.connect.filepulse.source.DefaultFileRecordsPollingConsumer)
[2021-08-12 14:48:05,697] INFO Closing iterator for: [uri=file:/data/mos/supplier_cleaned/cusomter_2/stockinfo_cusomter_2_20210812_151527.csv, name='stockinfo_cusomter_2_20210812_151527.csv', contentLength=3545983, lastModified=1628774135697, contentDigest=[digest=3751420300, algorithm='CRC32'], userDefinedMetadata={system.inode=-9079213280630669312, system.hostname=prodKafkaConnect000004}]  (io.streamthoughts.kafka.connect.filepulse.source.DelegateFileInputIterator)
[2021-08-12 14:48:05,703] DEBUG Completed source file '[uri=file:/data/mos/supplier_cleaned/cusomter_2/stockinfo_cusomter_2_20210812_151527.csv, name='stockinfo_cusomter_2_20210812_151527.csv', contentLength=3545983, lastModified=1628774135697, contentDigest=[digest=3751420300, algorithm='CRC32'], userDefinedMetadata={system.inode=-9079213280630669312, system.hostname=prodKafkaConnect000004}]' (io.streamthoughts.kafka.connect.filepulse.source.KafkaFileStateReporter)
[2021-08-12 14:48:05,703] DEBUG Put object in store with key={"name":"stockinfo_cusomter_2_20210812_151527.csv","path":"/data/mos/supplier_cleaned/cusomter_2"}, object=[source=[uri=file:/data/mos/supplier_cleaned/cusomter_2/stockinfo_cusomter_2_20210812_151527.csv, name='stockinfo_cusomter_2_20210812_151527.csv', contentLength=3545983, lastModified=1628774135697, contentDigest=[digest=3751420300, algorithm='CRC32'], userDefinedMetadata={system.inode=-9079213280630669312, system.hostname=prodKafkaConnect000004}], offset=[position=3545983, rows=147615, timestamp=1628779685196], status=COMPLETED] (io.streamthoughts.kafka.connect.filepulse.state.InMemoryFileObjectStateBackingStore)
[2021-08-12 14:48:06,812] INFO Moving file file:/data/mos/supplier_cleaned/cusomter_2/stockinfo_cusomter_2_20210812_151527.csv to file:///data/mos/supplier_cleaned/_status/processed/stockinfo_cusomter_2_20210812_151527.csv (io.streamthoughts.kafka.connect.filepulse.fs.reader.LocalFileStorage)
[2021-08-12 14:48:06,828] INFO File file:/data/mos/supplier_cleaned/cusomter_2/stockinfo_cusomter_2_20210812_151527.csv moved successfully (io.streamthoughts.kafka.connect.filepulse.fs.reader.LocalFileStorage)
[2021-08-12 14:48:06,828] DEBUG Put object in store with key={"name":"stockinfo_cusomter_2_20210812_151527.csv","path":"/data/mos/supplier_cleaned/cusomter_2"}, object=[source=[uri=file:/data/mos/supplier_cleaned/cusomter_2/stockinfo_cusomter_2_20210812_151527.csv, name='stockinfo_cusomter_2_20210812_151527.csv', contentLength=3545983, lastModified=1628774135697, contentDigest=[digest=3751420300, algorithm='CRC32'], userDefinedMetadata={system.inode=-9079213280630669312, system.hostname=prodKafkaConnect000004}], offset=[position=3545983, rows=147615, timestamp=1628779685196], status=CLEANED] (io.streamthoughts.kafka.connect.filepulse.state.InMemoryFileObjectStateBackingStore)

[2021-08-12 14:52:27,159] INFO Closing iterator for: [uri=file:/data/mos/supplier_cleaned/customer_1/stockinfo_cusomter_2_NOS20210812164452.csv, name='stockinfo_cusomter_2_NOS20210812164452.csv', contentLength=576385, lastModified=1628779510754, contentDigest=[digest=3568421620, algorithm='CRC32'], userDefinedMetadata={system.inode=-4467527262203281408, system.hostname=prodKafkaConnect000004}]  (io.streamthoughts.kafka.connect.filepulse.source.DefaultFileRecordsPollingConsumer)
[2021-08-12 14:52:27,159] INFO Closing iterator for: [uri=file:/data/mos/supplier_cleaned/customer_1/stockinfo_cusomter_2_NOS20210812164452.csv, name='stockinfo_cusomter_2_NOS20210812164452.csv', contentLength=576385, lastModified=1628779510754, contentDigest=[digest=3568421620, algorithm='CRC32'], userDefinedMetadata={system.inode=-4467527262203281408, system.hostname=prodKafkaConnect000004}]  (io.streamthoughts.kafka.connect.filepulse.source.DelegateFileInputIterator)
[2021-08-12 14:52:27,164] DEBUG Completed source file '[uri=file:/data/mos/supplier_cleaned/customer_1/stockinfo_cusomter_2_NOS20210812164452.csv, name='stockinfo_cusomter_2_NOS20210812164452.csv', contentLength=576385, lastModified=1628779510754, contentDigest=[digest=3568421620, algorithm='CRC32'], userDefinedMetadata={system.inode=-4467527262203281408, system.hostname=prodKafkaConnect000004}]' (io.streamthoughts.kafka.connect.filepulse.source.KafkaFileStateReporter)
[2021-08-12 14:52:27,164] DEBUG Put object in store with key={"name":"stockinfo_cusomter_2_NOS20210812164452.csv","path":"/data/mos/supplier_cleaned/customer_1"}, object=[source=[uri=file:/data/mos/supplier_cleaned/customer_1/stockinfo_cusomter_2_NOS20210812164452.csv, name='stockinfo_cusomter_2_NOS20210812164452.csv', contentLength=576385, lastModified=1628779510754, contentDigest=[digest=3568421620, algorithm='CRC32'], userDefinedMetadata={system.inode=-4467527262203281408, system.hostname=prodKafkaConnect000004}], offset=[position=576385, rows=31483, timestamp=1628779946659], status=COMPLETED] (io.streamthoughts.kafka.connect.filepulse.state.InMemoryFileObjectStateBackingStore)
[2021-08-12 14:53:14,920] DEBUG WorkerSourceTask{id=41eb8d123eaccf319f196c62f53566b1-input-external_stock-example-customer_1-207-0} Committing offsets (org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter)
[2021-08-12 14:53:14,920] INFO WorkerSourceTask{id=41eb8d123eaccf319f196c62f53566b1-input-external_stock-example-customer_1-207-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask)
[2021-08-12 14:53:14,921] INFO WorkerSourceTask{id=41eb8d123eaccf319f196c62f53566b1-input-external_stock-example-customer_1-207-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask)
[2021-08-12 14:53:14,921] DEBUG Submitting 1 entries to backing store. The offsets are: {{name=stockinfo_cusomter_2_NOS20210812164452.csv, path=/data/mos/supplier_cleaned/customer_1}={position=576385, rows=31483, timestamp=1628779946658}} (org.apache.kafka.connect.storage.OffsetStorageWriter)
[2021-08-12 14:53:14,951] INFO WorkerSourceTask{id=41eb8d123eaccf319f196c62f53566b1-input-external_stock-example-customer_1-207-0} Finished commitOffsets successfully in 31 ms (org.apache.kafka.connect.runtime.WorkerSourceTask)
````

@fhussonnois its the production system so I will need to test the new plugin first on staging environment.
Is there a reason why its not available via confluent-hub yet?
usually I try to avoid installing connectors by hand because our deployment usually works pipelines via jenkins and we install the needed connectors via the confluent hub.
I mean, I can do that for testing indeed.


Kiki being checked is not accounted for in logic

It used to be, but I probably broke it with the trade item checkmarks.

This was fixed a while back


Service-service auth across K8s cluster

This is a tracking issue to enable service to service authorization at cluster level

hi guys,
Congrats on the project guys. Istio auth looks like something incredibly relevant to what we're doing: building trust between microservices using TLS certs. In our approach we're using Vault as an issuer of short-lived certs for auth within and across our kubernetes clusters.
I was wondering whether the auth across k8s that you're planning will require federation of kubernetes clusters or whether it can be done through pure trust of the CA cert chained that the auth manager is using to issue stuff?

@mwitkow , thanks for your interest. Auth across k8s is a difficult problem. Essentially we need to figure out a way to build up trust train between cluster CAs to make clusters be able to talk to each other.

@mwitkow FYI federated auth for Istio (independent of cluster federation) and the necessary bootstrapping of trust is a goal of the SPIFFE project (see spiffe.io) that several folks on the istio/auth team are contributing to. Feel free to reach out to andrew AT scytale.io if you want to learn more.


RWC Added

Please note: This repository is being rebuilt to accept the new volume of token additions and modifications. PR merges will be delayed.

I agree to not ping anybody on Discord/Twitter/email about this pull request. Instead I will inquire by posting a new comment in the pull request if needed.

PRs are reviewed in bulk and and can take up to two weeks to be merged.
This repository is managed using an auto merge action. Please ensure your PR has no deleted lines, and it will be merged.
Please provide the following information for your token.
Please include change to the src/tokens/solana.tokenlist.json file in the PR.
DON'T modify any other token on the list.
At minimum each entry should have

Token Address: 5EvkywasgapiYDtoAdfvFoVnzQPN1bsap4RUyoLyY9fB
Token Name: REPUBLIC WORLD
Token Symbol: RWC
Logo: (logo should be uploaded under assets/mainnet//*.<png/svg>)
Link to the official homepage of token: https://Republicworld.io
Coingecko ID if available (https://www.coingecko.com/api/documentations/v3#/coins/get_coins__id_):

Auto merge requirements
Your pull request will be automatically merged if the following conditions are met:

Your pull request only adds new tokens to the list. Any modification to existing
tokens will require manual review to prevent unwanted modifications.

Your pull request does not touch unrelated code. In particular, reformatting changes to unrelated
code will cause the auto merge to reject your PR.

Any asset files added correspond to the token address you are adding. Asset files
must be PNG, JPG or SVG files.

Your change is valid JSON and conforms to the schema. If your change failed validation,
read the error message carefully and update your PR accordingly.

No other tokens shares the same name, symbol or address.

For example, this change would be rejected due to unrelated changes:

The bot runs every 60 minutes and bulk-merges all open pull requests to prevent conflicts.
This means that you need to wait up to 60 minutes for your pull request to be merged or reprocessed.

RWC Added


Uncaught TypeError: $file.fileupload is not a function

Everything is working well (including embeding youtube videos) but not images.
When I click the upload image icon, I'll get the error:
medium-editor-insert-plugin.js:1630 Uncaught TypeError: $file.fileupload is not a function

my bad. imported the wrong file

I'm getting the same error midway through the project ... can you provide more details as to which file was wrong and how you corrected it? Im browserifying the npm package, and it should pull in all its dependencies, so not sure what I'm doing wrong. Thanks!


container images view: fix node list

Fixes #4950

@simon3z @abonas please review

added navbar support:

@miq-bot add_label providers/containers, bug

LGTM :+1: @abonas @chessbyte @blomquisg

rebased, reordered entities in app/views/container_image/show.html.haml (no consequences on ui, just so the code is nicer) verified on top of #4981. There is no reason to create a dependency on it though.

@blomquisg @chessbyte please review

@miq-bot add_label ui

@dclarizio @martinpovolny @h-kataria Please review

@miq-bot add_label wip

Closing in favor of #5029 that includes similar changes that should be easier to review together.


Fix a grammar typo in README

Describe what this PR does / why we need it
a grammar typo in README, same as #2819
Describe how you did it
Fixed a grammar typo in README

Please add my dingtalk account steverao2021 and add you to our contributor group.


Failing to integrate Selenium

When Karma is running the tests/js/ui-test.js, we get the following error:
PhantomJS 2.1.1 (Windows 8 0.0.0) ERROR
SyntaxError: Unexpected token 'const'
at c:/Users/Balazs/AppData/Local/Temp/node_modules/selenium-webdriver/chrome.js:117:0 <- C:/Users/Balazs/AppData/Local/Temp/501c2ab91dc0af60d9a1e118f1a119a1.browserify:37205
The aforementioned test begins with:
require('selenium-webdriver')
as suggested here: https://www.npmjs.com/package/selenium-webdriver
Browserify deals with require(), set in the Karma configuration file my.conf.js:
preprocessors: {
  'tests/js/ui-test.js': [ 'browserify' ]
}

Browserify bundles up the selenium dependencies. The chrome.js is the first js file, and line 117 is the first line containing code and the const keyword.
However if I add a const variable in the anywhere in ui-test.js, it doesn't break PhantomJS.

I put the issue on hold for now as solving it proved to be very time-consuming and futile so far.
Instead I'm focusing on running Selenium tests purely with jasmine.


Hover feature does not work with Windows11 PC and v0.8.4 of the extension (Native Windows Crystal, not WSL)

Describe the bug
Even when the Hover feature is selected in extension settings the feature does not work on my W11 PC with latest extension 0.8.4
To Reproduce
Steps to reproduce the behavior:
Select hover in the extension settings
Reload VS Code
Open a .cr source file
Hover over items and nothing happens
Expected behavior
For the hover feature to be active as shown in the gif on the extension splash page
Piece of Code
Pick any code with functions and variables and hover over them
Screenshots
N/A
Desktop (please complete the following information):

OS: Windows 11 Pro, fully up to date
Crystal version: Crystal 1.9.2 [1908c81] (2023-07-19); LLVM: 16.0.3; Default target: x86_64-pc-windows-msvc
I have a native Crystal install, NOT WSL 1 or 2 running
This plugin version: 0.8.4

Additional context
Also when I start VS Code and load in the Crystal file it give this error: "Crystal compiler not found. Some features can through errors" but I have Crystal in the Env. Variables so I cannot figure this out.  Maybe this is why Hover does not work too.  The syntax highlighting does work nicely though.  I will submit another issue/bug report about the compiler not being found.

NOTE: with advice from Margret(nobodywasishere) I installed the beta version v0.8.5 via a vsix file and this bug has been fixed.  Thank you!


Add contract partial based instruction generation

This is an initial implementation of a basic API call to interpolate partials within a contract. Discussion is needed to address the HostOS contract that is currently hard-coded (should be pulled from somewhere else). Also, this functionality applies not just to device types, but any contract could potentially have partials that need to be interpolated. There currently isn't a place to handle such abstract functionality in the SDK (to my knowledge) and it seems like this would be a good place to use Contrato as it is going to be (should be) the de facto implementation of the Contract data structure and operations surrounding them. It might be better for the partial interpolation to live there and then anything that is contract based in the SDK have the ability to use Contrato types and functionality.
This will also be able to easily fix the current issue where the instructions are not correct in BalenaHub and open fleets since new devices added do not appear in the dashboard. This will allow different interfaces and uses of the partials to provide their own overarching template (Discussion for this specific issue on Flowdock: https://www.flowdock.com/app/rulemotion/balenahub/threads/UQ29ow3_IXlkXMOnzT-ea9H6V-q).
https://balena.zulipchat.com/#narrow/stream/349104-balena-io.2Fcontracts/topic/contract.20partial.20based.20instruction.20generation
TODO

[x] Updating the contracts repo to have necessary partials
[x] Improve implementation to handle the few edge cases that don't fall under the usual Etcher based install methods

[x] Fix cases with nested instructions like having different instructions for linux/mac/windows

Depends-on: https://github.com/balena-io/contracts/pull/236
Change-type: minor

Contributor checklist

[x] Includes tests
[x] Includes typings
[x] Includes updated documentation
[ ] Includes updated build output

/rebase

@balena-ci rebase

@balena-ci rebase

Tested this iterating over the getAllSupported result and got all of the instructions:
(await sdk.models.deviceType.getAllSupported()).forEach(async (x, i) => {
  var instructions = await sdk.models.deviceType.getInstructions(x.slug)
  console.log("Instructions for " + x.slug + ":\n");
  console.log(instructions);
});

There are a couple devices that get returned by getAllSupported that do not have contracts (and therefore no partials): smartcube-kbox-a150 and smartcube-kbox-a250. I'm not sure where these are coming from and why they are being returned, but I also don't see them as an option in Balena Cloud. These also do correctly throw an error indicating that the instruction partials are not defined rather than failing in an unexpected way.
I have also attached the following text file with the complete output from running the above code snippet (with a bit of prettying up of the formatting for easier reading and checking: all_instructions.txt

Let me know what you think of the changes made here @thgreasi if they are good to go from this point, then I will start writing tests. I also considered use loadash's template instead of handlebars and decided against it as I still haven't gotten to the point of figuring out what the output will produce.

@thgreasi PR ready to go. The methods have been tested and I will open a PR in docs to show how this looks on production for the final review.


Fix Android 13 screenshot orientation issue

Can you describe the specifics? I tested it and it was fine.
The code for the PR you submitted looks like nothing different. It's just a different way to get Rect

_ No description provided. _

Can you describe the situation? I tested my code without any problem.
The code for the PR you submitted looks like nothing different. It's just a different way to get Rect.

Occasionally, there is a direction error in taking screenshots on Android 13 and above.


Rename artifact detection to movement detection

A small PR.
In a previous PR, the filename for the movement artifact detection was named 'plot_artifact_detection.py' but I realize that it should be "plot_movement_detection.py". This is more in accordance with the new "plot_muscle_detection.py".

thx @AdoNunes
backported


IQL results different with the paper

Hi,
The IQL results from this repo seem to differ from the original paper.
According to the README of the IQL example code here, IQL scores an average raw return of about 1500 on hopper-medium-expert with offline training:
https://github.com/rail-berkeley/rlkit/tree/master/examples/iql

However, the original paper notes that IQL scores 91.5 in normalized average return (which is about 2950 in raw return):
https://arxiv.org/pdf/2110.06169.pdf

Can you take a look at this and check what is causing the difference?
Thank you!

Sorry for the late response, but the IQL experiments in the paper were run in jax and should be reproducible with the other repo: https://github.com/ikostrikov/implicit_q_learning
This reimplementation is pytorch is for convenience and likely has minor initialization differences etc.


Remove shim couch_replicator_manager module

Overview
I noticed that couch_replicator_manager has one foot out the door, and thought I'd try and finish it off.
Testing recommendations
I'll build a full database Erlang release and trigger replication integration tests specifically. This change doesn't reduce test coverage although we remove a line from a module-loading test pertaining to couch_replicator_manager.
Related Issues or Pull Requests
None specifically related to this change.
Checklist

[ ] Code is written and works correctly;
[ ] Changes are covered by tests;
[ ] Documentation reflects the changes;

Should I follow-through on the release testing or is the CI here satisfactory?


Support GRAND (column) ON [TABLE]

I need the ability to restrict access at the column level. PostgreSQL allows you to do this using GRAND (column) ON [TABLE]. But PostGraphQL does not allow this feature to be used, due to the fact that it chooses immediately the whole line, not the columns:
select to_json (__ local_0__) as value from "public". "users" as __local_0__ where "id" is not null and true and true and true order by "id" using <limit all
And for GRAND (column) ON [TABLE] it is necessary that the query be of the form:
select json_build_object ($ 1 :: text, __local_0 __. "name", $ 2 :: text, __local_0 __. "login", $ 3 :: text, __local_0 __. "id") as value from "public". "users" as __local_0__ where "id" is not null and true and true and true order by "id" using <limit all
@benjie  made a patch that fixes this problem in #340, but it was not accepted in the master.

@0x6368656174 #342 builds upon #340 and adds a lot more power (and a number of fixes/tweaks), so it's still open. You can use it now if you like - I do.

This is fixed in v4, we now only request the required columns


BUG: use pin_subpackage template in run_exports

pin_compatible only works in the requirements section. Outside of requirements, pin_compatible does add a version constraint and only adds the package name which can lead to unexpected behavior.
Hi @conda-forge/quantlib, sorry I messed up and used the wrong pinning expression. If you were seeing the wrong version of quantlib in the run requirements of packages that build against this package, this is why.

Checklist

[ ] Used a personal fork of the feedstock to propose changes
[ ] Bumped the build number (if the version is unchanged)
[ ] Reset the build number to 0 (if the version changed)
[ ] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[ ] Ensured the license file is being packaged.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

@conda-forge-admin, please rerender


Show the version of Wagtail being run in the Wagtail Admin all the time—not just when there's newer versions out

Is your proposal related to a problem?
Yes, it's related to a problem—but a minor problem, to be sure. Really low priority.
I manage a fair number of Wagtail sites; some run on hosts and some in containers. Configurations vary. It'd be nice to know what version of Wagtail is running where.
I can get that information when there's a new version of Wagtail:

But it'd be a nice little improvement to quality of life if the Admin had this info always available.
Describe the solution you'd like
I'd like to see a little info panel somewhere in the admin that had something like the following:

Wagtail version: 5.2
Django version: 4.2.6
Python version: 3.11.6

Heck, to be really thorough, you could add in:

WSGI: yes
Server: Gunicorn 20.1.0
My IP: <IP_ADDRESS>6

(The "My IP" is nice for those Wagtail systems that make outgoing connections for app-specific reasons so I can make firewall exceptions to those destinations. That's a niche use case, though.)
Other ideas: load average, detection if inside a container yes/no, host OS type, etc.
Describe alternatives you've considered
I could roll my own I guess using wagtail_hooks.py and the wagtail_version template tag.
Additional context
This was inspired by two things recently (aside from the "My IP" case I made above):

I have a QC engineer on one project who checks, amongst other things, the version of Wagtail running. She does so by looking for the "Wagtail upgrade available". If there's no upgrade available, QC fails. There's got to be a better way!
Plone does this; from their control panel:

@nutjob4life It's at the bottom of the settings menu - does that do the job?


Pixel artifacts

Thank you for this great model which performs very quickly. I notice a significant issue that could be related to rgb_mean and img_range values. When using your pretrained model spanx4_ch48.pth, upscaled image almost always contain pixel artifacts that make final picture unusable (see image) :

How can I solve this problem ? Thanks.

When I use their codes, I normalize the image as in their codes but don't do de-normalization, but clip the output to [0, 1]. By doing so, the output looks good to me.