Would it be possible to create a platform release ?

Hi I was just wondering if it is at all possible to have a second candidate that is setup for platformio, as Cmake is messy, complicated and unreliable for noobs like myself who turn to raspberry pi as a simple alternative.
And now that the Pi-Pico is supported via Wiz-IO-Pico on platformio.
Would assume that it would be a trivial exercise for an experienced programmer to convert.
I have all but given up on Cmake as it only works sometimes and often finds the wrong paths.
Thank You

After a fresh clone and following the instructions to the letter, this is the output.
MacBook-Pro:~ gomond$ cd pico-lorawan/
MacBook-Pro:pico-lorawan gomond$ ls
API.md			LICENSE			examples		pico_sdk_import.cmake
CMakeLists.txt		README.md		lib			src
MacBook-Pro:pico-lorawan gomond$ mkdir build
MacBook-Pro:pico-lorawan gomond$ cd build/
MacBook-Pro:build gomond$ cmake .. -DPICO_BOARD=pico
Using PICO_SDK_PATH from environment ('/Users/gomond/pico-sdk')
PICO_SDK_PATH is /Users/gomond/pico-sdk
Defaulting PICO_PLATFORM to rp2040 since not specified.
Defaulting PICO platform compiler to pico_arm_gcc since not specified.
-- Defaulting build type to 'Release' since not specified.
PICO compiler is pico_arm_gcc
-- The C compiler identification is GNU 11.2.1
-- The CXX compiler identification is GNU 11.2.1
-- The ASM compiler identification is GNU
-- Found assembler: /usr/local/bin/arm-none-eabi-gcc
Build type is Release
PICO target board is pico.
Using board configuration from /Users/gomond/pico-sdk/src/boards/include/boards/pico.h
-- Found Python3: /usr/local/Frameworks/Python.framework/Versions/3.9/bin/python3.9 (found version "3.9.13") found components: Interpreter
TinyUSB available at /Users/gomond/pico-sdk/lib/tinyusb/src/portable/raspberrypi/rp2040; enabling build support for USB.
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/gomond/pico-lorawan/build
MacBook-Pro:build gomond$ ls
CMakeCache.txt			CMakeDoxygenDefaults.cmake	Makefile			elf2uf2				generated
CMakeDoxyfile.in		CMakeFiles			cmake_install.cmake		examples			pico-sdk
MacBook-Pro:build gomond$ make
Scanning dependencies of target bs2_default
[  0%] Building ASM object pico-sdk/src/rp2_common/boot_stage2/CMakeFiles/bs2_default.dir/compile_time_choice.S.obj
[  0%] Linking ASM executable bs2_default.elf
[  0%] Built target bs2_default
[  0%] Creating directories for 'ELF2UF2Build'
[  0%] No download step for 'ELF2UF2Build'
[  0%] No update step for 'ELF2UF2Build'
[  1%] No patch step for 'ELF2UF2Build'
[  1%] Performing configure step for 'ELF2UF2Build'
-- The C compiler identification is AppleClang <IP_ADDRESS>00029
-- The CXX compiler identification is AppleClang <IP_ADDRESS>00029
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/gomond/pico-lorawan/build/elf2uf2
[  1%] Performing build step for 'ELF2UF2Build'
[ 50%] Building CXX object CMakeFiles/elf2uf2.dir/main.cpp.o
[100%] Linking CXX executable elf2uf2
[100%] Built target elf2uf2
[  1%] No install step for 'ELF2UF2Build'
[  1%] Completed 'ELF2UF2Build'
[  1%] Built target ELF2UF2Build
[  2%] Generating bs2_default.bin
[  2%] Generating bs2_default_padded_checksummed.S
[  2%] Built target bs2_default_padded_checksummed_asm
Scanning dependencies of target pico_lorawan_default_dev_eui
[  2%] Building C object examples/default_dev_eui/CMakeFiles/pico_lorawan_default_dev_eui.dir/main.c.obj
[  2%] Building C object examples/default_dev_eui/CMakeFiles/pico_lorawan_default_dev_eui.dir///src/lorawan.c.obj
[  2%] Building C object examples/default_dev_eui/CMakeFiles/pico_lorawan_default_dev_eui.dir///lib/LoRaMac-node/src/apps/LoRaMac/common/CayenneLpp.c.obj
[  2%] Building C object examples/default_dev_eui/CMakeFiles/pico_lorawan_default_dev_eui.dir///lib/LoRaMac-node/src/apps/LoRaMac/common/cli.c.obj
[  2%] Building C object examples/default_dev_eui/CMakeFiles/pico_lorawan_default_dev_eui.dir///lib/LoRaMac-node/src/apps/LoRaMac/common/LmHandlerMsgDisplay.c.obj
[  2%] Building C object examples/default_dev_eui/CMakeFiles/pico_lorawan_default_dev_eui.dir///lib/LoRaMac-node/src/apps/LoRaMac/common/NvmDataMgmt.c.obj
[  3%] Building C object examples/default_dev_eui/CMakeFiles/pico_lorawan_default_dev_eui.dir///lib/LoRaMac-node/src/apps/LoRaMac/common/LmHandler/LmHandler.c.obj
[  3%] Building C object examples/default_dev_eui/CMakeFiles/pico_lorawan_default_dev_eui.dir///lib/LoRaMac-node/src/apps/LoRaMac/common/LmHandler/packages/FragDecoder.c.obj
In function 'FragGetParityMatrixRow',
inlined from 'FragDecoderProcess' at /Users/gomond/pico-lorawan/lib/LoRaMac-node/src/apps/LoRaMac/common/LmHandler/packages/FragDecoder.c:360:9:
/Users/gomond/pico-lorawan/lib/LoRaMac-node/src/apps/LoRaMac/common/LmHandler/packages/FragDecoder.c:594:22: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
594 |         matrixRow[i] = 0;
|         ~^
/Users/gomond/pico-lorawan/lib/LoRaMac-node/src/apps/LoRaMac/common/LmHandler/packages/FragDecoder.c: In function 'FragDecoderProcess':
/Users/gomond/pico-lorawan/lib/LoRaMac-node/src/apps/LoRaMac/common/LmHandler/packages/FragDecoder.c:307:13: note: at offset [12, 252] into destination object 'matrixRow' of size 3
307 |     uint8_t matrixRow[(FRAG_MAX_NB >> 3 ) + 1];
|             ^
In function 'FragGetParityMatrixRow',
inlined from 'FragDecoderProcess' at /Users/gomond/pico-lorawan/lib/LoRaMac-node/src/apps/LoRaMac/common/LmHandler/packages/FragDecoder.c:360:9:
/Users/gomond/pico-lorawan/lib/LoRaMac-node/src/apps/LoRaMac/common/LmHandler/packages/FragDecoder.c:594:22: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
594 |         matrixRow[i] = 0;
|         ~^
/Users/gomond/pico-lorawan/lib/LoRaMac-node/src/apps/LoRaMac/common/LmHandler/packages/FragDecoder.c: In function 'FragDecoderProcess':
/Users/gomond/pico-lorawan/lib/LoRaMac-node/src/apps/LoRaMac/common/LmHandler/packages/FragDecoder.c:307:13: note: at offset [13, 253] into destination object 'matrixRow' of size 3
307 |     uint8_t matrixRow[(FRAG_MAX_NB >> 3 ) + 1];
|             ^
In function 'FragGetParityMatrixRow',
inlined from 'FragDecoderProcess' at /Users/gomond/pico-lorawan/lib/LoRaMac-node/src/apps/LoRaMac/common/LmHandler/packages/FragDecoder.c:360:9:
/Users/gomond/pico-lorawan/lib/LoRaMac-node/src/apps/LoRaMac/common/LmHandler/packages/FragDecoder.c:594:22: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
594 |         matrixRow[i] = 0;
|         ~^
/Users/gomond/pico-lorawan/lib/LoRaMac-node/src/apps/LoRaMac/common/LmHandler/packages/FragDecoder.c: In function 'FragDecoderProcess':
/Users/gomond/pico-lorawan/lib/LoRaMac-node/src/apps/LoRaMac/common/LmHandler/packages/FragDecoder.c:307:13: note: at offset [14, 254] into destination object 'matrixRow' of size 3
307 |     uint8_t matrixRow[(FRAG_MAX_NB >> 3 ) + 1];
|             ^
[  3%] Building C object examples/default_dev_eui/CMakeFiles/pico_lorawan_default_dev_eui.dir///lib/LoRaMac-node/src/apps/LoRaMac/common/LmHandler/packages/LmhpClockSync.c.obj
[  3%] Building C object examples/default_dev_eui/CMakeFiles/pico_lorawan_default_dev_eui.dir///lib/LoRaMac-node/src/apps/LoRaMac/common/LmHandler/packages/LmhpCompliance.c.obj
[  3%] Building C object examples/default_dev_eui/CMakeFiles/pico_lorawan_default_dev_eui.dir///lib/LoRaMac-node/src/apps/LoRaMac/common/LmHandler/packages/LmhpFragmentation.c.obj
[  3%] Building C object examples/default_dev_eui/CMakeFiles/pico_lorawan_default_dev_eui.dir///lib/LoRaMac-node/src/apps/LoRaMac/common/LmHandler/packages/LmhpRemoteMcastSetup.c.obj
[  4%] Building C object examples/default_dev_eui/CMakeFiles/pico_lorawan_default_dev_eui.dir///lib/LoRaMac-node/src/boards/mcu/utilities.c.obj
[  4%] Building C object examples/default_dev_eui/CMakeFiles/pico_lorawan_default_dev_eui.dir///lib/LoRaMac-node/src/mac/region/Region.c.obj
[  4%] Building C object examples/default_dev_eui/CMakeFiles/pico_lorawan_default_dev_eui.dir///lib/LoRaMac-node/src/mac/region/RegionAS923.c.obj
[  4%] Building C object examples/default_dev_eui/CMakeFiles/pico_lorawan_default_dev_eui.dir///lib/LoRaMac-node/src/mac/region/RegionAU915.c.obj
/Users/gomond/pico-lorawan/lib/LoRaMac-node/src/mac/region/RegionAU915.c: In function 'RegionAU915GetPhyParam':
/Users/gomond/pico-lorawan/lib/LoRaMac-node/src/mac/region/RegionAU915.c:244:13: internal compiler error: Illegal instruction: 4
244 |             phyParam.fValue = AU915_DEFAULT_MAX_EIRP;
|             ^~
Please submit a full bug report,
with preprocessed source if appropriate.
See https://bugs.linaro.org/ for instructions.
make[2]: *** [examples/default_dev_eui/CMakeFiles/pico_lorawan_default_dev_eui.dir///lib/LoRaMac-node/src/mac/region/RegionAU915.c.obj] Error 1
make[1]: *** [examples/default_dev_eui/CMakeFiles/pico_lorawan_default_dev_eui.dir/all] Error 2
make: *** [all] Error 2
MacBook-Pro:build gomond$

Hi @gomond,

Hi I was just wondering if it is at all possible to have a second candidate that is setup for platformio

There are no plans to support this, I would prefer to only support Cmake as that is what the official Pico SDK uses and supports.

/Users/gomond/pico-lorawan/lib/LoRaMac-node/src/mac/region/RegionAU915.c:244:13: internal compiler error: Illegal instruction: 4

Could you please provide more details on the OS and gcc version you are using so I can try to reproduce this locally.
For example, for gcc version:
% arm-none-eabi-gcc --version
arm-none-eabi-gcc (GNU Arm Embedded Toolchain 10-2020-q4-major) 10.2.1 20201103 (release)
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


Pin Zarr for tests

Tests just failed today due to a surprise break from Zarr release yesterday - some issue with decoding Blosc for our custom backend configuration stuff
Shouldn't effect actual typical usage of the backend feature aside from Blosc I assume, since none of the other compressor tests complain
Filed issue with hdmf-zarr so team knows about it, and if it's an issue with the way they specify the read from the IO level (I doubt it, but never know)
Downpinning in tests temporarily to fix the CI, will check over time if resolved automagically

I think that we should put upper bounds everywhere so people who install our package don't get the version that does not work, right?

I think that we should put upper bounds everywhere so people who install our package don't get the version that does not work, right?

Thing is, most of the package works, just perhaps not this one particular case that showed up in one corner of our tests - so I'd wait to see if this gets fixed before totally restricting the latest upper bound in general
Maybe circle back in a week?

Filed an issue on HDMF for the dev tests

Waiting a week seems fine to me.

You still want this merge to have the tests working?

You still want this merge to have the tests working?

Yes please


Choose implementation for WETH predeploy on L2

Optimistic Ethereum's L2 does not have a native token like Ethereum's Ether. Rather we have an WETH-like OMV_ETH contract pre-deployed in the genesis state:
We need to finalize on the implementation we'll build on prior to our public mainnet launch.
Any ETH deposited to L2 will be held in this contract.
Wanted:

Secure and battle tested
Supports EIP-2612 (ie. using permit() to set approvals via signature)
Permissive licensing

Questions:

Should we use WETH10 (with effectively unlimited flashminting?)

Candidate Implementations:

UniswapV2ERC20.sol
Dai (requires modifying permit slightly to match spec)

a) Yes flashminting, allows arbing assets that become depegged fast which is important when liquidity is low in the early days.
b) Can we evaluate using an ERC777 WETH? It's got a standard for receiving contracts that's formalized (unlike WETH10)

To bring the twitter discussion over here: me and @TransmissionsDev have been discussing trying to encourage more usage of ERC777 on Optimism. I've already started work on PR to make the Optimism WETH contract support ERC777.
ERC777 has a number of benefits over ERC20, including:

Being backwards compatible with ERC20, so existing applications & apps don't need to update
Allows apps to use "receive hooks", removing the need for users to "approve" every application. This has a number of benefits:

Better UX. Users aren't prompted with confusing "approve" transactions
Better security. Users aren't approving access to infinite funds, meaning a contract with an exploit could potentially drain all funds
Cheaper transfers, since allowances don't need to be updated. Admittedly, not as much of an issue on L2, but still always good to write efficient code

The biggest drawback of ERC777 is that transfer hooks open up the potential for re-entrancy attacks, such as the attack on LendFme.
However, transferring ETH has the same issue (see DAO hack), and mitigating re-entrancy is a well understood security pattern at this point.
The move to L2 is a once-in-a-lifetime to start with a blank slate, fixing some of the UX problems on L1. I'd really encourage the Optimism team & community to consider support for this. 🙂🎰

Worth noting that EIP-2612 (while great) is not a replacement for ERC-777 functionality. EIP-2612 does not solve problems for contract-to-contract calls. Given the complexity growth of DeFi apps, it's reasonable to expect that contract-to-contract interactions will eventually outnumber EOA-to-contract interactions

Is 1820 supported?

@carlfarterson I've created a PR (#212) to add 1820 as a precompile.
Regardless of whether WETH is 777 or not, that PR is important to ensuring that at least other projects can use 777.

I think flashminting is pretty reasonable as long as there's a cap set to make it hard to overflow a value with. With the cap in place, it seems like any vulnerability that could be exploited by flash minting, could also be exploited using flash loans or by a rich attacker. The potential benefits around capital efficiency and access to liquidity for users and dapps on Optimism are worth it IMHO.
I'm a fan of ERC777 in general but worry about its security implications more than flashminting. On top of reentrancy, there are DoS attacks where recipients revert the transaction preventing some upstream or downstream action. I do see the benefits though and the security issues are easy to avoid if developers are conscious of them. Dapps that aren't built for ERC777 will very likely have issues though.

flash minting = arbitrage
arbitrage = more efficient markets

@cwhinfrey AFAIK the "DoS" issue you described would only come from an app that needs to successfully transfer a token to an arbitrary address to succeed.
Given that most apps use a pull/withdraw pattern instead of push/send, seems like this issue would be somewhat rare. Do you know of any major apps that would be vulnerable to this?

IMO, ERC777 should be the baseline of all tokens on L2.
RUs are a chance to reimagine things. Let's not get stuck with ERC20's abysmal UX forever.

Reentrancy concerns seem overblown. As they are a known issues anyhow, they can be addressed by SC devs

Keep scope as tight as you can.
WETH10 is bloated with features of questionable value. Use WETH9.
If you need permit, add permit to WETH9 and use that.

Thank you everyone for chiming in here, we really appreciate it. 🙏  Still weighing the considerations, will keep you posted.

@cwhinfrey AFAIK the "DoS" issue you described would only come from an app that needs to successfully transfer a token to an arbitrary address to succeed.
Given that most apps use a pull/withdraw pattern instead of push/send, seems like this issue would be somewhat rare. Do you know of any major apps that would be vulnerable to this?

It came up a few times in this audit, but ERC777 was still relatively new. https://blog.openzeppelin.com/augur-core-v2-audit/
See:
[C05] Unfillable orders can be placed to constrain or halt markets
[C07] All dispute bonds can be held hostage or permanently frozen when a Market is in a forking universe

But yeah, reentrancy and this aren't the end of the world. ERC777 has nearly the same security implications as dealing with ETH directly at the end of the day.

I'll just update on our thinking here:
IMO using ERC777 is just too much of a leap from what developers expect. It's likely to break existing code, which contradicts our design goals.
To avoid surprises, we're leaning towards just using a well tested ERC20 + EIP 2612 impl without flashminting.
We're also considering

returning to be as close as possible to the EVM, without a built-in WETH
Having a combined ETH/WETH, as discussed here

2. here

Combined WETH/ETH sounds super cool.
Can you explain why no flashminting tho?
I think it would be very helpful to arb prices especially when liquidity is fragmented and low at the start.

While my vote is still for 777, I understand the hesitance around it.
If the decision is made to not use 777, I'd strongly encourage use of WETH10, with or without flash mints. Features such as ERC677 transferAndCall can have significant improvements for cost, UX and security.


Articles in explorer should be shown according to their order number

Currently, they are displayed in DESCENDING order, and it should be ASCENDING order.

Fixed


[Event Request] Page 11000007 Payment History List - OnPrintDocketBeforeRunReport

Please add a new publisher event to the function PrintDocket - OnAction() as follows:
PrintDocket - OnAction()
.
.
PaymHist := Rec;
PaymHist.SETRANGE("Export Protocol","Export Protocol");
PaymHist.SETRANGE("Our Bank","Our Bank");
PaymHist.SETRANGE("Print Docket",TRUE);

//>> Start New Publisher Call
OnPrintDocketBeforeRunReport(Rec, SentProtocol, PaymHist);
//<< End New Publisher Call

REPORT.RUNMODAL(SentProtocol."Docket ID",TRUE,TRUE,PaymHist);

CurrPage.UPDATE;
.
.

New Publisher Event:
LOCAL [IntegrationEvent] OnPrintDocketBeforeRunReport(Rec : Record "Payment History";VAR SentProtocol : Record "Export Protocol";VAR PaymHist : Record "Payment History")

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. We will update this issue with information about availability.


Fixes #367 - appYamls vs Services

Remove RunConfiguration.appYamls and use RunConfiguration.services instead.

Codecov Report

Merging #368 into master will decrease coverage by 0.08%.
The diff coverage is 75%.

@@            Coverage Diff             @@
##           master     #368      +/-   ##
==========================================
- Coverage   62.61%   62.52%   -0.09%
==========================================
  Files          65       65
  Lines        1701     1697       -4
  Branches      254      254
==========================================
- Hits         1065     1061       -4
  Misses        530      530
  Partials      106      106

Impacted Files
Coverage Δ

...ppengine/cloudsdk/CloudSdkAppEngineDevServer1.java
70.29% <ø> (-0.3%)
:arrow_down:

...pengine/api/devserver/DefaultRunConfiguration.java
100% <ø> (ø)
:arrow_up:

...appengine/cloudsdk/CloudSdkAppEngineDevServer.java
67.24% <75%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 76b51a8...372fc52. Read the comment docs.


[Proposal] Replacing eslint-plugin-react with eslint-react

eslint-react is a modern version of eslint-plugin-react
https://eslint-react.xyz/docs/faq
https://github.com/Rel1cx/eslint-react/issues/85
I will send a PR shortly if @AndreaPontrandolfo likes this idea.

Hey @Talent30 !
Sheriff is already using @eslint-react/eslint-plugin, but just for a couple of rules that i find useful and are not covered by eslint-plugin-react.
That being said, at this stage i don't consider @eslint-react/eslint-plugin a replacement for eslint-plugin-react. It's a very interesting project but it is still young and is still growing, it will take a while to reach the level of maturity of a project like eslint-plugin-react.
In summary, Sheriff is already employing @eslint-react/eslint-plugin so no PR is needed, thank you anyway for the report.
I'm gonna close this now.

@AndreaPontrandolfo, maybe move this to a discussion? I think that at some point, it'd be nice to swap over to @eslint-react, but stick with both for now, given that it's a maturity/compatibility issue. Just my 2¢.

So, one of the issues with this, is that @eslint-react/eslint-plugin isn't a drop-in replacement for eslint-plugin-react, as outlined here.
So transitioning wouldn't be trivial, and it's questionable whether it would be an objective improvement or not.

So, one of the issues with this, is that @eslint-react/eslint-plugin isn't a drop-in replacement for eslint-plugin-react, as outlined here.  So transitioning wouldn't be trivial.

Yeah, so maybe wait for v30? (Maybe make majors that're multiples of majorly major? I personally like RedwoodJS's Epochs concept.)

It's questionable whether it would be an objective improvement or not.

Maybe not yet, but at some point it will likely mature, and then it's a more flexible, feature-complete, and more maintained plugin.

Anyway, I still vote this becomes an "ideas" discussion.


vo registry complient fixes

Coverage increased (+0.004%) to 77.815% when pulling 803ce005a4d03b7372913dab05925070047dfc9f on tap-registry into 94996f7b50f0e973bcf6c1b1d5f533afb0a993d7 on master.


Error when trying create an agent

Hi,
From the EvilVM directory, I run the "build.rb" script to generate a new agent:
./build.rb -n -i $server -p 443 -E -o agent.exe
In the background the "server.rb" is running but I still get an error:
root@ubuntu-s:~/EvilVM# ./build.rb -n -i $server -p 443 -E -o agent.exe
IP: -p
/tmp/object20210413-1712-1abst77: file not recognized: File truncated
Traceback (most recent call last):
        3: from ./build.rb:421:in `<main>'
        2: from ./build.rb:161:in `compile'
        1: from ./build.rb:150:in `link'
./build.rb:150:in `read': No such file or directory @ rb_sysopen - /tmp/binary20210413-1712-80hf4l (Errno::ENOENT)

Please your help
Ubuntu 18
ruby 2.7

Is there any chance that you don't have nasm installed on your system?  When I remove nasm from my path, I get the same error message, so perhaps that is it.  If which nasm doesn't turn anything up, though, let me know, and we can perhaps add some trace statements and see what else might be going on.

OK, it's been a few days -- I'm going to assume that a missing nasm was the issue, or that you've resolved the issue somehow.  If not, feel free to reply again, and I'll be more than happy to help debug your situation.  Thanks!


Easy way to delete range/set of IPs

Which package/s are you suggesting this feature for?
Dashboard
Is your feature request related to a problem? Please describe
Dashboard allows you to add a range of IPs. Would it be possible to allow the selection of a set of IPs for deletion instead of going through them one by one?
I understand that this task may not be an everyday one, but making it easier will help users appreciate the flawlessness when it comes time to use it.
Describe the solution you'd like
I think adding a standard checkbox for every IP and allowing one to select/deselect all checkboxes would be a nice UX  improvement.

Work Completed:
Remove the delete button and add delete all action

Work Completed:
Some code enhancement as per pull request comments

Work Completed:
Create removeFarmIps function to delete all ips in one action

Work in Progress (WIP):
Working on PR comment

Work in Progress (WIP):
faced some issues while build , still investigating ..

Work Completed:
Add removeFarmIps function in tf grid

Work Completed:

Document removeFarmIps methods.

https://github.com/threefoldtech/tfgrid-sdk-ts/issues/2610#issuecomment-2111933541

Work in Progress (WIP):
investigating..

Work Completed:
Fix farms public IPs table selection by removing the current data-table and passing props to list-table component.

verified on devnet
version : 85a6fa4
a standard check box is now added to the ip table to make deleting a contract or multiple contracts easier


Add script/console to help debug

script/console will open a PRY session with Octokit debug output enabled and pseudo-production values as the default to help diagnose failed builds.

Thank you!
@jekyllbot: merge +dev

@jekyllbot: merge +dev


Certain required fields can get omitted from the markdown files

Describe the issue that you're seeing. Any Loom videos or screenshots usually help a lot!
I'm not sure if this is a bug or a design choice, but it's already caused issues with a Tina site for me. My site generator expects certain fields to be present, so when a field gets omitted, it causes errors.
Tina will omit a boolean field from the output file unless it's explicitly set to "true" or "false" by the user. If you create a new file and don't interact with the field, leaving it in the default "false" state, then it's omitted. Using required: true has no effect.
I've also observed the same behavior with string fields that use ui: {component: "textarea"}. This causes the required: true to be ignored and, if the textarea is left blank, then Tina omits the field when saving the file.
It's possible I don't understand the intended semantics of required: true, but I definitely want to be able to specify that certain fields will always be written to the markdown files, even with a just a default value.
Reproduction
https://github.com/johnridesabike/tina-bug-report
Steps to reproduce

Create a new markdown page
Add text to the body but ignore the other required fields.
Save the file
The required fields are not in the output.

Example: https://github.com/johnridesabike/tina-bug-report/blob/3c3af99f540b7aa802125ff53b5763a72e6cef4a/content/markdown/missing_required_values.md
Relevant configuration: https://github.com/johnridesabike/tina-bug-report/blob/3c3af99f540b7aa802125ff53b5763a72e6cef4a/.tina/config.js#L39
System Info
System:
    OS: macOS 13.2
    CPU: (6) x64 Intel(R) Core(TM) i5-8500B CPU @ 3.00GHz
    Memory: 90.02 MB / 8.00 GB
    Shell: 5.8.1 - /bin/zsh
  Binaries:
    Node: 19.3.0 - /usr/local/bin/node
    Yarn: 1.22.19 - ~/code/tina-bug-report/my-tina-app/node_modules/.bin/yarn
    npm: 9.4.1 - /usr/local/bin/npm
  Browsers:
    Chrome: 110.0.5481.77
    Firefox: 109.0.1
    Safari: 16.3
  npmPackages:
    @tinacms/cli: ^1.1.0 => 1.1.0
    tinacms: ^1.2.0 => 1.2.0

Validations

[X] Follow our Code of Conduct
[X] Read the docs.
[X] Check that there isn't already an issue that reports the same bug.

I noticed that this behavior seems to have improved in recent versions. Now, Tina will block you from saving if you leave a boolean field in its default "false" state (previously, it would just save and omit the field in the output). This is better, but it's still slightly confusing if you want to set it to false, since you now have to switch it to true and then back to false.


[QOL] Make objects bucket a singleton

It's tedious to pass around the objects bucket through the fuzzer to the fengine module. Refactoring the objects bucket from a simple dictionary into its singleton class will allow us to contain all objects-bucket-related functions in a single file.
Deliverable
Create a objects_bucket.py in the which has a singleton for the objects bucket. Reference this for all calls to the objects bucket except for the first call in the fuzzer initialization step.
Extension
Add not only IDs that are seen in the responses, but also other object fields

Wont-do
To keep the objects bucket thread-safe, we will continue with the dependency injection method instead. However, we should refactor the objects bucket to be a class with function invocations on the bucket.


Potentially switch from log to tracing

Is your feature request related to a problem? Please describe.
tracing is becoming more common in the async ecosystem, while still providing functionality to support emitting log messages with the log feature. Having sqlx switch to tracing enables the ability to group events emitted by sqlx with tracing::spans
Describe the solution you'd like
Switch the internal logging over to tracing (I would be more than willing to follow through on this work)
Describe alternatives you've considered
Just keep using log instead of tracing
Additional context
This has been mentioned in passing in a couple of issues (Namely #1896 and #942): however, those mentioned tracing as a part of some other initiative, while this issue just focuses on the feasibility of swapping log for tracing
log::LevelFilter is used publicly. This would likely be swapped for an enum exposed by sqlx to avoid the public dependency on another crate. This means that this would be a breaking change

This is a switch we've been wanting to do for a while. I'd be glad to land it as part of 0.7.0 but I don't have time to work on it personally.

I'll start working on it, but no promises that it'll be finished in time for v0.7
Feel free to ping me for progress updates 👍

@LovecraftianHorror Do you think we can finish this faster if two of us work on it together? I would love to help expedite this change.

@taylor1791 I can throw up a draft PR with whatever I end up getting through this weekend. I would just divvy up the crates with you, but there will be some design changes that would be best to keep consistent since tracing handles some things slightly differently than log
I would definitely appreciate a second set of eyes on things, and I'll definitely ping you if I end up stalling out on the work (life has been busy lately)

Initial work done in #2185, to track the work of adding spans everywhere I suppose we could either keep this issue open or create a new one. Creating a new one is more likely to get the attention of someone driving by looking for some open-source work to do.

Just wondering, would it be possible to make a release without this issue being solved? There are lots of great improvements on main already that are useful without this issue.

I don't think this issue is blocking release


Design feedback, iteration 1

Small stuff, just to make sure it doesn't get lost

[ ] "Job Posting Total" stuff doesn't interact well with other elements
for example - screenshot
[x] Can we have less whitespace between elements? It looks nice, but makes the page a lot longer.
[x] Headers that wrap have too much space between lines, see "What location will this position mostly be working form?"
[ ] Maybe put a back button on the top of the second page (in addition to the one at the bottom)? Mostly because if I'm up there I forget to scroll & click the button and use the browser back instead.
[ ] Colors for the skill selection rows differ from the existing colors in the project, making it start to look busy.
[x] Extra comma in "Please select**,** N/A, Preferred or Requ..."
[ ] Instead of N/A to (eventually) remove a skill, add minus buttons to the end of the row, also allows for removal of manually entered skills (see
rough mockup)
[ ] Possibly group non-interactive fields at the end or reconsider how much we need them (make it easier to demonstrate value)
[ ] Responsiveness in general

Working off of these designs: https://drive.google.com/drive/u/0/folders/0B32ewVuRPhuLSnBQdk5rblhPOGc


A thought for Discussion: Rename Shell to AppShell?

With Shell.xaml now at the heart of the Hamburger Menu (as the root visual) we'll no doubt see more reference code like Views.Shell.SetBusyIndicator() (as used to show the busy ring on Settings page). I have just implemented error and status notifications (in preference to popup dialogs, which I intend to share this with a pull request) so I have references like Views.Shell.NotifyErrorMessage("error message to show") and Views.Shell.NotifyStatusMessage("status message to show").
This is obviously a matter of opinion and getting used to, but the reference "Views.Shell" just looks a bit far-removed. Compare this with "Views.AppShell" and feels closer to home; we're referring to the app's own shell class.
If this makes sense to most of you, this is to request for renaming of Shell as AppShell. Any thoughts?

We've already been talking about this. We are thinking of the following, possible new controls:

Template10.Controls.Shell

Property Shell.Modals : List where T: IModal
Method Shell.ShowModal(params[]) where T: IModal
Method Shell.HideModal() where T: IModal
Template10.Controls.IModal
Template10.Controls.BusyModal : IModal
Template10.Controls.LoginModal (potential) : IModal
Template10.Controls.ErrorsModal (potential) : IModal

Template10.Controls.HamburgerShell : Shell

HamburgerShell.HeaderContent
HamburgerShell.PrimaryCommands
HamburgerShell.SecondaryCommands

Template10.Controls.TabShell : Shell

TabShell.HeaderContent
TabShell.Commands

The idea is, we are building some nice functionality into Shell that is very valuable with or without the Hamburger. Also, we want to build a pivot interface because the request for it is growing.
As far as calling it AppShell, there's nothing stopping us from doing that. Though I like to name things with as simple names as possible. I am open to the idea and have read your points.
We're also considering making the AppShell (as you call it) a property off Bootstrapper with the assumption that an app would only use one (primary) shell and you could access it, then, with BootStrapper.Current.Shell or something like that. Again, just starting to consider that one.

Obviously there is much more thought to the naming idea than my rather simplistic one. I am quite content that the need for naming is in existence already which means hopefully more logical names will evolve and arrived at in a wider context Jerry has just outlined.


Do we have Lazer pointer or colored pens in presentation?

Describe the solution you'd like
I wonder do we have Lazer pointer or colored pens when in the presentation?
It will be better if we can write sth with either Lazer pointer or colored pens on the page during the presentation.
Thanks.

Think we could embed some libraries that does that. Any recommendations?

Hi,
Found some example in reveal.js
https://github.com/caiofcm/plugin-revealjs-mouse-pointer
https://github.com/djsutherland/reveal.js-notes-pointer

I am working on it.

https://twitter.com/Slidevjs/status/1437261881928867841

Hey was looking for this exact same feature,
Can I suggest one improvement on top of it? the colored pen works well, but drawing stays on your presentation.
a laser pointer that fades away after 1 or 2 seconds would be really cool. laser pointer helps to just highlight the things like in an architecture diagram. with pen we have to press Ctrl-z after drawing every highlight.

No plans for this one? I can work on it if it's easy I can make a PR if you point me to right direction


src/common should be installed inside package

When running pip install . from the latest main branch, I end up with common, spdx and the spdx_tools-0.7.2.dev397+gd9a2e01.dist-info directories inside my venv/lib/python3.10/site-packages directory.
This does not seem right, as common should rather be part of the spdx distribution itself to avoid confusion.

Good point. We will look into that.

Hi @stefan6419846, thanks for spotting this! We had to introduce another top package to solve this issue as we want to be able to add more subpackages (see #595). So unfortunately you'll need to adapt your imports a bit but feel free to give us feedback if this works out for you and if the installation now works as expected.

No worries - I have been aware that the current main branch is still work-in-progress and I cannot always expect it to not generate any backwards-incompatible changes. Nevertheless, the actual issue might have led to unexpected side-effects as common as a package name is just too common (no pun intended) - I have verified that the new package structure indeed fixes this.


Split Hosting Extensions in separate Package

Extract BackgroundServiceBase and HostedServiceBase in Thor.Hosting.Extensions in order to be used in AspNetCore or in GenericHost


add remove(Above|Below)(Percentile|Value)

I try to implement functions on demand.
These functions have been demanded.

removeBelowValue and removeAboveValue are now included.
https://github.com/dgryski/carbonapi/pull/95


18_simple_demo_tutorial: Add Windows instructions

🦟 Bug fix
Fixes https://github.com/gazebosim/gazebo_test_cases/issues/1473
Summary
Add Windows instructions to the tutorial.
I've tested it to work with these instructions.
Checklist

[x] Signed all commits for DCO
[ ] Added tests
[x] Updated documentation (as needed)
[ ] Updated migration guide (as needed)
[ ] Consider updating Python bindings (if the library has them)
[ ] codecheck passed (See contributing)
[ ] All tests passed (See test coverage)
[ ] While waiting for a review on your PR, please help review another open pull request to support the maintainers

Note to maintainers: Remember to use Squash-Merge and edit the commit message to match the pull request summary while retaining Signed-off-by messages.

@osrf-jenkins run tests please!


Feature request: Add option to ask for help with a diagnostic.

https://github.com/piersolenski/wtf.nvim is a plugin to ask for explanation for your diagnostic message.
It would be a nice addition to get this inside of neovim with Cody.
So the gist of this would be fetch the diagnostic, give cody some context (maybe with some treesitter help?) and let it explain what I'm doing wrong.
If this is not something in scope of this project feel free to close this issue.
If it is inside of the scope I can try to implement this.

For anyone interested, this works in my personal config:
keymap("n", "gw", utils.search_diagnostics_cody, opts)
keymap("v", "gw", utils.search_diagnostics_cody, opts)

function M:search_diagnostics_cody()
	local start_line, end_line
	local mode = string.lower(vim.fn.mode())
	if mode == "v" then
		start_line = vim.fn.line("v")
		end_line = vim.fn.line(".")
		if start_line > end_line then
			start_line, end_line = end_line, start_line
		end
	else
		start_line = vim.fn.line(".")
		end_line = start_line
	end

	local bufnr = vim.api.nvim_get_current_buf()
	local all_diagnostics = vim.diagnostic.get(bufnr)
	local diagnostics = {}
	for _, diagnostic in ipairs(all_diagnostics) do
		if diagnostic.lnum + 1 >= start_line and diagnostic.lnum < end_line then
			table.insert(diagnostics, diagnostic)
		end
	end

	if #diagnostics == 0 then
		vim.notify("No diagnostics", vim.log.levels.WARN)
		return
	end

	local index = 1
	if #diagnostics > 1 then
		local inputList = { "Which diagnostic do you want explained?:" }
		for i, diagnostic in ipairs(diagnostics) do
			table.insert(inputList, string.format("%d. %s", i, diagnostic.message))
		end
		index = vim.fn.inputlist(inputList)
		if index <= 0 or index > #diagnostics then
			vim.notify("No diagnostic selected", vim.log.levels.INFO)
			return
		end
	end

	local clean_message = diagnostics[index].message:gsub("[A-Za-z0-9:/\\._%-]+[.][A-Za-z0-9]+", "")
	clean_message = clean_message:gsub("[A-Za-z0-9:/\\._%-]+[/\\][A-Za-z0-9:/\\._%-]+[.][A-Za-z0-9]+", "")

	vim.cmd(
		":"
			.. start_line
			.. ","
			.. end_line
			.. " CodyAsk I got: "
			.. clean_message
			.. " from "
			.. diagnostics[index].source
	)
end

Oh, this is a great idea! let me see if I can get something cleaned up based on this and in the next release (I'm currently rewriting some of the chat stuff, so this will have to wait a little bit)

For other people reading.
This is the prompt I am using following the wtf plugin that uses openai:
  local msg = "You are an expert coder and helpful assistant who can help debug code diagnostics, such as warning and error messages."
  .. "When appropriate, give solutions with code snippets as fenced codeblocks with a language identifier to enable syntax highlighting.\n"
  .. "This is a the diagnostic message for the code:\n"
  .. clean_message

Not sure what @arnevm123 uses.


MSID: 100009 Version: 1 DOI: 10.1101/2<IP_ADDRESS>1933

MSID: 100009
Version: 1
Preprint DOI: https://doi.org/10.1101/2<IP_ADDRESS>1933
Step 1. Awaiting reviews
Editorial to post reviews via hypothesis
Useful links:

DocMap: https://data-hub-api.elifesciences.org/enhanced-preprints/docmaps/v2/by-publisher/elife/get-by-manuscript-id?manuscript_id=100009
New model tracking: https://docs.google.com/spreadsheets/d/1_fHaoOy7hjyocptKtVJRijeNpUY4hBS7Ck_aVmx6ZJk/
Reviews on sciety: https://sciety.org/articles/activity/10.1101/2<IP_ADDRESS>1933

For trouble shooting (e.g. no Docmaps available):

DocMap issue addressing: https://miro.com/app/board/uXjVNCwK6EI=/
Explore DataHub DocMaps API: https://lookerstudio.google.com/reporting/4c2f0368-babb-4beb-b5b3-497e7e7b0f08/page/ejphD
Unmatched submissions and preprints: https://lookerstudio.google.com/u/0/reporting/9f86204f-3bf7-477c-9b18-5c5ef141bf69/page/p_gxi57ha93c
Unmatched manuscripts spreadsheet: https://docs.google.com/spreadsheets/d/15QcK8w-ssB7109RQEDtFpJPZ0J5HTGxoHa_2TtpMBbg/edit#gid=1336081641

Step 2. Preview reviewed preprint
Production QC content ahead of publication
Instructions:

QC preview: https://prod--epp.elifesciences.org/previews/100009v1
Update ticket with any problems (add blocked label)
When QC OK, add QC OK label to ticket and add publication date and time to https://docs.google.com/spreadsheets/d/1amAlKvdLcaDp5W8Z8g77NmkwbMF5n_u89ArSqPMO8jg
Move card to next column
(At end of the day post link in #enhanced-preprint and ask for PDF to be generated)

Useful links:

Preprint DOI :  https://doi.org/10.1101/2<IP_ADDRESS>1933
Confirm reviews returned by EPP: https://prod--epp.elifesciences.org/api/reviewed-preprints/100009/v1/reviews
To update the MECA path in the docmap: https://docs.google.com/spreadsheets/d/1mctCQuNFBjSn97Lihy7_vBO6z7-N-oqyLv4clyi6zHg

Step 3: Awaiting search reindex
This step adds the reviewed preprint to the homepage: https://elifesciences.org
The search reindex is triggered once an hour. We need the reviewed preprint to be indexed as the search application serves the journal homepage.
Useful links:

Jenkins pipeline to reindex search can be triggered sooner or monitored here: https://alfred.elifesciences.org/job/process/job/process-reindex-reviewed-preprints/

Step 4: Published! PDF requested
Waiting for PDF to be generated
Useful links:

PDF tracking: https://docs.google.com/spreadsheets/d/106_XeDjmuBae7gexOTNzg60lapeqjl2aRn9DzupGyS8/

Step 5: Introduce PDF to data folder and git repo
Upload PDF to relevent folder in git repo https://github.com/elifesciences/enhanced-preprints-data/
Step 6: Done!

[ ] Kettle is on!

warn: [journal-ref-text-content] This journal reference (id c10) has untagged textual content - ), Article 1. . Is it tagged correctly?
warn: [journal-ref-text-content] This journal reference (id c12) has untagged textual content - ), Article 6. . Is it tagged correctly?
warn: [journal-ref-text-content] This journal reference (id c14) has untagged textual content - ), Article p5. . Is it tagged correctly?
error: [journal-ref-article-title] This journal reference (id c20) has no article-title element.


computing sources in standard volume space?

Hello, does anyone know if there is a way to compute sources in standard space (MNI, or any other template) on a volumetric grid? i would like to compute raw, continuous sources (no baseline) over a sleep study.
i have simultaneous EEG-FMRI data, and i would like to compute my sources and align the source data with the FMRI data, so i can compute simultaneous EEG-FMRI correlations in each voxel separately.
also, due to the fact that i am localizing rather long recordings (2+hrs) i have quite a bit of data (~2 million time points at 250Hz), obviously if i localize this to a source grid of 3x3x3mm or similar it will explode my memory...i'm wondering if there is a way to do this in a more memory efficient way? or, i can just compute sources over a lower resolution grid (8x8x8mm) and then later upsample it to FMRI resolution.
thanks,
Russell

does anyone know if there is a way to compute sources in standard space (MNI, or any other template) on a volumetric grid?

fsaverage is in MNI space, and you can set up a volume grid source space for it with setup_volume_source_space

obviously if i localize this to a source grid of 3x3x3mm or similar it will explode my memory...i'm wondering if there is a way to do this in a more memory efficient way?

Probably but it depends on what you want to do / localize. For example source power band spectra can be done in a memory efficient manner:
http://mne-tools.github.io/dev/auto_examples/datasets/plot_opm_rest_data.html#sphx-glr-auto-examples-datasets-plot-opm-rest-data-py

so i can compute simultaneous EEG-FMRI correlations in each voxel separately.

Just realized you said what you wanted to do...
I would use apply_inverse_epochs with generator=True, and for each epoch compute the correlation. Correlatinos can be computed in a piecewise manner, this is what we use in raw data covariance matrix calculation:
https://github.com/mne-tools/mne-python/blob/master/mne/cov.py#L465-L476
If you don't want to chunk in time, technically you could "chunk in space" by computing the inverse just in individual labels at a time (label argument to apply_inverse_raw) to limit the set of sources. Then you iterate over sources. Not sure how it would work with a volume source space but it should be possible to hack it to work.

yeah, i am basically trying to get bandpass filtered, rectified time series in the same sampling rate as FMRI (TR=2.2s)...for 1-100Hz in source space.
i'll try fsaverage with that link you sent in your first post and let you know how that goes.
chunking in time is ok. i was probably going to do that anyways (to average out the influence of outlying time points on the correlation) and then average the correlation across all 'chunks'.
thanks

Closing as this seems solved, feel free to reopen if not


Implement organizations

Ready for review.

One minor comment, otherwise looks good!

@justinvdm Ready for another review.

@justinvdm Ready for review again.

:+1:


runtime: GC causes latency spikes with single processor

(This may be the same as #14812, but the discussion for that issue has veered towards sync.Pool and there are many processors available in the traces people have posted to that issue.)
What version of Go are you using (go version)?
go1.7.4  Testing with the latest master branch does not make a difference for my real application.
What operating system and processor architecture are you using (go env)?
Linux on an x64 single-core 1.4GHz Atom processor
What did you do?
It does not run on the playground (takes too long and cannot write trace file), but here's a minimal test case that reproduces the issue
https://play.golang.org/p/aJmmumzKpD
(original example that was slightly more complicated: https://play.golang.org/p/_tQTfBR33g)
Specifically, that reproduces the approximate amount of memory, GC frequency, GC irregularity that I see in our application.  Concretely, this reproduction test case has GCs that collect 50-100MB but also sometimes cause application delays with long tails on the order of 100ms.
To mimic the single core processor from my laptop, I was running the test with CPU affinity like this: "GODEBUG=gcpacertrace=1,gctrace=1 taskset -c 0 ./main"
What did you expect to see?
The main goroutine is driven off of a 100ms tick.  Ideally that tick would arrive every 100ms and then the main goroutine would run immediately after the tick then and be done.
What did you see instead?
Sometimes the tick gets delayed and sometimes the main goroutine gets delayed even after the tick fires.  Measuring the deviation away from the expected 100ms interval, I see a latency distribution like this:
  0.0% -64.359857ms
  1.0% -41.943823ms
  5.0% -19.403262ms
  50.0% 121.887µs
  95.0% 20.226485ms
  99.0% 40.27582ms
  100.0% 75.602625ms

The mean is as expected, but the 75ms deviation is much higher than we want to see.  (The negative values are the ticker catching up after previous delays).
The gctrace output before the 75ms delay looks like this:
gc 79 @38.948s 10%: 0.099+133+0.59 ms clock, 0.099+78/33/0+0.59 ms cpu, 72->78->70 MB, 78 MB goal, 1 P
pacer: sweep done at heap size 76MB; allocated 6MB of spans; swept 10013 pages at +2.094311e-004 pages/byte
pacer: assist ratio=+4.354140e+000 (scan 103 MB in 116->140 MB) workers=0+1
Higher latency than expected at iteration 395.  Tick was delayed 26.735579ms
pacer: H_m_prev=73777328 h_t=+6.622503e-001 H_T=122636387 h_a=+7.576055e-001 H_a=129671440 h_g=+1.000000e+000 H_g=147554656 u_a=+4.951149e-001 u_g=+2.500000e-001 W_
a=51445176 goalΔ=+3.377497e-001 actualΔ=+9.535521e-002 u_a/u_g=+1.980459e+000
gc 80 @39.688s 10%: 0.11+118+0.26 ms clock, 0.11+29/30/28+0.26 ms cpu, 116->123->55 MB, 140 MB goal, 1 P
pacer: sweep done at heap size 55MB; allocated 0MB of spans; swept 15855 pages at +3.785020e-004 pages/byte
pacer: assist ratio=+5.837807e+000 (scan 85 MB in 96->111 MB) workers=0+1
pacer: H_m_prev=58283288 h_t=+7.367016e-001 H_T=101220679 h_a=+7.406318e-001 H_a=101449744 h_g=+1.000000e+000 H_g=116566576 u_a=+2.500000e-001 u_g=+2.500000e-001 W_
a=34801104 goalΔ=+2.632984e-001 actualΔ=+3.930191e-003 u_a/u_g=+1.000000e+000
gc 81 @40.666s 9%: 0.11+56+0.20 ms clock, 0.11+0/16/39+0.20 ms cpu, 96->96->37 MB, 111 MB goal, 1 P
pacer: sweep done at heap size 37MB; allocated 0MB of spans; swept 12413 pages at +3.744516e-004 pages/byte
pacer: assist ratio=+1.236822e+001 (scan 62 MB in 70->75 MB) workers=0+1
Higher latency than expected at iteration 409.  Tick was delayed 75.602625ms

My lay-person reading of the trace is that the "child" garbage-generating goroutines are asked to assist with the GC, but then they end up blocking execution of both the runtime.timerproc goroutine and the main goroutine.
This seems like a scheduler issue and goroutines that generate minimal garbage should be prioritized over goroutines assisting with the GC.
gctrace=1,gcpacertrace=1 output
runtime trace  The 74ms delay occurs around 41 seconds into the trace.  The tick should have fired at 41003ms but did not fire until 41079ms and the G1 goroutine did not run until 41140ms.   The complete trace is split up into multiple chunks, but this GC can be found 12.6s into its chunk.

@rlh @aclements @dvyukov Is this on your radar for go1.9?  Have you had a chance to try the example code yet?
I tried the patches that Austin posted for #14812, but those did not resolve the issue in the example that I posted.  I also tried playing around with more scheduler options in the runtime, but I'm not familiar with the GC and scheduler and none of the basic "reschedule now" changes that I made seemed to help.

Your taskset invocation does not accurately model the behavior of a single CPU system. In particular, Go is still going to use a larger GOMAXPROCS, which is actively going to make this worse by trying to schedule multiple threads on a single physical core.
I think you should be able to get closer by setting GOMAXPROCS=1 when running the test, in addition to taskset. What does that do to the latency profile?

On Linux, the default NumCPU and GOMAXPROCS is determined with
sched_getaffinity so I think the command does emulate a one-cpu system.

I am also able to reproduce this issue on darwin amd64 with go1.8 and GOMAXPROCS=1.
$ GOMAXPROCS=1 ./bug18534
  0.0% -68.680343ms
  1.0% -36.249343ms
  5.0% -21.846205ms
  50.0% 33.946µs
  95.0% 21.279699ms
  99.0% 35.011957ms
  100.0% 58.775745ms
~ $ ./bug18534             # 4 cores
  0.0% -6.406381ms
  1.0% -4.595611ms
  5.0% -3.099409ms
  50.0% -22.001µs
  95.0% 3.246163ms
  99.0% 4.806314ms
  100.0% 7.189159ms

The scheduler/GC seems to need those extra cores to keep latency down.

Some more data points:
$ taskset -c 0 ./issue18534
  0.0% -70.533562ms
  1.0% -52.267319ms
  5.0% -35.006899ms
  50.0% -2.676µs
  95.0% 36.313112ms
  99.0% 57.591539ms
  100.0% 86.679896ms

GOMAXPROCS=2 taskset -c 0 ./issue18534
  0.0% -61.117609ms
  1.0% -31.714017ms
  5.0% -21.586722ms
  50.0% 949ns
  95.0% 20.895062ms
  99.0% 33.908325ms
  100.0% 62.686614ms

$ GOMAXPROCS=4 taskset -c 0 ./issue18534
  0.0% -33.988405ms
  1.0% -26.603924ms
  5.0% -11.102534ms
  50.0% 1.049µs
  95.0% 10.910585ms
  99.0% 23.701329ms
  100.0% 34.930726ms

$ GOMAXPROCS=5 taskset -c 0 ./issue18534
  0.0% -41.173521ms
  1.0% -15.659458ms
  5.0% -7.67135ms
  50.0% 70ns
  95.0% 7.62678ms
  99.0% 15.607855ms
  100.0% 41.169155ms

$ GOMAXPROCS=8 taskset -c 0 ./issue18534
  0.0% -17.46882ms
  1.0% -10.194657ms
  5.0% -5.408951ms
  50.0% -315ns
  95.0% 5.410479ms
  99.0% 10.139459ms
  100.0% 16.585539ms

$ GOMAXPROCS=16 taskset -c 0 ./issue18534
  0.0% -21.709796ms
  1.0% -8.175178ms
  5.0% -4.094512ms
  50.0% -173ns
  95.0% 4.09989ms
  99.0% 8.111605ms
  100.0% 21.653956ms

(top indicates that taskset is constraining the process to 1 core as expected.)
My understanding is that setting GOMAXPROCS will force the runtime to create more threads.  Even when those threads are running on the same core and have to be scheduled by the OS (linux in this case), that seems to lead to lower latency than just letting the Go runtime schedule a single thread.
Here are some measurements without taskset, but with different values for GOMAXPROCS:
$  ./issue18534 # 40 logical cores, GOMAXPROCS set by default
  0.0% -8.913432ms
  1.0% -103.472µs
  5.0% -51.566µs
  50.0% -56ns
  95.0% 50.329µs
  99.0% 107.945µs
  100.0% 8.917999ms

$ GOMAXPROCS=4 ./issue18534
  0.0% -16.824243ms
  1.0% -2.714634ms
  5.0% -80.119µs
  50.0% -358ns
  95.0% 82.118µs
  99.0% 2.521557ms
  100.0% 16.79671ms

$ GOMAXPROCS=5 ./issue18534
  0.0% -11.703387ms
  1.0% -205.105µs
  5.0% -72.006µs
  50.0% -1.152µs
  95.0% 72.07µs
  99.0% 181.71µs
  100.0% 11.744029ms

$ GOMAXPROCS=4 ./issue18534
  0.0% -13.447356ms
  1.0% -5.392172ms
  5.0% -89.006µs
  50.0% -401ns
  95.0% 90.569µs
  99.0% 5.325731ms
  100.0% 13.443492ms

$ GOMAXPROCS=5 ./issue18534
  0.0% -10.005695ms
  1.0% -205.612µs
  5.0% -68.098µs
  50.0% 241ns
  95.0% 64.149µs
  99.0% 186.699µs
  100.0% 10.021666ms

$ GOMAXPROCS=8 ./issue18534
  0.0% -180.467µs
  1.0% -106.693µs
  5.0% -56.36µs
  50.0% -228ns
  95.0% 56.368µs
  99.0% 91.327µs
  100.0% 213.66µs

It's interesting that there is a consistent change in the 99% latency between GOMAXPROCS=4 and =5.  I suspect that is an artifact of some decision within the runtime about the number of idle GC workers, but have not looked at a trace.  A similar looking change can also be seen in the taskset samples when going from GOMAXPROCS=4->5.
To me, these samples point to a deficiency in the Go runtime's handling of scheduling/preemption during GC.  That issue is probably hidden on systems with more than 4 cores, which many server systems have these days.

The delays do not correlate with the GCs. I lowered the frequency of the
GCs by setting GOGC=4000 thus trading RAM for fewer GCs. I also printed
when the Tick was delayed so we could correlate with the gctrace time
indicating when the GC ran. The resulting output is below and shows that
the GC is not running when the tick delays are detected.  Furthermore the
gctrace seems to indicate that STW portions of the GC are well below 1
millisecond.
My guess is that you have 11 goroutines running and occasionally the
scheduler doesn't get around to the goroutine that services the tick.
This is on tip.
rlh@rlh:~/work/code/bugs/latencyBug/src$ GOGC=4000 GODEBUG=gctrace=1
GOMAXPROCS=1 ./latencyBug
gc 1 @2.147s 0%: 0.011+77+0.009 ms clock, 0.011+0.46/20/21+0.009 ms cpu,
160->171->50 MB, 3498 MB goal, 1 P
Higher latency @8.096342913s than expected at iteration 79.    Tick was
delayed 25.805553ms
gc 4 @56.186s 0%: 0.015+91+0.010 ms clock, 0.015+43/22/0+0.010 ms cpu,
1534->1540->46 MB, 1752 MB goal, 1 P
Higher latency @57.60534838s than expected at iteration 574.  Tick was
delayed 23.092542ms
Higher latency @1m1.582175184s than expected at iteration 614.  Tick was
delayed 20.844583ms
Higher latency @1m2.407811598s than expected at iteration 622.  Tick was
delayed 21.701094ms
gc 5 @78.309s 0%: 0.008+67+0.007 ms clock, 0.008+33/20/2.2+0.007 ms cpu,
1773->1776->44 MB, 1899 MB goal, 1 P
0.0% -34.534406ms
1.0% -17.807413ms
5.0% -11.569303ms
50.0% -1.282µs
95.0% 11.480867ms
99.0% 17.951876ms
100.0% 34.711117ms
On Fri, May 19, 2017 at 1:48 AM, tarm<EMAIL_ADDRESS>wrote:

Some more data points:
$ taskset -c 0 ./issue18534
0.0% -70.533562ms
1.0% -52.267319ms
5.0% -35.006899ms
50.0% -2.676µs
95.0% 36.313112ms
99.0% 57.591539ms
100.0% 86.679896ms
GOMAXPROCS=2 taskset -c 0 ./issue18534
0.0% -61.117609ms
1.0% -31.714017ms
5.0% -21.586722ms
50.0% 949ns
95.0% 20.895062ms
99.0% 33.908325ms
100.0% 62.686614ms
$ GOMAXPROCS=4 taskset -c 0 ./issue18534
0.0% -33.988405ms
1.0% -26.603924ms
5.0% -11.102534ms
50.0% 1.049µs
95.0% 10.910585ms
99.0% 23.701329ms
100.0% 34.930726ms
$ GOMAXPROCS=5 taskset -c 0 ./issue18534
0.0% -41.173521ms
1.0% -15.659458ms
5.0% -7.67135ms
50.0% 70ns
95.0% 7.62678ms
99.0% 15.607855ms
100.0% 41.169155ms
$ GOMAXPROCS=8 taskset -c 0 ./issue18534
0.0% -17.46882ms
1.0% -10.194657ms
5.0% -5.408951ms
50.0% -315ns
95.0% 5.410479ms
99.0% 10.139459ms
100.0% 16.585539ms
$ GOMAXPROCS=16 taskset -c 0 ./issue18534
0.0% -21.709796ms
1.0% -8.175178ms
5.0% -4.094512ms
50.0% -173ns
95.0% 4.09989ms
99.0% 8.111605ms
100.0% 21.653956ms
(top indicates that taskset is constraining the process to 1 core as
expected.)
My understanding is that setting GOMAXPROCS will force the runtime to
create more threads. Even when those threads are running on the same core
and have to be scheduled by the OS (linux in this case), that seems to lead
to lower latency than just letting the Go runtime schedule a single thread.
Here are some measurements without taskset, but with different values for
GOMAXPROCS:
$  ./issue18534 # 40 logical cores, GOMAXPROCS set by default
0.0% -8.913432ms
1.0% -103.472µs
5.0% -51.566µs
50.0% -56ns
95.0% 50.329µs
99.0% 107.945µs
100.0% 8.917999ms
$ GOMAXPROCS=4 ./issue18534
0.0% -16.824243ms
1.0% -2.714634ms
5.0% -80.119µs
50.0% -358ns
95.0% 82.118µs
99.0% 2.521557ms
100.0% 16.79671ms
$ GOMAXPROCS=5 ./issue18534
0.0% -11.703387ms
1.0% -205.105µs
5.0% -72.006µs
50.0% -1.152µs
95.0% 72.07µs
99.0% 181.71µs
100.0% 11.744029ms
$ GOMAXPROCS=4 ./issue18534
0.0% -13.447356ms
1.0% -5.392172ms
5.0% -89.006µs
50.0% -401ns
95.0% 90.569µs
99.0% 5.325731ms
100.0% 13.443492ms
$ GOMAXPROCS=5 ./issue18534
0.0% -10.005695ms
1.0% -205.612µs
5.0% -68.098µs
50.0% 241ns
95.0% 64.149µs
99.0% 186.699µs
100.0% 10.021666ms
$ GOMAXPROCS=8 ./issue18534
0.0% -180.467µs
1.0% -106.693µs
5.0% -56.36µs
50.0% -228ns
95.0% 56.368µs
99.0% 91.327µs
100.0% 213.66µs
It's interesting that there is a consistent change in the 99% latency
between GOMAXPROCS=4 and =5. I suspect that is an artifact of some decision
within the runtime about the number of idle GC workers, but have not looked
at a trace. A similar looking change can also be seen in the taskset
samples when going from GOMAXPROCS=4->5.
To me, these samples point to a deficiency in the Go runtime's handling of
scheduling/preemption during GC. That issue is probably hidden on systems
with more than 4 cores, which many server systems have these days.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/18534#issuecomment-302614700, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AA7WnweLAPHJQai8FiqCIFwIX5ICu5aIks5r7S1BgaJpZM4Lcgtv
.

I re-ran your reproducer with current master, which annotates mark assists in the execution trace, and added a go badness() just before the "high latency" print so I'd have something to search for in the execution trace.
This does appear to be caused by GC and scheduling, though several things have to go wrong simultaneously. Here's the new trace I'm analyzing with tip.
The worst delay is a 57ms delay that gets reported at 1,571.288 ms into the trace. First, a fractional mark worker starts at 1,473.412 ms. Since there's just one P, this necessarily delays all other work. This is supposed to run for at most 10ms, but for reasons I don't fully understand runs for 20ms. This delays the timer goroutine, so six wakeups all happen at 1,493.612 ms when the fractional worker finishes. It so happens these are all childLoops that happened to randomly pick short sleeps. These childLoops each run for ~20ms, partly because they're doing a lot of assist work (#14951) and partly because they take a long time to run even when not assisting. Again, I'm not sure why this is 20ms instead of 10ms. Together, these consume the next 60ms of CPU time. The mainLoop wake up is supposed to happen fairly early in this 60ms window. Because these are saturating the CPU, again, the timer goroutine doesn't get to run, so another three timer wakeups pile up until 1,557.327 ms. One of these (it happens to be the last one serviced), is mainLoop.
I need to dig in to why all of these preemptions are happening at 20ms. My guess is that sysmon has fallen back to a 10ms tick and has to observe them in the same goroutine for two ticks in a row before preempting, which seems a bit bogus. I suspect fixing that and fixing the over-assist problem (#14951) would fix this issue.

@aclements Thanks for your work on this!
I cherry-picked your 2 patches onto tip and ran our actual application (not just the test reproduction case in this ticket):
Go v1.9.0:
I0830 00:43:14.905531  196377 main.go:804] logic loop time distribution (10000 samples): min = 4.808µs, 50% <= 11.472µs, 90% <= 18.879µs, 99% <= 5.018271ms, 99.9% <= 8.561956ms, max = 99.805794ms
I0830 00:59:59.306973  196377 main.go:804] logic loop time distribution (10000 samples): min = 5.443µs, 50% <= 10.039µs, 90% <= 16.947µs, 99% <= 53.701µs, 99.9% <= 11.057286ms, max = 136.385102ms
I0830 01:16:46.312292  196377 main.go:804] logic loop time distribution (10000 samples): min = 5.516µs, 50% <= 15.208µs, 90% <= 20.33µs, 99% <= 5.425571ms, 99.9% <= 12.001578ms, max = 122.486144ms

Go tip (d77d4f509) + patches applied
I0830 00:19:54.003992  185790 main.go:804] logic loop time distribution (10000 samples): min = 4.541µs, 50% <= 14.558µs, 90% <= 18.88µs, 99% <= 30.912µs, 99.9% <= 6.240578ms, max = 25.673079ms
I0830 00:36:35.013456  185790 main.go:804] logic loop time distribution (10000 samples): min = 4.521µs, 50% <= 17.13µs, 90% <= 20.187µs, 99% <= 29.954µs, 99.9% <= 6.296957ms, max = 12.587176ms
I0830 00:53:18.005010  185790 main.go:804] logic loop time distribution (10000 samples): min = 4.688µs, 50% <= 10.847µs, 90% <= 17.855µs, 99% <= 26.167µs, 99.9% <= 5.468955ms, max = 7.078815ms
I0830 01:10:01.610305  185790 main.go:804] logic loop time distribution (10000 samples): min = 5.305µs, 50% <= 13.549µs, 90% <= 19.775µs, 99% <= 31.4µs, 99.9% <= 6.350999ms, max = 16.407586ms

The 99.9% latencies moved from about 10ms -> 6ms, but the worst case latencies (out of 10k samples) went from ~120ms -> ~15ms.   That is huge improvement in the worst case mutator latencies and would be enough to resolve this issue for us.
The PRs are still marked as DO NOT SUBMIT until benchmarking is complete.  I am eager to see them merged, so let me know if there are is more I can help with to move them along.  (My real application does not create that much garbage, and so is not useful for benchmarking odd GC assist interactions.)

@tarm, that's great! Could you also check what effect the patches have on heap size and throughput?
You're running with GOMAXPROCS=1, right? There's still an issue with fractional workers running for a long time without preemption, which I suspect is responsible for the 15ms worst-case you saw. I just filed an issue focused on that problem (#21698).

For the recent benchmarking, I have been running with taskset -c 0 but that should constrain it to one core as well or better than GOMAXPROCS=1.
This application is a tick based application instead of a run-to-completion type application so throughput is more difficult to measure.  Over about 12 hours of running, the accumulated CPU time (as reported through top) was about 1.4% higher in the version with the patches, but that also includes changes from v1.9.0 to tip, so some of that may not be from the patches.
The application avoids allocation and is not a heavy user of RAM, so is also not a great stressor of the heap size and we have not specifically instrumented for heap usage.  That said, the memory usage reported through top was about 1% higher for the new version.


Torii create indices on members in model tables

We'll have to handle migration manually for these dynamically created tables, def a needed feature and bigger task.


Implement JobStorageConnection.GetFirstByLowestScoreFromSet's batch get version

Implement JobStorageConnection.GetFirstByLowestScoreFromSet's batch get version. It can reduce the time that get the recurring jobs id.
In my case, I have thousands of recurring job, they would triggered same time, but the max delay time of triggered would be ten miniutes. But when i add the implementation of  batch get method. the max delay time was significantly reduce.
You can click this to see why the batch get version can reduce the recurring job triggered delay time

@zh6335901,
Thanks!
I would have expected the two different approaches to produce the same query?
I will create a new release branch and merge it.

@zh6335901,
Thanks!
I would have expected the two different approaches to produce the same query?
I will create a new release branch and merge it.

Yes, The two different approaches produce the same recurring job ids.


document approvals_default_format

as @markijbema gave me the hint in https://github.com/kytrinyx/approvals/pull/69
let's document this possibility.

:+1:

Excellent idea! Thank you.


Add Github Action to publish releases to PyPI

This action will publish tagged versions to PyPI

It would be nice if this could also create a github release with a link to pypi. Bonus points if it can include changelog from something like git log.

I changed the action to run on release (similar to Logstash and FluentD release actions).


Misc updates for build changes

Signed-off-by: Mike Ng<EMAIL_ADDRESS>@lennysgarage @chenz4027 please take a look whenever you have some cycles. Thanks.
Note: some new line changes are intentional. Thanks.

hmm seems like the e2e failed


formatting changes to satisfy markdownlint

I'm just doing a simple PR with some formatting changes to get the ball rolling.  I use https://marketplace.visualstudio.com/items?itemName=DavidAnson.vscode-markdownlint to conform to some markdown formatting standard.  It's a VS Code extension I have configured to check markdown files globally (in case you wanted to do the same).
Let me know if any of these changes break the Codelab rendering (particularly the --- used to break out markdown metadata).
I'll pick this back up tomorrow morning to add some Typescript.

Technically this pr is merged.


[PINOT-3882] Bumping up zookeeper version to handle large number of watches

Controller connection to zookeeper can flap due to bug in zookeeper(ZOOKEEPER-706)
when large number of watches are set. Bumping up to zookeeper client
version that addresses this issues

Current coverage is 64.35% (diff: 100%)

No coverage report found for master at 8678561.

Powered by Codecov. Last update 8678561...0164449

Yes. Zookeeper SREs also suggested upgrading to this version.
-Adwait
On Tue, Oct 18, 2016 at 7:33 PM, Kishore Gopalakrishna <
<EMAIL_ADDRESS>wrote:

@kishoreg commented on this pull request.
Can you please confirm the zookeeper server version and make sure this
client version is compatible with server version.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/linkedin/pinot/pull/695#pullrequestreview-4795298,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABAmRoN50Kap8cYeC6lfemTSGO47OjJeks5q1YFzgaJpZM4KaXOb
.

Confirmed with zk team that this is safe version to use.
Checking this in to test mp change.


Feature/typescript

Re-written in TypeScript!

Now tests are working! Let's merge this into the main branch!


SAMA5D2 has no support for Flexcom SPI - best method?

To add flexcom spi support for SAMA5D2 I can:

create dedicated sam_flexcom_spi.c and .h files, keeping it separate from the existing sam_spi.c code. Or
Enhance the sam_spi.c code to  add the flexcom spi structs etc,. and the related set up functions

Any preference from anyone?

@TimJTi I don't know about SAMA5D2, but I think SAM5's flexcom is similar to SAMD21's sercom. Take a look how it is done on sam_sercom.c and how sam_spi.c uses it.

Aha! I searched the repo for flex and found no other helpful references and didn't know they were called sercon in other devices.
I'll go and take a look. It's why I asked 😎

SAMD21 and SAME5 only have sercom ports which can be SPI/I2C/USART/LIN whereas SAMA5D2 has 2x dedicated SPI, 2x dedicated I2C, 2x dedicated UART, *plus 5xFLEXCOM which are quite possibly unique to the SAMA5D2. Trust me to pick this one.
Since the arch is currently set up to manage the main SPI, I2C (TWI) and UARTs independently to the FLEXCOM ports and there was skeleton code for FLEXCOM USART (now finished off by me) I will stick with that philosophy and keep FLEXCOM SPI completely separate.
I have no FLEXCOM I2C (TWI) devices on my board so I'll leave that for someone else, but will be adding LIN support sometime next year.
PR to follow in a few days...followed by the usual 17+ iterations before I get it right...

Yes, I think you can take inspiration from sercom driver to implement a separated FLEXCOM SPI driver. You need to think how to let users select it in a flexible way in the menuconfig. So user could select 1 or 2 dedicated SPI and up to 5 FLEXCOM SPI. Since you are not planing to add FLEXCOM I2C (and UART?) just create the base file to be extended in the future case someone want to add them.

USART support already there and now working (via one of my PRs), so this will add SPI via a new sam_flexcom_spi.c file.
Kconfig support is there already, actually, but just doesn't do anything for SPI as no underlying support. Each of the FLEXCOMs have their own bool settings for SPI, TWI or USART and, in time, will gain one each for LIN.
I will proceed. Thanks

Hi @TimJTi I saw that SPI Flexcom support got accepted in https://github.com/apache/nuttx/commit/e6bfaa14ee353bdcc90682ceeb2d2ee456dd9b4a - it is without DMA. Do you plan to add DMA support?
Also are you working on I2C Flexcom support? (I am working on SAMA5D2 drivers too and don't want to duplicate work.)

I will be looking at DMA but not for a while - so if you do it first you'll save me the hassle! I think the SAMA5D2 DMA is fundamentally not working in NuttX so it does need some work.
I won't be adding I2C as my custom board has no need of it, meaning I have nothing to test it with of course.

Oh - I also have ClassD audio as a WIP at the moment and will be adding that in due course.

Ok, then I'll see if I can get the SPI Flexcom DMA working. I need that for my application. I'll will be adding I2C Flexcom support too. I'll also test the SPI Flexcom code with my custom board.

Thank you for adding the SPI Flexcom, by the way - it will save me a lot of hassle too. :)
BTW what is your application if you can say? If you would rather email, my contact info is on my GitHub profile.
Here's mine: https://github.com/starcat-io/jupiter-nano


The decryption error

Hi,
If the pkc12 file will be used as decryptionKey, the encryption.DecryptPayload returns an error: panic: crypto/rsa: decryption error. Is it intended?
Regards, Daniyar.

Hi @floorstyle,
Can you provide more details on the error that you're facing? Is this a p12 file that you're downloading from Mastercard Developers? If so, can you provide us with more details as to where/when the error is occurring?
We do have a test that loads a p12 file here

Hi @danny-gallagher ,
So I have tried the p12 tested key in this test  and got an error.
func TestEncryptPayload_ShouldEncryptRootArrays(t *testing.T) {
	payload := "[" +
		"   {}," +
		"   {}" +
		"]"

	decryptionKeyPath := "../testdata/keys/pkcs12/test_key.p12"
	certificatePath := "../testdata/certificates/test_certificate-2048.der"
	password := "Password1"

	decryptionKey, err := utils.LoadDecryptionKey(decryptionKeyPath, password)
	assert.Nil(t, err)
	certificate, err := utils.LoadEncryptionCertificate(certificatePath)
	assert.Nil(t, err)

	cb := jwe.NewJWEConfigBuilder()
	jweConfig := cb.WithDecryptionKey(decryptionKey).
		WithEncryptionPath("$", "$").
		WithDecryptionPath("encryptedData", "$").
		WithEncryptedValueFieldName("encryptedData").
		WithCertificate(certificate).
		Build()

	encryptedPayload := encryption.EncryptPayload(payload, *jweConfig)
	decryptedPayload := encryption.DecryptPayload(encryptedPayload, *jweConfig)

	assert.Equal(t, "[{},{}]", decryptedPayload)
}

Am I doing anything wrong?

Hi @floorstyle,
Can you try following the steps outlined here to convert your .p12 file to an RSA .key and let us know if this works for you?
openssl pkcs12 -in '<location of the .p12 file>' | openssl rsa -out <Filename of the key generated ending with .key>

Hi @floorstyle, it looks like the error you're getting above is actually related to the decryption process, and not with the loading of the decryption key.
The decryption (private) key you're using to decrypt the payload isn't associated with the encryption (public) certificate used to encrypt the payload.
Have you tried using the RSA (.key) file to decrypt an encrypted response from a Mastercard service?

Closing issue


How to run cucumber test by the example https://github.com/angular/protractor/tree/master/spec/cucumber

Hi Julie,
i'm new to protractor and cucumber. I've set up my new cucumber test like yours in : https://github.com/angular/protractor/tree/master/spec/cucumber
but i don't find any document for how to run the feature file (and then call the protractor test step). I know there is already adapter for cucumber in protractor runner. But how can i/ do i have to set up any thing more, so that cucumber knows about protractor? And how can i run cucumber tests?
thanks a lot in advance for a quick answer
BR, Emy

Hey @emyphan  can you please share what you have done? thanks!


clean up SmoothFunctions

We should check the subpackge BS/Utilities/SmoothFunctions.
https://github.com/UdK-VPT/BuildingSystems/tree/master/BuildingSystems/Utilities/SmoothFunctions
All functions that are usefull should get a documentation, a corresponding block and a runnable example with reference results. Then they can be added to Annex60/Utilities/Math:
https://github.com/iea-annex60/modelica-annex60/tree/master/Annex60/Utilities/Math
All other functions should be deleted or replaced with their MSL or Annex60 equivalent.

Statistics for concerend functions:

softcut: 16 function calls in 11 files
Buildings\Airvolumes\AirvolumeCompressible0D
Buildings\Constructions\Windows\RadiationTransmission
Climate\SolarRadiationTransformers\SolarRadiationTransformerGeneral
Climate\SolarRadiationTransformers\SolarRadiationTransformerIsotropicSky
Technologies\Chillers\CompressionChiller
Technologies\ElectricalStorages\BatterySimple.mo
Technologies\Photovoltaics\BaseClasses\ElectricalModels\ElectricalModelOneDiode
Technologies\Photovoltaics\BaseClasses\ElectricalModels\ElectricalModelOneDiodeMPP
Technologies\SolarThermal\ThermalCollector
Utilities\MoistAirFunctions\T_sat
Utilities\NcDataReader2\NcDataReader

softcut_lower: 6 function calls in 5 files
Buildings\Constructions\Windows\RadiationTransmission
Technologies\Chillers\CompressionChiller
Technologies\Photovoltaics\BaseClasses\ElectricalModels\ElectricalModelOneDiode
Technologies\Photovoltaics\BaseClasses\ElectricalModels\ElectricalModelOneDiodeMPP
Technologies\SolarThermal\ThermalCollector

softswitch: 12 function calls in 4 files Buildings\Data\AirpathCharacteristics\AirpathCharacteristicWindow
HAM\ConvectiveHeatTransfer\Gaps\alphaGap
HAM\ConvectiveHeatTransfer\Surfaces\alphaPlateFree
Utilities\MoistAirFunctions\enthalpy

softcut_upper: 4 function calls in 3 files
Climate\SolarRadiationTransformers\SolarRadiationTransformerIsotropicSky
Technologies\Chillers\CompressionChiller
Technologies\ElectricalStorages\BatterySimple

softfabs: 2 function calls in 2 files
Buildings\Airpathes\AirpathPrescribedPressure
Buildings\BaseClasses\AirpathGeneral

savepow: 2 function calls in 2 files:
Buildings\Airpathes\AirpathPrescribedPressure
Buildings\BaseClasses\AirpathGeneral

softpow: 12 function calls in 1 file
HAM\ConvectiveHeatTransfer\Surfaces\alphaPlateFree

No function calls within the entire library

powerLinearized
safereciprocal
safetanh

Actually I realized that these statistics are only lower boundaries, as several functions use relative paths.
I especially checked for powerLinearized and safereciprocal and it seems, that these really are not used within the library.

@jraedler : Do you still have the old documentation about the SmootFunctions (old SMILE context) and can you send it to @thorade

https://www.j-raedler.de/2010/10/smooth-transition-between-functions-with-tanh/

Here is my opinion on each function (alphabetically):

powerLinearized is not used, not documented, and it does not have a test, so I think we should just remove it
safepow should be deleted, just use normal power, and where that fails, explicitly use sign(x)
sagereciprocal is not used and does not have a test, so I think we should remove it
safetanh can be replaced by normal tanh
softcut_lower and softcut_upper do the same as IBPSA softMin and softMax. We should do testing which one is better (faster, more stable, higher continuity) and then use that function everywhere, and remove the other one.
softcut uses softcut_upper or softcut_lower. After fixing the above point, we should possibly rename it, then add a block and example, and move it to IBPSA.
softabs: not sure yet
softpow: not sure yet
softswitch: not sure yet


GooglePlus and OpenID

Any plans on making a change to support GooglePlus via OpenID?
Looking at the sources, I see GooglePlus via Oauth2 is supported, just not OpenID.
Believe this is a straight forward and simple change.
With Google deprecating Google and OpenID and moving to GooglePlus OpenID in 4/2015... it would be great to have this in place prior.
https://developers.google.com/+/api/auth-migration#oid2
Thanks
David

Hey there,
I opened an issue some weeks ago : https://github.com/omab/python-social-auth/issues/477
And fixed Google OpenID on my fork https://github.com/cambonf/python-social-auth/tree/bug/google-openid-connect
I haven't opened a PR yet. Maybe there's a better solution (I'm quite new to this project) but it works for us.

Thanks @cambonf
Looked at your branch and its very straight forward and in line with the architecture principles of this package... though I am a newbie to this package.
Any chance I can encourage you to do a PR with the owner and merge?  In general I don't like working off branches.  Take a chance, the author seems like a nice fellow.
Thanks
David

Hi sorry for the late answer,
Yup I'll do a PR and see with the owner :)


Suspected typo in lambda configuration

Hey,
I believe there is a typo in the env var that the lambda reads for configuration. Currently, one variable controls the lambdas interval AND the lambdas timeout:
https://github.com/buildkite/buildkite-agent-scaler/blob/02292a7f8aa5f853c4b48f929f261582e690a7be/lambda/main.go#L54-L66
These two variable were controlled with two env var prior to #13 :
https://github.com/buildkite/buildkite-agent-scaler/blob/dfd3df82bdb787ac6ec88561a499882dea937310/lambda/main.go#L46-L60
Looks like a refactor with an accidental copy paste error?

Looks like this is fixed? Should this ticket be closed too?

Yep, thanks @eculver and @amu-g 👍🏼


[Derived] When adding members to conformance context on extension, add to nominal

Without this we're adding the members to the extension itself which then triggers 'Added member to the wrong context' assertions in debug builds, so instead we can try to add to the nominal.
Resolves rdar://140888172

@swift-ci please smoke test


Reply suggestion dataset

https://www.microsoft.com/en-us/research/uploads/prod/2021/06/mrs_acl2021_zhang.pdf

I will take this.


Update shell scripts

Proposed changes
Issue Number: https://github.com/apache/doris/discussions/9163
Problem Summary:
Update the start and stop scripts.
Checklist(Required)

Does it affect the original behavior: No
Has unit tests been added: No Need
Has document been added or modified: No Need
Does it need to update dependencies: No
Are there any changes that cannot be rolled back: No

I find the default log path is not fixed, it is just the current working directory, but I do not know how to change it.


[SMALLFIX]Change md

Change md

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/2885/
Test PASSed.


Make new sdks repo public

###Acceptance criteria:

[ ] This  repo public https://github.com/commercetools/commercetools-java-sdks

Thanks

Done 🎉


Currency answer component

What is the context of this PR?
Added currency answer component
How to review

Check component against trello task here: https://trello.com/c/asIXePnn/163-add-edit-currency-answer-component-3
View in storybook
Run tests

 Add/edit currency answer component (3)


[LibraryOfCongress] Added new extractor (Closes #3188)

Added extractor of loc.gov, which closes #3188. I am not an experienced programmer, so I am sure I did a bunch of mistakes, but the extractor works (for me at least).

I know I mistakenly added multiple independent commits in one pull request. I'll close my other PR and avoid it next time. Apologies.

Note that the extractor does not include urls in the from of https://www.loc.gov/item/mp, as they have can easily be downloaded from a browser (download option) and the generic extractor also seems to be able to download them.

thanks for feedback @dstftw. Does this seem good?

thanks for feedback @yan12125. Does this look good?

Cherry-picked in 9c3c447eb389726d98189d972a2d772ef729132d, thanks.


HDDS-9825. Fix ListKeys trailing slash prefix issue.

What changes were proposed in this pull request?
When prefix is passed with trailing slash in getOMKeyInfoIfExists still we are querying in fileTable and returning back the result. Instead this prefix is assumed to be checked only in directory table in this case.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-9825
How was this patch tested?
New integration test added.

@mladjan-gadzic @GeorgeJahad can you please take a look?

@SaketaChalamchala can you please take a look as well?

I think a better solution would be to normalize the keyName for FSO instead of checking for trailing slash. While normalizing, don't preserve trailing slash.

I think a better solution would be to normalize the keyName for FSO instead of checking for trailing slash. While normalizing, don't preserve trailing slash.

@sadanand48 Thanks for the review, We cannot just use the normalize here as normalize will remove the trailing slash. But we want to use trailing slash here to determine whether key check has to be checked in directory table or file table or both.

But we want to use trailing slash here to determine whether key check has to be checked in directory table or file table or both.

Thanks @ashishkumar50 for clarifying  and the offline discussion. Makes sense for use case of ozone sh / S3 client to send the prefix key name that has a slash and the ending slash must not be removed indicating that it needs to display all the children (sub-files/subdirs) in that prefix(directory)


Support proxy protocol

Proxy protocol is a way for a load balancer to indicate the original client address of the connection.   It serves a similar function to the X-Forwarded-For header, but it's protocol-agnostic, e.g. it works with tunneled TLS requests and any other non-HTTP TCP connection.   Proxy protocol is gaining wide adoption in "cloud" environments, so it would be nice to see support for it in TS.
The protocol is documented here; essentially, some data is sent at the start of the TCP connection containing the client address.
There are two separate parts to feature:

TS should optionally accept proxy protocol in incoming requests, and store the client address somewhere.
TS should optionally send a proxy protocol header in origin connections.  This header would contain the stored client address if the client sent its own proxy protocol header, or else the client socket address.

This should be configurable:

On TCP listeners to accept incoming proxy protocol (e.g. using a port definition like 80:proxy 8443:ssl,proxy).
On origin connections, possibly by a default records.config option that can be changed in a remap.
On origin connections in a plugin, using TSHttpTxnConfigIntSet(txnp, TS_CONFIG_PROXY_PROTOCOL, 1).
There should be a way for plugins to change the stored client address, e.g. with TSHttpTxnSetProxyAddress(txnp, sockaddr).

Previous JIRA issue for this: https://issues.apache.org/jira/browse/TS-4398

Chiming in, this would be pretty handy to have. There are times when we have some abstraction layer that doesn't talk HTTP and we need to pass along the client IP in some way.

I would be willing to take a stab at implementing this, if I could get some guidance on the direction to take.
After my first look at the code, my impression is to follow the pattern of the handoff to SocksProxy in UnixNetProcessor.  If we are using proxy protocol, create a Continuation that will handle events until the proxy protocol header is fully parsed, and afterwards pass through all events to the original Continuation.

I will try to do a sketch of how this should be implemented.

@SolidWallOfCode bump

Support has been added for incoming/client connections.
I am closing this for now until someone wants to add support for origin connections.


InternalServerError with EnhancedDateTimePicker 2.0.2 and Vaadin 21.0.2

This time I'm running a Vaadin 21.0.2 app on Windows 10 and get the below error on startup. Without this component, the app starts and displays fine.
<dependency>
    <groupId>org.vaadin.addons.componentfactory</groupId>
    <artifactId>enhanced-date-time-picker</artifactId>
    <version>2.0.2</version>
</dependency>

There was an unexpected error (type=Internal Server Error, status=500).
java.lang.IllegalStateException: Failed to find the following files: - ./enhancedTimepickerConnector.js Locations searched were: - `C:\Users\*****\IdeaProjects\komunumo-server\.\frontend` in this project - `META-INF/frontend` in included JARs Please, double check that those files exist. If you use a custom directory for your resource files instead of default `frontend` folder then make sure you it's correctly configured (e.g. set 'vaadin.frontend.frontend.folder' property)
com.vaadin.flow.server.ServiceException: java.lang.IllegalStateException:

  Failed to find the following files:
      - ./enhancedTimepickerConnector.js

  Locations searched were:
      - `C:\Users\*****\IdeaProjects\komunumo-server\.\frontend` in this project
      - `META-INF/frontend` in included JARs

  Please, double check that those files exist. If you use a custom directory for your resource files instead of default `frontend` folder then make sure you it's correctly configured (e.g. set 'vaadin.frontend.frontend.folder' property)

	at com.vaadin.flow.server.frontend.AbstractUpdateImports.getUniqueEs6ImportPaths(AbstractUpdateImports.java:385) ~[flow-server-8.0.2.jar:8.0.2]
	at com.vaadin.flow.server.frontend.AbstractUpdateImports.getModuleLines(AbstractUpdateImports.java:398) ~[flow-server-8.0.2.jar:8.0.2]
	at com.vaadin.flow.server.frontend.AbstractUpdateImports.collectModules(AbstractUpdateImports.java:311) ~[flow-server-8.0.2.jar:8.0.2]
	at com.vaadin.flow.server.frontend.AbstractUpdateImports.run(AbstractUpdateImports.java:115) ~[flow-server-8.0.2.jar:8.0.2]
	at com.vaadin.flow.server.frontend.TaskUpdateImports.execute(TaskUpdateImports.java:369) ~[flow-server-8.0.2.jar:8.0.2]
	at com.vaadin.flow.server.frontend.NodeTasks.execute(NodeTasks.java:762) ~[flow-server-8.0.2.jar:8.0.2]
	at com.vaadin.base.devserver.startup.DevModeInitializer.runNodeTasks(DevModeInitializer.java:499) ~[vaadin-dev-server-8.0.2.jar:na]
	at com.vaadin.base.devserver.startup.DevModeInitializer.lambda$initDevModeHandler$0(DevModeInitializer.java:387) ~[vaadin-dev-server-8.0.2.jar:na]
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run$$$capture(CompletableFuture.java:1736) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.exec(CompletableFuture.java:1728) ~[na:na]
	at java.base/java.util.concurrent.ForkJoinTask.doExec$$$capture(ForkJoinTask.java:290) ~[na:na]
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java) ~[na:na]
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) ~[na:na]
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) ~[na:na]
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) ~[na:na]
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) ~[na:na]

I tried a mvnw clean and deleted the node_modules to force a clean build, without success.

Additional info: version 2.0.1 works fine!

I created a minimal project with only one view and only this component to show the error:
https://github.com/McPringle/vaadin21-enhanceddatetimepicker

Hello. Thanks for reporting this. I just did a checkout of your project and it's working okay. Also tested it in a sampler project of mine and couldn't reproduce the mentioned error either. Maybe you could try the Vaadin Dance on your project to get rid of any wrong updates.

Hi @paodb, never heard of the Vaadin Dance before. Thanks for pointing at it, it might be useful for other problems. I tried the Vaadin Dance with my test project, still the same error - but the message changed slightly. On my first attempt, I got the above-mentioned error message on the first request to the view. After the Vaadin Dance, I get the message while starting the server (while running the Maven task vaadin:build-frontend):
ERROR in ../target/frontend/generated-flow-imports.js
Module not found: Error: Can't resolve './enhancedTimepickerConnector.js' in 'C:\Users\McPringle\IdeaProjects\vaadin21-enhanceddatetimepicker\target\frontend'

Followed by a NPE:
java.lang.NullPointerException: Cannot invoke "java.util.concurrent.locks.Lock.lock()" because the return value of "com.vaadin.flow.server.VaadinSession.getLockInstance()" is null
	at com.vaadin.flow.server.VaadinSession.lock(VaadinSession.java:685) ~[flow-server-8.0.2.jar:8.0.2]
	at com.vaadin.flow.server.VaadinSession.valueUnbound(VaadinSession.java:212) ~[flow-server-8.0.2.jar:8.0.2]
	at org.apache.catalina.session.StandardSession.removeAttributeInternal(StandardSession.java:1860) ~[tomcat-embed-core-9.0.53.jar:9.0.53]
	at org.apache.catalina.session.StandardSession.expire(StandardSession.java:861) ~[tomcat-embed-core-9.0.53.jar:9.0.53]
	at org.apache.catalina.session.StandardSession.isValid(StandardSession.java:660) ~[tomcat-embed-core-9.0.53.jar:9.0.53]
	at org.apache.catalina.session.ManagerBase.processExpires(ManagerBase.java:608) ~[tomcat-embed-core-9.0.53.jar:9.0.53]
	at org.apache.catalina.session.ManagerBase.backgroundProcess(ManagerBase.java:591) ~[tomcat-embed-core-9.0.53.jar:9.0.53]
	at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5622) ~[tomcat-embed-core-9.0.53.jar:9.0.53]
	at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1365) ~[tomcat-embed-core-9.0.53.jar:9.0.53]
	at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1369) ~[tomcat-embed-core-9.0.53.jar:9.0.53]
	at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1369) ~[tomcat-embed-core-9.0.53.jar:9.0.53]
	at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1347) ~[tomcat-embed-core-9.0.53.jar:9.0.53]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[na:na]
	at java.base/java.util.concurrent.FutureTask.runAndReset$$$capture(FutureTask.java:305) ~[na:na]
	at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java) ~[na:na]
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.53.jar:9.0.53]
	at java.base/java.lang.Thread.run(Thread.java:833) ~[na:an]

I'm running this on Windows (latest patch level) within IntelliJ IDEA (latest version) if it might be important.

I made another test. This time I went to start.vaadin.com and created a Flow app with Vaadin 21, Java 11, and one empty view. I downloaded and extracted the project, opened it within IntelliJ IDEA, and started it: Everything is working fine.
Then I added the dependency for EnhancedDateTimePicker version 2.0.2 to the pom.xml and to the view. Now it crashes while starting with the error message mentioned in my report. I can reproduce this every time, even when I start with a new and empty project.