Benchmarking

I'm trying to make a simple benchmark to a very simple code using "tiny_http" like this:

fn main(){
let server = Server::http("<IP_ADDRESS>:80");
match server {
	Ok(_) => println!("Success"),
	Err(_) => println!("Error")
}

for request in server.expect("REASON").incoming_requests() {
    let response = Response::from_string("Hello!");
	let _ = request.respond(response);
}

}
I'm doing an extremely soft "benchmark" and getting those results:

hugo@anakin:~/hermit-rs-template$ ab -n 10 -c 3 -r "http://<IP_ADDRESS>/pruebastress"
This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking <IP_ADDRESS> (be patient).....done
Server Software:        tiny-http
Server Hostname:        <IP_ADDRESS>
Server Port:            80
Document Path:          /pruebastress
Document Length:        0 bytes
Concurrency Level:      3
Time taken for tests:   0.286 seconds
Complete requests:      10
Failed requests:        20
(Connect: 0, Receive: 10, Length: 0, Exceptions: 10)
Total transferred:      156 bytes
HTML transferred:       13 bytes
Requests per second:    34.97 [#/sec] (mean)
Time per request:       85.794 [ms] (mean)
Time per request:       28.598 [ms] (mean, across all concurrent requests)
Transfer rate:          0.53 [Kbytes/sec] received
Connection Times (ms)
min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:    12   14   4.5     12      27
Waiting:        0    0   0.0      0       0
Total:         12   14   4.5     12      27
Percentage of the requests served within a certain time (ms)
50%     12
66%     12
75%     13
80%     14
90%     27
95%     27
98%     27
99%     27
100%     27 (longest request)
Looks like concurrent connections are not handled also with no concurrency at all I get this:

hugo@anakin:~/hermit-rs-template$ ab -n 1000 -r "http://<IP_ADDRESS>/pruebastress"
This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking <IP_ADDRESS> (be patient)
Completed 100 requests
Completed 200 requests
(...)
Finished 1000 requests
(...)
Document Path:          /pruebastress
Document Length:        13 bytes
Concurrency Level:      1
Time taken for tests:   213.315 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      156000 bytes
HTML transferred:       13000 bytes
Requests per second:    4.69 [#/sec] (mean)
Time per request:       213.315 [ms] (mean)
Time per request:       213.315 [ms] (mean, across all concurrent requests)
Transfer rate:          0.71 [Kbytes/sec] received
Connection Times (ms)
min  mean[+/-sd] median   max
Connect:       20   30   4.3     32      56
Processing:   146  183   6.2    185     208
Waiting:      132  166   5.9    167     189
Total:        170  213   5.9    213     256
Percentage of the requests served within a certain time (ms)
50%    213
66%    214
75%    215
80%    215
90%    217
95%    219
98%    222
99%    224
100%    256 (longest request)
My console output (qemu-system-aarch64):

[LOADER][INFO] Loader: [0x40200000 - 0x4021f000]
[LOADER][INFO] Found ELF file with size 30194072
[LOADER][INFO] Parsing kernel from ELF at 0x48000000..0x49ccb998 (len = 0x1ccb998 B / 30194072 B)
[LOADER][INFO] Loading kernel to 0x40400000..0x40a2d228 (len = 0x62d228 B / 6476328 B)
[LOADER][INFO] TLS is at 0x408d90a8..0x408d9148 (len =  0xa0 B / 160 B)
[LOADER][INFO] Detect 1 CPU(s)
[LOADER][INFO] Detect UART at 0x9000000
[LOADER][INFO] Jumping to HermitCore Application Entry Point at 0x406b2794
[0][INFO] Welcome to Hermit 0.6.7
[0][INFO] Kernel starts at 40400000
[0][INFO] BSS starts at 0x40a2b7b0
[0][INFO] tls_info = Some(
TlsInfo {
start: 0x408d90a8,
filesz: 0x20,
memsz: 0xa0,
align: 0x8,
},
)
[0][INFO] RAM starts at physical address 40000000
[0][INFO] Physical address range: 16384GB
[0][INFO] Support of 4KB pages: true
[0][INFO] Support of 16KB pages: false
[0][INFO] Support of 64KB pages: true
[0][INFO] Total memory size: 500 MB
[0][INFO] Kernel region: [40400000 - 40c00000]
[0][INFO] A pure Rust application is running on top of Hermit!
[0][INFO] Heap: size 432 MB, start address 40c00000
[0][INFO] Heap is located at 0x40c00000..0x5bc00000 (0 Bytes unmapped)
[0][INFO]
[0][INFO] ===================== PHYSICAL MEMORY FREE LIST ======================
[0][INFO] 0x0000005BCD4000 - 0x00000060000000
[0][INFO] ======================================================================
[0][INFO]
[0][INFO]
[0][INFO] ================== KERNEL VIRTUAL MEMORY FREE LIST ===================
[0][INFO] 0x0000005BC00000 - 0x00000100000000
[0][INFO] ======================================================================
[0][INFO]
[0][INFO] Intialize generic interrupt controller
[0][INFO] Found GIC Distributor interface at 8000000 (size 0x10000)
[0][INFO] Found generic interrupt controller at 80a0000 (size 0xF60000)
[0][INFO]
[0][INFO] ========================== CPU INFORMATION ===========================
[0][INFO] Processor compatiblity:  arm,cortex-a72
[0][INFO] Counter frequency:       62500000 Hz (from CNTFRQ_EL0)
[0][INFO] Run on hypervisor
[0][INFO] ======================================================================
[0][INFO]
[0][INFO] Hermit booted on 2023-12-29 11:39:58.0 +00:00:00
[0][INFO] Mapping PCI Enhanced Configuration Space interface to virtual address 60000000 (size 0x10000000)
[0][INFO] Scanning PCI Busses 0 to 255
[0][INFO] Compiled with PCI support
[0][INFO] Compiled with ACPI support
[0][INFO] Compiled with FSGSBASE support
[0][INFO] Compiled with SMP support
[0][INFO]
[0][INFO] ======================== PCI BUS INFORMATION =========================
[0][INFO] 00:00 Host bridge [0600]: Red Hat, Inc. QEMU PCIe Host bridge [1B36:0008]
[0][INFO] 00:01 Ethernet controller [0200]: Red Hat, Inc. Virtio network device [1AF4:1041], IRQ 4, BAR1 Memory32 { address: 0x0, size: 0x1000, prefetchable: false }, BAR4 Memory64 { address: 0x8000000000, size: 0x4000, prefetchable: true }
[0][INFO] ======================================================================
[0][INFO]
[0][INFO] Hermit is running on common system!
[0][INFO] Found virtio network device with device id 0x1041
[0][WARN] Currently only mapping of 64 bit BAR's is supported!
[0][WARN] Currently only mapping of 64 bit BAR's is supported!
[0][INFO] Non Virtio PCI capability with id 11 found. And NOT used.
[0][ERROR] Found virtio capability whose BAR is not mapped or non existing. Capability of type 5 and id 0 for device 1041, can not be used!
[0][INFO] Feature set wanted by network driver are in conformance with specification.
[0][INFO] Feature set wanted by network driver are in conformance with specification.
[0][INFO] Driver found a subset of features for virtio device 1041. Features are: [VIRTIO_NET_F_CSUM, VIRTIO_NET_F_GUEST_CSUM, VIRTIO_NET_F_MAC, VIRTIO_NET_F_MRG_RXBUF, VIRTIO_NET_F_STATUS, VIRTIO_F_RING_INDIRECT_DESC, VIRTIO_F_VERSION_1]
[0][INFO] Features have been negotiated between virtio network device 1041 and driver.
[0][INFO] Created SplitVq: idx=0, size=256
[0][INFO] Created SplitVq: idx=1, size=256
[0][INFO] Network driver successfully initialized virtqueues.
[0][INFO] Device specific initialization for Virtio network device 1041 finished
[0][INFO] Network device with id 1041, has been initialized by driver!
[0][INFO] Virtio-net link is up after initialization.
[0][INFO] Virtio network driver initialized.
[0][INFO] Install virtio interrupt handler at line 4
[0][INFO] Trying to initialize network!
[0][INFO] MAC address 52-54-00-12-34-56
[0][INFO] Configure network interface with address <IP_ADDRESS>/24
[0][INFO] Configure gateway with address <IP_ADDRESS>
[0][INFO] MTU: 1514 bytes
[0][WARN] Unable to read entropy! Fallback to a naive implementation!
Success
I'm using Hermit 0.8.0. I'm missing something for sure as this low performance is not normal.... any hint?

I have to check. In the past, we had some performance issue. But I thought that we have solved it. Maybe it come back with an PR. Currently, our checks do not evaluate, if a PR creates some power losses.

On my system I got better performance. Naive question. Do you build the Hermit application with --release?

No. But I got 4/5 requests/second which is several orders of magnitude
below what I would expect... how much performance are you getting? (just in
case I'll try with --release...).
El jue., 4 ene. 2024 19:34, Stefan Lankes @.***>
escribió:

On my system I got better performance. Naive question. Do you build the
Hermit application with --release?
—
Reply to this email directly, view it on GitHub
https://github.com/hermit-os/hermit-rs/issues/516#issuecomment-1877576780,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AQVGKU5TPEAOWFSD6SIMEVLYM3Y4DAVCNFSM6AAAAABBGQ7ZBKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZXGU3TMNZYGA
.
You are receiving this because you authored the thread.Message ID:
@.***>

Yeah, it's expected to be slow without --release. We should add a hint to the README in https://github.com/hermit-os/hermit-rs-template.

Network performance is something that we are actively looking into, so it should get better in the future. :)

My apologies.  I'm getting much, much more. This 40 req/sec is because an experimental KASLR I was implementing and also plus SSL offloading proxy before.  Just tried your vanilla code with plain HTTP and I get 10.000 req/sec, which looks pretty good!!!
hugo@anakin:~/home/hugo$ wrk -c100 -d10000 -t 10 http://<IP_ADDRESS>:9975
Running 167m test @ http://<IP_ADDRESS>:9975
  10 threads and 100 connections
^C  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     8.97ms    2.27ms  25.83ms   82.25%
    Req/Sec     1.08k   323.60    10.69k    86.06%
  164378 requests in 15.32s, 38.22MB read
Requests/sec:  10731.98
Transfer/sec:      2.50MB

Ah, I am relieved. Thanks for clarifying!


Crashes at launch

Hello,
My instance crashes right at launch.
When I run starknet-devnet I get
Traceback (most recent call last):
  File "/Users/alexandre/code/alexandrewolff/nile-cairo-examples/env/bin/starknet-devnet", line 8, in <module>
    sys.exit(main())
  File "/Users/alexandre/code/alexandrewolff/nile-cairo-examples/env/lib/python3.8/site-packages/starknet_devnet/server.py", line 329, in main
    app.run(host=args.host, port=args.port)
  File "/Users/alexandre/code/alexandrewolff/nile-cairo-examples/env/lib/python3.8/site-packages/flask/app.py", line 920, in run
    run_simple(t.cast(str, host), port, self, **options)
  File "/Users/alexandre/code/alexandrewolff/nile-cairo-examples/env/lib/python3.8/site-packages/werkzeug/serving.py", line 1047, in run_simple
    fd = int(os.environ["WERKZEUG_SERVER_FD"])
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/os.py", line 675, in __getitem__
    raise KeyError(key) from None
KeyError: 'WERKZEUG_SERVER_FD'

Same with nile node
Traceback (most recent call last):
  File "/Users/alexandre/code/alexandrewolff/nile-cairo-examples/env/bin/starknet-devnet", line 8, in <module>
    sys.exit(main())
  File "/Users/alexandre/code/alexandrewolff/nile-cairo-examples/env/lib/python3.8/site-packages/starknet_devnet/server.py", line 329, in main
    app.run(host=args.host, port=args.port)
  File "/Users/alexandre/code/alexandrewolff/nile-cairo-examples/env/lib/python3.8/site-packages/flask/app.py", line 920, in run
    run_simple(t.cast(str, host), port, self, **options)
  File "/Users/alexandre/code/alexandrewolff/nile-cairo-examples/env/lib/python3.8/site-packages/werkzeug/serving.py", line 1047, in run_simple
    fd = int(os.environ["WERKZEUG_SERVER_FD"])
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/os.py", line 675, in __getitem__
    raise KeyError(key) from None
KeyError: 'WERKZEUG_SERVER_FD'
Traceback (most recent call last):
  File "/Users/alexandre/code/alexandrewolff/nile-cairo-examples/env/bin/nile", line 8, in <module>
    sys.exit(cli())
  File "/Users/alexandre/code/alexandrewolff/nile-cairo-examples/env/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/alexandre/code/alexandrewolff/nile-cairo-examples/env/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/alexandre/code/alexandrewolff/nile-cairo-examples/env/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/alexandre/code/alexandrewolff/nile-cairo-examples/env/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/alexandre/code/alexandrewolff/nile-cairo-examples/env/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/alexandre/code/alexandrewolff/nile-cairo-examples/env/lib/python3.8/site-packages/nile/cli.py", line 190, in node
    node_command(host, port)
  File "/Users/alexandre/code/alexandrewolff/nile-cairo-examples/env/lib/python3.8/site-packages/nile/core/node.py", line 21, in node
    subprocess.check_call(["starknet-devnet", "--host", host, "--port", str(port)])
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['starknet-devnet', '--host', '<IP_ADDRESS>', '--port', '5000']' returned non-zero exit status 1.

I run on macOs 12.3.1 under apple m1 silicon cpu.
Any idea what causes that and how to fix ?

You can try running pip install werkzeug==2.0.3, this should fix the issue for now. Fix will should be deployed in the upcoming verison

Indeed it works now. Thanks !


Adding reconciliation loop functionality to pgbench workload

This change establishes several states for the pgbench workload and implements loop control through the operator reconciliation function.


Query patterns on DbSet (Local, Store, etc.)

Historically, the fact that iterating/querying DbSet directly gives you results from the database has been confusing (because it doesn't reflect additions/deletions made in the change tracker). We should revisit this design to try and do something less confusing.
One proposition is to not have DbSet be queryable and instead have methods like .LocalQuery, .DatabaseQuery etc. to make things more explicit. We could also have separate methods for Tracked and Untracked queries, which makes it more obvious that there is a choice.
We should also consider what we could do in the future with merging local and database queries (i.e. fall back to database when it's not in-memory etc.). We probably wouldn't do this for the initial EF7 release, but we should come up with a design that supports things like this in the future. See this CodePlex issue for some customer feedback - https://entityframework.codeplex.com/workitem/2561.

i'm hunting for the DbSet Local method. Found this ...will keep looking. (and Find) :)

@julielerman Local is not implemented yet. You can query against the ChangeTracker (context.ChangeTracker.Entries<T>) to get at local data. There is nothing yet that does the data binding stuff from .Local.
See https://github.com/aspnet/EntityFramework/issues/797 for Find. It's not done and we are still debating whether to do it at all. See the other work item linked from it. We will likely do one or the other, but maybe not until after RTM.


{Compute} az ppg and az restore-point collection: Migrate commands to CodeGen V2

Related command

az ppg show/delete
az restore-point collection list-all/list/delete
Description

Migrate part of az ppg and az restore-point collection commands.
Some commands in az ppg are not migrated yet:
az ppg create/list/update
Some commands in az restore-point collection are not migrated yet:
az restore-point collection create/show/update
Testing Guide

History Notes

[Component Name 1] BREAKING CHANGE: az command a: Make some customer-facing breaking change
[Component Name 2] az command b: Add some customer-facing feature

This checklist is used to make sure that common guidelines for a pull request are followed.

[x] The PR title and description has followed the guideline in Submitting Pull Requests.

[x] I adhere to the Command Guidelines.

[x] I adhere to the Error Handling Guidelines.

Compute


[rlgl] Custom blend mode with one function call

Right now setting custom blend mode requires two function calls: rlSetBlendFactors() and rlSetBlendMode.
I imagine instead of doing rlSetBlendFactors to store the factors and rlSetBlendMode to apply them, just having something like rlSetBlendModeCustom where you pass blend factors and it just applies them. There's no reason for user to just store them, like, there's no situation where a user might do a bunch of rlSetBlendFactors without rlSetBlendMode.
The rlSetBlendMode function stays pretty much the same and the new way of setting custom blend mode will also be just one function call, instead of two.

I've been reviewing this issue. Here a recap of current implementation:

rlSetBlendFactors()/rlSetBlendFactorsSeparated(): Store blending mode factors but does not apply them.
rlSetBlendMode(mode): Apply a pre-defined blend mode, in case of RL_BLEND_CUSTOM/RL_BLEND_CUSTOM_SEPARATE, apply the stored factors. It also guards the re-applying of an already applied blend mode, for performance cost concerns.
BeginBlendMode()/EndBlendMode(): Wraps rlBlendMode() call on raylib side. It is done this way because OpenGL pipeline state-changes that require issuing a draw call (and force batching) are wrapped in Begin*()/End*() statements.

As per mi understanding, the proposed addition would be a function like:
void rlSetBlendModeCustom(int glSrcFactor, int glDstFactor, int glEquation)
{
    rlDrawRenderBatch(RLGL.currentBatch);
    glBlendFunc(glSrcFactor, glDstFactor);
    glBlendEquation(glEquation);
}

In this case, if this function is called in the middle of a drawing process, despite the factors could have not changed, the pipeline is stalled, a draw call is issued and the blendind states are set again, implying a performance cost.
There is also the separate blend factors option, probably a new separate function should be added for it but same issue will happen.

there's no situation where a user might do a bunch of rlSetBlendFactors without rlSetBlendMode.

Actually there is. In my tools I set the factors for a custom blend mode at initialization and later in the loop I can just enable/disable the custom blend mode as desired.
Personally I neither like that rlSetBlendFactors()/rlSetBlendFactorsSeparated() require OpenGL specific input parameters, afaik are the only functions in raylib that do not abstract those parameters.
What is your proposed redesign considering this explanation?

@veins1 I'm afraid that solution is not possible, BeginBlendMode() belongs to rcore module and rlSetBlendMode() should belong to rlgl that does not depend on rcore. Also, if that function was somewhat moved to rcore it would expose at raylib level internal OpenGL values that should be defined and exposed to users, breaking the abstraction and increasing considerable the complexity.
Sincerely, I prefer to keep the approach as it is, despite the users have to call two functions to set the custom blend factors.

I reviewed it again and I got to the conclusion it adds an extra amount of duplicate code just to simplify a function that can be very easely implemented at user side with currently provided functions:
void rlSetBlendModeCustom(int glSrcFactor, int glDstFactor, int glEquation)
{
    rlSetBlendFactors(glSrcFactor, glDstFactor, glEquation);
    rlSetBlendMode(RL_BLEND_CUSTOM);
}

Similar function with separate factors can also be added at user side, no need to be part of rlgl.


chore(deps): upgrade dependencies

Upgrades project dependencies. See details in workflow run.

Automatically created by projen via the "upgrade-main" workflow

Superseded by https://github.com/cdklabs/construct-hub/pull/684


add a specific docs requirements, rtd yaml

fixing the failing build

note: I enabled the readthedocs build-on-pr setting.


No context menu through occluder

Fixes #1933
@ancientpower how's this
It's not 100% perfect because it pulls out entities within a 1-1 box within the click (the default) which means it can still technically go slightly through walls (if you click the edge and something is next to it) though I'm not sure of the best way to deal with that.
Could maybe have it check if any of the entities are walls and fallback to a tile-based method / center it on the wall's tile and call GetEntitiesIntersecting again?

Going slightly into walls isn't too big of a deal compared to being able to examine things that should be hidden behind 5 walls from across the screen, I'm sure a more elegant solution can be made later but this sounds like it works well enough.


add contributing docs on making a plugin [skip ci]

cc @rwaldron (thanks for helping me realize this)

Nice! 👍


Remove a Recurring Gift from Elevate Batch When Deleted from the Batch Gift Entry Table

Critical Changes
Changes
Issues Closed
Community Ideas Delivered
Features Intended for Future Release
Features for Elevate Customers
New Metadata
Deleted Metadata

Track W-11562653


پروکسی واتساپ64214800

Before reporting a bug, please see our FAQ in FAQ.md!
Description

Failed Step

More Details

Ok


Update -requirement-machine-* flags in various tests

Don't pass 'verify' since it's now the default
Update tests where diagnostics changed in a correct way to pass 'on' instead
Delete compiler_scale/explicit_requirements_perf.swift since it's not testing anything with the requirement machine

@swift-ci Please smoke test


bug(MatCheckbox): check/uncheck not working in QQ browser on iPhone

Reproduction
Steps to reproduce:

Open https://material.angular.io/components/checkbox/overview QQ browser on iPhone
try check/uncheck checkbox

Expected Behavior
checkbox should change view when check  /uncheck
Actual Behavior
checkbox view does not change
Environment

Angular:10.1.2
CDK/Material:10.2.1
Browser(s): QQ Browser <IP_ADDRESS>39
Operating System (e.g. Windows, macOS, Ubuntu): iPhone ios 12.4.8

https://apps.apple.com/ru/app/qq浏览器-热点新闻头条短视频抢先看/id370139302
https://www.dropbox.com/s/aj2685owjni4m1b/IMG_0582.PNG?dl=0
https://www.dropbox.com/s/r6w4uz7rc1uduq3/RPReplay_Final1603376785.MP4?dl=0

Unfortunately this isn't a browser we support.


Calculate Italian bban checksum code and fix Italian iban generation

Hi @alitaker,
I'm not quite sure why the result of the CI pipeline did not get propagated back to this pull-request (I'm gonna look into that), but there are several things that the automatic testing/format-checking did discover (https://travis-ci.org/github/mdomke/schwifty/builds/666876707). Can you please have a look at that?
Thanks for your contribution!

Do you think is ok now?

Do you think it's ok now?

@alitaker Even though this is already merged: Are you sure this implementation is actually correct? It looks a little different to this one https://github.com/globalcitizen/php-iban/blob/master/php-iban.php#L1241 Do you maybe have a resource that explains the used algorithm?

@mdomke I got the information from here:
http://alexandrerodichevski.chiappani.it/doc.php?n=218&lang=it#algo (sorry, it's in Italian).
They are very similar, the only larger difference is that the even digits just use the index for my implementation, while the link you're referring to uses an array of even indices, which are... the indices themselves! (from 1 to 28).
The same for the letters array, which are calculated using the ascii value in the implementation I wrote.
I find the version you linked more elengant, but both will work.
You may test them using the following website for free:
https://it.ibancalculator.com/iban_validieren.html


YAML is not recognized when using !Sub

When trying to use sam-local with the project https://github.com/taimos/serverless-todo-demo it fails with the following error:
ERROR:  did not find expected alphabetic or numeric character (line: 34; col: 0)

Line 34 uses the string substitution for the IAM permission to DynamoDB:
- !Sub arn:aws:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${TodoTable}

Yet another goformation bug. I'll take a look shortly

Fixed on goformation. Need to update this release.

Need a test case to validate fix before release. Is there one?

I tried with the project I mentioned above and ran sam local start-api or sam validate and both failed with the given error.

I got pass the initial failure of not being able to parse !Sub by converting a line like:
Prop: !Sub "my-sub-${RefValue}" throws an error: ☯ sam validate -t sam.yaml 2017/08/14 11:12:19 0.1.0 ERROR: Invalid indentation in template (line: 586; col: 0)
to
  - "my-sub-${RefValue}"
  - {}

and got this error:
2017/08/14 11:12:54 0.1.0
panic: interface conversion: interface {} is map[interface {}]interface {}, not string

goroutine 1 [running]:
github.com/awslabs/aws-sam-local/vendor/github.com/awslabs/goformation/resources.safeProcessStringArray(0x196b820, 0xc42079ad20, 0x167733c, 0x8, 0xc4201d1bd8)
	/home/ubuntu/repo/src/github.com/awslabs/aws-sam-local/vendor/github.com/awslabs/goformation/resources/safe-processing.go:46 +0x15f
github.com/awslabs/aws-sam-local/vendor/github.com/awslabs/goformation/resources.(*functionTemplate).Scaffold(0xc420286420, 0x196b860, 0xc420139200, 0x0, 0x0, 0xe, 0x168b7c4, 0x26, 0x0)
	/home/ubuntu/repo/src/github.com/awslabs/aws-sam-local/vendor/github.com/awslabs/goformation/resources/aws-serverless-function.go:146 +0x3bf
github.com/awslabs/aws-sam-local/vendor/github.com/awslabs/goformation/resources.(*awsServerlessFunction).Scaffold(0x19bcdc0, 0x196b860, 0xc420139200, 0x0, 0x0, 0x26, 0x0, 0x0, 0x0)
	/home/ubuntu/repo/src/github.com/awslabs/aws-sam-local/vendor/github.com/awslabs/goformation/resources/aws-serverless-function.go:80 +0x6a
github.com/awslabs/aws-sam-local/vendor/github.com/awslabs/goformation/resources.(*awsServerlessFunction).ClassConstructor(0x19bcdc0, 0x196b860, 0xc420139200, 0x0, 0x0, 0x0, 0x0)
	/home/ubuntu/repo/src/github.com/awslabs/aws-sam-local/vendor/github.com/awslabs/goformation/resources/aws-serverless-function.go:61 +0x55
github.com/awslabs/aws-sam-local/vendor/github.com/awslabs/goformation.postProcess(0x196e4e0, 0xc420132190, 0x196e4e0, 0xc420132190, 0x0, 0x0)
	/home/ubuntu/repo/src/github.com/awslabs/aws-sam-local/vendor/github.com/awslabs/goformation/post-process.go:46 +0x533
github.com/awslabs/aws-sam-local/vendor/github.com/awslabs/goformation.Parse(0xc420350000, 0x4efb, 0x7e00, 0x4efb, 0x7e00, 0x0, 0x0, 0xc4204a2028, 0x0)
	/home/ubuntu/repo/src/github.com/awslabs/aws-sam-local/vendor/github.com/awslabs/goformation/goformation.go:72 +0x407
github.com/awslabs/aws-sam-local/vendor/github.com/awslabs/goformation.read(0x1966360, 0xc4204a2028, 0xc4204a2028, 0x0, 0x0, 0x0, 0x1, 0x1)
	/home/ubuntu/repo/src/github.com/awslabs/aws-sam-local/vendor/github.com/awslabs/goformation/goformation.go:40 +0x105
github.com/awslabs/aws-sam-local/vendor/github.com/awslabs/goformation.Open(0x7fff5fbfec40, 0x8, 0x7fff5fbfec40, 0x8, 0x8, 0x0, 0x0, 0xc420316f90)
	/home/ubuntu/repo/src/github.com/awslabs/aws-sam-local/vendor/github.com/awslabs/goformation/goformation.go:29 +0x27e
main.validate(0xc4203ccf00)
	/home/ubuntu/repo/src/github.com/awslabs/aws-sam-local/validate.go:13 +0x7a
github.com/awslabs/aws-sam-local/vendor/github.com/codegangsta/cli.HandleAction(0x1587e60, 0x169a720, 0xc4203ccf00, 0xc4204a8800, 0x0)
	/home/ubuntu/repo/src/github.com/awslabs/aws-sam-local/vendor/github.com/codegangsta/cli/app.go:487 +0x7c
github.com/awslabs/aws-sam-local/vendor/github.com/codegangsta/cli.Command.Run(0x1677894, 0x8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x16971ee, 0xa7, 0x0, ...)
	/home/ubuntu/repo/src/github.com/awslabs/aws-sam-local/vendor/github.com/codegangsta/cli/command.go:207 +0xb72
github.com/awslabs/aws-sam-local/vendor/github.com/codegangsta/cli.(*App).Run(0xc42058e680, 0xc42007a080, 0x4, 0x4, 0x0, 0x0)
	/home/ubuntu/repo/src/github.com/awslabs/aws-sam-local/vendor/github.com/codegangsta/cli/app.go:250 +0x7d0
main.main()
	/home/ubuntu/repo/src/github.com/awslabs/aws-sam-local/main.go:302 +0x1be1

I am still getting this error :/


Warn against using the vagrant-vbguest plugin

Sometimes students install the vagrant-vbguest plugin that is supposed to automatically check whether VirtualBox Guest additions are up-to-date and to install an update if they aren't. However, the base boxes we use don't have the prerequisites (c-compiler, etc.) so this fails, resulting in the guest additions no longer working. Biggest problem with this is that the /vagrant directory is no longer mounted.

So, students should never install vagrant-vbguest and the assignment should warn against this.


Failed to publish entry for<EMAIL_ADDRESS>to the Bazel Central Registry.

No logs, no nothing. Not sure how to proceed without assistance.

Looks like this was caused by not having a version field in your MODULE.bazel, which caused parsing to fail.


[SwiftCompilerModules] Configured Package.swift use its own directory

Previously, it used ${CMAKE_CURRENT_BINARY_DIR} as the package directory. However, since that directory contains all build artifacts so opening it in Xcode shows those unnecessary files.
Instead of using ${CMAKE_CURRENT_BINARY_DIR}, create its own directory (SwiftCompilerCompilerModulesPackage/) in the root build directory.

@swift-ci Please smoke test

@swift-ci Please smoke test

@swift-ci Please smoke test Linux


Fix bad link in governance/ doc

Joining:

https://github.com/fluxcd/source-controller/pull/1200

Closing this issue (and the related issues linked above) will enable us to add testing of internal links in the future, to prevent these types of issues on website from regressing.

The bad link is currently visible here:
https://deploy-preview-1573--fluxcd.netlify.app/governance/#meetings

the public developer meeting.

I was getting link blindness, and this was the last of about 150 links I fixed yesterday... I didn't even think to look for it there 😂
Maybe we should just link directly to that, with an absolute link?

There, that will fix it just as well. No, I do think we should leave the README intact in the community repo
The goal of these PRs was only to fix links with minimal structural changes. If you find any links I've replaced in fluxcd/website#1573 or friends had a more direct analogue, more obvious intended target than whatever I've corrected the links to point at, let's use that instead.


Automated cherry pick of #8357: bugfix: qcloud pull metric

Cherry pick of #8357 on release/3.4.
#8357: bugfix: qcloud pull metric

Codecov Report

Merging #8364 into release/3.4 will decrease coverage by 2.67%.
The diff coverage is 0.00%.

@@              Coverage Diff               @@
##           release/3.4   #8364      +/-   ##
==============================================
- Coverage         6.22%   3.54%   -2.68%
==============================================
  Files              759     303     -456
  Lines           117145   62249   -54896
==============================================
- Hits              7289    2205    -5084
+ Misses          109197   59891   -49306
+ Partials           659     153     -506

Flag
Coverage Δ

#aFlag
3.54% <0.00%> (-2.68%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

pkg/appsrv/stats.go
0.00% <0.00%> (ø)

pkg/compute/models/buckets.go
1.10% <0.00%> (+0.03%)
:arrow_up:

pkg/compute/models/cachedimages.go
1.70% <0.00%> (-0.11%)
:arrow_down:

pkg/compute/models/capabilities.go
0.00% <0.00%> (ø)

pkg/compute/models/cloudaccounts.go
0.48% <0.00%> (-4.66%)
:arrow_down:

pkg/compute/models/dbinstance_skus.go
2.83% <0.00%> (-0.02%)
:arrow_down:

pkg/compute/models/dns_recordsets.go
1.82% <0.00%> (-0.24%)
:arrow_down:

pkg/compute/models/dns_zones.go
1.62% <0.00%> (-0.06%)
:arrow_down:

pkg/compute/models/elasticips.go
1.08% <0.00%> (-0.01%)
:arrow_down:

pkg/compute/models/guest_actions.go
0.00% <0.00%> (ø)

... and 449 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 41bdfff...db6175b. Read the comment docs.

/lgtm
/approve


exif.d.ts type error

interface EXIFStatic {
getData(img: string, callback: any): any;
getTag(img: any, tag: any): any;
getAllTags(img: any): any;
pretty(img: any): string;
readFromBinaryFile(file: any): any;
}
declare var EXIF : EXIFStatic;
export = EXIF;
but this is error, it should be:
interface EXIFStatic {
getData: (img: HTMLImageElement, callback: () => void) => void
getTag(img: any, tag: any): any;
getAllTags(img: any): any;
pretty(img: any): string;
readFromBinaryFile(file: any): any;
}
var EXIF : EXIFStatic
export { EXIF }

I'm having a similar issue - EXIF.getAllIptcTags is something I require, but I can't access it as it's not part of the interface - Property 'getAllIptcTags' does not exist on type 'EXIFStatic'.

here is need to add some generics for method getData
like getData<T = string>(img: T, callback: any): any;
now i use this lib in react project and type of payload is File
and i have error
TS2345: Argument of type 'File' is not assignable to parameter of type 'string'.


How do I run it?

Hey! This looks super cool :)
I'm unfamiliar with running this type of project. How exactly do I run it?

me too @bryan1anderson

Download Xcode. Open the Workspace file.

Then run it with the tool bar icon once its finished running

Confirming this runs the app and closing the issue :)


Bundle extension for performance

When publishing the extension, vsce tells me:

For performance reasons, you should bundle your extension: https://aka.ms/vscode-bundle-extension

Should look into this.

Completed in #5 and released as version 1.3.0.


mmdetection  AttributeError: 'ConfigDict' object has no attribute 'data'

Thanks for your error report and we appreciate it a lot.
Checklist

I have searched related issues but cannot get the expected help.
I have read the FAQ documentation but cannot get the expected help.
The bug has not been fixed in the latest version.

Describe the bug
我在mmdetection v2.28版本下，运行官网上更新的centernet_r50-caffe_fpn_ms-1x_coco.py以及centernet-update_r50_fpn_8xb8-amp-lsj-200e_coco.py，并在common文件夹中添加 lsj-coco-detecton的100e及200e文件。报错信息为
Traceback (most recent call last):
File "tools/train.py", line 242, in
main()
File "tools/train.py", line 135, in main
setup_multi_processes(cfg)
File "/home/mmdetection/mmdet/utils/setup_env.py", line 30, in setup_multi_processes
workers_per_gpu = cfg.data.get('workers_per_gpu', 1)
File "/home/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/mmcv/utils/config.py", line 519, in getattr
return getattr(self._cfg_dict, name)
File "/home/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/mmcv/utils/config.py", line 50, in getattr
raise ex
AttributeError: 'ConfigDict' object has no attribute 'data'
Reproduction
I ran  python tools/train.py configs/centernet/centernet-update_r50_fpn_8xb8-amp-lsj-200e_coco.py --work-dir work-dir/centernet-update_r50_fpn_8xb8-amp-lsj-200e_coco.py  on coco dataset.
Environment
sys.platform: linux
Python: 3.7.13 (default, Mar 29 2022, 02:18:16) [GCC 7.5.0]
CUDA available: True
GPU 0,1: NVIDIA GeForce RTX 3090
CUDA_HOME: :/usr/local/cuda-11.3:/usr/local/cuda-11.3
GCC: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
PyTorch: 1.12.0
PyTorch compiling details: PyTorch built with:

GCC 9.3
C++ Version: 201402
Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
OpenMP 201511 (a.k.a. OpenMP 4.5)
LAPACK is enabled (usually provided by MKL)
NNPACK is enabled
CPU capability usage: AVX2
CUDA Runtime 11.3
NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
CuDNN 8.3.2  (built against CUDA 11.5)
Magma 2.5.2
Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.3, CUDNN_VERSION=8.3.2, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.12.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF,

TorchVision: 0.13.0
OpenCV: 4.6.0
MMCV: 1.6.0
MMCV Compiler: GCC 9.3
MMCV CUDA Compiler: 11.3
MMDetection: 2.25.0+56e42e7

Hi, I guess the issue is related to mmcv version. Latest mmdetection uses train_dataloader keyword in configs instead of data keyword. Thus mmcv can't find data. Compare this and this. You may notice that there are more differences. Perhaps upgrading mmcv to version >=2.0 will help. The best way I guess is to build mmdet from scratch following the latest installation guide.

Hello
I have same error while using mmdetection. How you solved that?

@mahilaMoghadami hi, what are the versions you have for mmdet and mmcv. Check my response above. That should be a reason if you have the same error and environments.

thanks
yes I did, but Im using repository which is need to install mmcv==0.6.2!!!!
and now, I don't know what should I do

您好，mmdet3和mmdet2的注册机制好像有点不同，一个是@MODELS.register_module()一个是@HEADS.register_module()，请问你是怎么修改文件让他能成功运行的呢？


#3181 fix script error when pie chart and table chart contain alias

Description

fix script error when pie chart and table chart contain alias
Related Issue : #3181

How Has This Been Tested?

Go to a dashboard with 'Sales' datasource.
Create a text table chart.
Place Category and Sub-category on the second shelf and set the alias
and In the Common Setting, set the Text Style Aggregation Column to On (Horizonal).
Alias is applied.
Create a pie chart
Place Category and Profit on the shelf and set the alias.
See without error

Need additional checks?
Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[ ] I have read the CONTRIBUTING document.
[ ] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have added tests to cover my changes.

Additional Context

deploy to 2

@minhyun2
If you select "Original Data" when drawing a table chart with sales data, the following console error will occur.

deploy to 2

@minhyun2 We confirmed normal operation regard to "Alias" on the shelf. I don't know if it's related to this issue, but if I set Alias in the data source itself, it should look like this:

@kyungtaak The alias setting of the datasource itself is not related to this issue. However, we will find ways to improve it by managing it with new issues.


VITE: Bundler switch from Webpack to Vite

Switch bundler from webpack to vite

Significantly faster for development, production builds, and testing.

@jamestalton I'd advise against this change as it introduces a new tool that is not as well known across teams as Webpack. Also I'm not sure it's not gonna introduce more problems with using the framework on c.r.c Maybe it's worth more attention from other teams to assess the risks?
cc @h-kataria

@jamestalton has any other team tried to switch to Vite yet from webpack, want to know if they ran into any issues or had to make a significant amount of changes to support the new tool.

My concern here would be the insights platform support. frontend-components-config is based on webpack, and won't work without it.

Holding off on this PR until we can discuss it more as a platform team.


GoReleaser and Homebrew tap

Implement goreleaser and create homebrew tap.

@estahn This repository has a history of welcoming pull requests.


SetRenderUsingCustomAction only works for 5 pins.

I was trying to change the icons of the clustered pins by using the SetRenderUsingCustomAction method of the ClusteredMap.ClusterOptions.
In the following code the s parameter is always 5 and it's being triggered only once.
var options = new ClusterOptions();

options.SetRenderUsingCustomAction((s =>
{
	if (s == "5")
		return BitmapDescriptorFactory.FromBundle("Cluster5.png");
	else if (s == "10")
		return BitmapDescriptorFactory.FromBundle("Cluster10.png");

	return BitmapDescriptorFactory.FromBundle("ClusterAny.png");
}));

I was expecting to be called for all the Buckets (which I haven't changed)
Only happens in Android.

@gnout I tested it and it's called only for buckets numbers (by default 10, 50, 100, 200, 1000) and the first bucket that contains less than 10 pins. I agree that a callback should be called for all buckets.

In the following code the s parameter is always 5 and it's being triggered only once.

I tested it and it was called when a bucket was rendered. How many clusters do you have?

@mierzynskim I have buckets up to 200. Which means 10, 50 etc.
Also I have buckets with less than 10, 4, 5, 6.
In iOS works fine but in Android it's called only once and always the s parameter is 5. As a result all the clustered pins have the image that I specified for 5 (even if the cluster contains 200 pins)

@mierzynskim I'm working with version 0.3.2

@gnout Interesting... When I test it, a callback is called every time I zoom in/out and a new cluster appears. It's intended behaviour besides the fact it's not called for clusters less than five which is clearly a bug. Can you share a repro project?

@mierzynskim I'm currently working on a project (that includes your library) and we will deploy early next week. After that I will probably be able to create an example.

@gnout What Android version do you use? Maybe it depends on it because the issue is not present when I test it on my device (API 26)

@mierzynskim I think that you're right on this. It works on a device that I have with Android 7.1 but it doesn't work on the device (that I use for development) with Android 8.0

@mierzynskim setting custom images to clusters seems to work on Android now (!?) I'm not sure if you made some changes to NuGet. I was working all Friday with the clustered map and I wasn't able to set specific images to the clusters. This morning, custom cluster images are working as expected!

@mierzynskim SetRenderUsingCustomAction is not working properly while debugging which is not a big of a problem but you might wan't to take a look at it.

@gnout I'm glad it finally worked for you. I haven't made any changes on NuGet. I only noticed that there are some problems with setting icons for clusters < 10. To be more precise, it wasn't possible to set different images for clusters 5, 6, 7, 8, 9. Have you experienced any thing like this?

@mierzynskim Almost correct. I have the following segments
<10
>=10 && <20
>= 20 && < 50

etc.
The segment for <10 is working but I don't have specific segment for 5,6,7,8 and 9.
Then there is gap for the other 2 segments, so I never see the >=10 && <20 nor the >= 20 && < 50
Segments from 50+ and above works correctly (i.d 100+, 200+, 500+ although I don;t currently have more than 500 pins.

@mierzynskim I think that I solved the issue. I saw that in your code the ResetBuckets doesn't specify the segments that you mentioned.
I specify my own segments by using SetBuckets to 5,10,20,50,100,200 and 500 and everything worked as expected.

@gnout Thanks for checking this but if you change minimum bucket size to 10, the issue will appear. The good news is that the fix is almost ready. I'll try to look at the issue with SetRenderUsingCustomAction not working in debug mode here.


rancher-backup fails to install from UI - error in s3-secret.yaml template

What kind of request is this (question/bug/enhancement/feature request):
bug
Steps to reproduce (least amount of steps as possible):

install rancher-backup-operator from Apps in 2.5
choose S3 as the default storage provider

Result:
Error: YAML parse error on rancher-backup/templates/s3-secret.yaml: error converting YAML to JSON: yaml: line 23: could not find expected ':'
Other details that may be helpful:

using minio as endpoint
secret configured and selected
this seems to have something to do with folder in the template - tried both with and without a folder. same error.

Environment information

Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI): 2.5.3
Installation option (single install/HA): k3s

Cluster information

Cluster type (Hosted/Infrastructure Provider/Custom/Imported): k3s
Machine type (cloud/VM/metal) and specifications (CPU/memory): KVM
Kubernetes version (use kubectl version): 1.19.4+k3s1

(paste the output here)

Docker version (use docker version):

(paste the output here)

It looks like we should allow multi-line in the chart for the certificate input: https://github.com/rancher/charts/blob/dev-v2.5/charts/rancher-backup/templates/s3-secret.yaml#L25
When the enpointCA is added, the yaml doesn't render correctly.


CODE_OF_CONDUCT

Resolves #312
Although this is a draft...still open to changes...it could be more personalized/customized to swarmauri codebase/community  as it's a bit generic

this is a good start for a discussion on documentation that we are missing. thank you very much for your submission. we'll merge your change onto it's own feature branch so we can continue a discussion regarding documentation in more detail


Feature request: commands output to be captured to log file

Please develop feature for commands' output to be captured to log file, ideally such a log should be shipped to logging endpoint to be protected from deletion.
This is required in my use case to ensure traceability of commands run by operator and to eliminate repeatable command's running. Log shipping may be achieved by some telemetry collector for logs, unless there is control over the place log is being written.
Also such a feature should help securing commands output but not sending this output to 3rd party service (as it may contain sensitive data).
feature discussed during SRECon23 in Dublin

Thanks for filing the feature request @serafin-tech! I have taken notes of our conversation at SRECon 2023 in Dublin. Will circle back with a plan shortly.

Hi @sourishkrout
sorry for long response time
re 1. it really depends on the use-case, but given the fact I'd like to protect from repetitive command executions I would save history of all runs. On the other hand I can imagine scenario that one is interested only in recent output, so maybe a feature-flag for that might be solution.
re 2. I would say having the output in markdown document would be one of the solutions. One can attach it to the ticket/change control as a proof of work. What might be needed in such case is some protection from overwriting vanilla runbook with copy containing output of the commands. Probably suggestion to write it under new name would be enough.
re 3. That's a good question. In the use-case I'm thinking about opening IDE with runbook would mean opening new session. Probably retaining the session between two openings of runbook in IDE would be valid but I wouldn't invest time in it now and keep it as separate feature.
re 4. IMO critical are timestamps from executions. But what might be good to record IMO is sha of last commit for the runbook, to record the version used by operator. Of course it will no guarantee runbook had not been modified but at least give a hint if operator used recent version while dong the job.
I wouldn't add more features now not to over-complicate your work. I'd really like to see outcome to start experimenting.
And again - appreciate your work on the tool

@serafin-tech I agree that starting simple and providing feedback iteratively is likely the better path than trying to anticipate all the corner cases. Thank you for the feedback; this will help us get started on the first version for review.
will keep you posted on the progress.

Here's some work in progress, @serafin-tech. However, instead of writing the output (plus runtime info) back to the original markdown file in an initial version, Runme will write a full copy per session (enabled via feature flag). Shooting for an opt-in release in the next 1-2 weeks.

stdio.md
stdio-cl3dopjjtojcs1tcq150.md

We are making good headway. Remaining step to a first version is to write full copy of the markdown/notebook file including captured outputs to a separate file.

We just shipped Runme v2.1.4 with a feature-flagged experiment to save Runme notebook outputs to a separate session file.
It's easy to turn on in Settings (CMD + , on Mac) under @ext:stateful.runme runme.experiments.outputPersistence. Check the box and restart.
Here's an example. Noticed the "Session Outputs" button which will open the corresponding file to the side in markdown preview.

Hi @sourishkrout,
I've tested this new feature and it looks very promising. Output is saved and secured as you promised.
re your comments:

The "session file with output" may be easily reused to restore previous session. The challenge may be environment variables, but I would leave solving it for runbook developer as for now. Additional data in frontmatter may be helpful here,
Saving output needs to be reviewed. Now I see the save is done when next command block is being executed - this is not acceptable. If this is to be used for execution tracking save have to be done immediately after block execution is completed. I discovered this gap when I lost one output...

And thanks a lot for Your work on this software
Ser@fin

@sourishkrout,
I noticed one more thing - in my test runbook I have a command which does not emit any output (simple curl) thus the execution capture is empty. Would you be able to record process return code in such cases?
TIA

@sourishkrout,
I noticed one more thing - in my test runbook I have a command which does not emit any output (simple curl) thus the execution capture is empty. Would you be able to record process return code in such cases?
TIA

I can take a look into it. Do you have an example shell snippet for such a curl that doesn’t produce an output (just in case). I’ll try reproducing it tmrw and will get back regarding your other comments.

@sourishkrout,
I noticed one more thing - in my test runbook I have a command which does not emit any output (simple curl) thus the execution capture is empty. Would you be able to record process return code in such cases?
TIA

I can take a look into it. Do you have an example shell snippet for such a curl that doesn’t produce an output (just in case). I’ll try reproducing it tmrw and will get back regarding your other comments.

nothing sophisticated: curl -o file.name https://url.to.be/downloaded

@sourishkrout,
I noticed one more thing - in my test runbook I have a command which does not emit any output (simple curl) thus the execution capture is empty. Would you be able to record process return code in such cases?
TIA

I can take a look into it. Do you have an example shell snippet for such a curl that doesn’t produce an output (just in case). I’ll try reproducing it tmrw and will get back regarding your other comments.

nothing sophisticated: curl -s -o file.name https://url.to.be/downloaded

Repro'd it adding the slient flag -s. Fixed the issue. Runme will now write empty outputs as long as they have a valid "execution summary". We get this out in the next release in the next few days, @serafin-tech.

Shipped: https://runme.dev/blog/runme-v3-pipeline-logs-and-artifacts


Update blob-backup-support-matrix.md

Removing the preview text as the feature is not in Preview anymore Please review

Learn Build status updates of commit 39775d4:
:warning: Validation status: warnings

File
Status
Preview URL
Details

articles/machine-learning/v1/how-to-log-view-metrics.md
:warning:Warning

Details

articles/backup/blob-backup-support-matrix.md
:white_check_mark:Succeeded

articles/machine-learning/v1/how-to-log-view-metrics.md

Line 182, Column 4: [Warning: codesnippet-not-found] Invalid code snippet link: '~/MachineLearningNotebooks/how-to-use-azureml/training/train-on-local/train.py'.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Learn Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings or suggestions unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the learn.microsoft.com contributor guidesPost your question in the Learn support channel

Closing/reopening PR to clear the Validation warning.

Learn Build status updates of commit 39775d4:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

articles/backup/blob-backup-support-matrix.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the learn.microsoft.com contributor guidesPost your question in the Learn support channel

@v-amallick
Can you review the proposed changes? IMPORTANT: When the changes are ready for publication, add a #sign-off comment to signal that the PR is ready for the review team to merge.
#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

#sign-off


Expose raw http and https servers

Resolves #31

Coverage remained the same at 100.0% when pulling 7af85b7a32e2b0e71c55380d6578fbfa6af22c5e on expose-servers into a730cff77d970a918a01cfa7600b800ef3e2f8a0 on master.

Coverage remained the same at 100.0% when pulling 7af85b7a32e2b0e71c55380d6578fbfa6af22c5e on expose-servers into a730cff77d970a918a01cfa7600b800ef3e2f8a0 on master.


Error in signature calculation

I ccompared with that web site :
http://oauth.googlecode.com/svn/code/javascript/example/signature.html
the signature calculated in OAuthSwiftClient, authorizationHeaderForMethod give wrong value.
this imply that on some web site, upload fail.
here is something I did, which works.
hope this help.
Olivier
// convert a Dictionary, in array of string, with escaped string
func getEscapedArrayParam(parameters: Dictionary<String, AnyObject>) -> [String] {
    // escape all oauth parameter
    var encodedParam = [String]()
    for(k, v) in parameters {
        let str = k + "=" + (v as! String)
        let escapedStr = str.stringByAddingPercentEncodingWithAllowedCharacters(.URLHostAllowedCharacterSet())
        encodedParam.append(escapedStr!)
    }
    // sort the result
    encodedParam.sortInPlace{ $0 < $1 }
    return encodedParam
}

// do almost like stringByAddingPercentEncodingWithAllowedCharacters, but take in account more character to be changed in %xyz
//!*'();:@&=+$,/?%#[]
func escapeString(str: String) -> String {
    let encoding: NSStringEncoding = NSUTF8StringEncoding
    // !*'();:@&=+$,/?%#[]
    let charactersToBeEscaped = ":/?&=;+!@#$()',*" as CFStringRef
    let charactersToLeaveUnescaped = "[]." as CFStringRef
    let raw: NSString = str
    let result = CFURLCreateStringByAddingPercentEscapes(kCFAllocatorDefault, raw, charactersToLeaveUnescaped, charactersToBeEscaped, CFStringConvertNSStringEncodingToEncoding(encoding))
    return result as String
}

public func encodeParameters(method: String, url: NSURL, parameters: Dictionary<String, AnyObject>, credential: OAuthSwiftCredential) -> String {

    // algorithm used : https://dev.twitter.com/oauth/overview/creating-signatures
    // will add the oauth_signature parameter

    //define the oauth parameters
    var authorizationParameters = Dictionary<String, AnyObject>()
    authorizationParameters["oauth_version"] = OAuth.version
    authorizationParameters["oauth_consumer_key"] = credential.consumer_key
    authorizationParameters["oauth_timestamp"] = String(Int64(NSDate().timeIntervalSince1970))
    authorizationParameters["oauth_nonce"] = (NSUUID().UUIDString as NSString).substringToIndex(8)
    authorizationParameters["oauth_signature_method"] =  OAuth.signatureMethod
    // add token is it exist
    if (credential.oauth_token != ""){
        authorizationParameters["oauth_token"] = credential.oauth_token
    }
    // add additionnal oauth (optionnal) parameters if not already existing
    // example, oauth_callback, defined when requesting the token
    for (key, value) in parameters {
        if key.hasPrefix("oauth_") {
            authorizationParameters.updateValue(value, forKey: key)
        }
    }

    // escape all oauth parameter
    let encodedParam = getEscapedArrayParam(authorizationParameters)
    // convert it into a string, each param separated by &
    var outputStr:String = ""
    for v in encodedParam {
        outputStr += v + "&"
    }
    // remove last "&"
    outputStr.removeAtIndex(outputStr.endIndex.predecessor())
    // percent encode the oauth sorted, appened, parameters
    let percentOutput = self.escapeString(outputStr)

    // build the signature base string
    let urlPercented = self.escapeString(String(url)) //  String(url).stringByAddingPercentEncodingWithAllowedCharacters(.URLHostAllowedCharacterSet())
    var signBaseString:String = method + "&" + urlPercented
    signBaseString += "&" + percentOutput

    // build the signin key
    let signingKey = self.escapeString( credential.consumer_secret) + "&" + self.escapeString(credential.oauth_token_secret)

    // Build the signature
    let sha1 = HMAC.sha1(key: signingKey.dataUsingEncoding(NSUTF8StringEncoding)!, message: signBaseString.dataUsingEncoding(NSUTF8StringEncoding)!)!
    let oauth_signature = sha1.base64EncodedStringWithOptions([])
    authorizationParameters.updateValue(self.escapeString( oauth_signature) , forKey: "oauth_signature")

    // add the signature to the parameters
    //encodedParam.append("oauth_signature=" + oauth_signature )
    //encodedParam.sortInPlace{ $0 < $1 } // not sure it is useful r not

    // create an array, with escape before =
    var headerComponents = [String]()
    for (key, value) in authorizationParameters {
        headerComponents.append("\(key)=\"\(value)\"")
    }

    let finalUrl:String =  "OAuth " + headerComponents.joinWithSeparator(", ")
    return finalUrl
}

make a fork, a branch and commit your code into, then it's easy to make a diff online
here it's difficult to see what you have done and where

I'm seeing this issue too. olivier38070 fix didn't work for me

@App8ite if you try olivier fix, could you commit into your fork? (into a branch)

sorry to have been lazy. I should have committed my stuff, and forgot it
(too overloaded in my job)
I will try to do it soon, I have to get my source back.
2015-11-11 10:01 GMT+01:00 Phi Mage<EMAIL_ADDRESS>
@App8ite https://github.com/App8ite if you try olivier fix, could you
commit into your fork? (into a branch)
—
Reply to this email directly or view it on GitHub
https://github.com/dongri/OAuthSwift/issues/115#issuecomment-155709144.

Is there a test case that can demonstrate what's wrong? I am curious if #141 fixed this. There have been some changes in this area and I'd like to know if it's still open. If you can describe the case, I'll test.

added unit test about signature 4ec66e4fd1d3133e75a32880f655b0f17016d4a2 (based on google unit test)
to fix potential issues or label this issue as bug, I need an oauth data set (URL,parameters,consumer key,consumer secret,token,token secret,timestamp,nonce)

When I done a POST to twitter api, it responsed a message 'Could not authenticate you.'. The same issue?

@fewspider thank you to look for a potential identical issue, but for this issue I wait for a sample code with problem (api, version etc...) from @olivier38070
Some signature problem has been fixed last month
You can open a new issue with the version used of OAuthSwift, the code, parameters, etc.. and I will test

@phimage ok,wait a minute

@phimage done

Hi,
sorry, I was too busy by my job.
now, I updated the code, and it seems that it has evolve a lot.
I Will have difficulties to commit anything, as my code also is a fork from
an old version, with some modification to work.
what do you suggest I do ?
I could try to get the latest version, remove all my code in my application
and check what if it is working...
(I remind that on my side, I could login on smugmug, but I couldn't
send/upload any images)
here is what gave me trouble :
the signature calculation gave me error, in smugmug, when trying to do
upload.
the code bellow (Part A) is replacing this :
public func signatureForMethod(method: OAuthSwiftHTTPRequest.Method,
url: NSURL, parameters: Dictionary<String, AnyObject>) -> String {
}
located in OauthSwiftCredential.swift
Also, I had trouble with the encoding of the request, when sending datas :
Part A :
// convert a Dictionary, in array of string, with escaped string
func getEscapedArrayParam(parameters: Dictionary<String, AnyObject>) ->
[String] {
// escape all oauth parameter
var encodedParam = String
for(k, v) in parameters {
let str = k + "=" + (v as! String)
let escapedStr =
str.stringByAddingPercentEncodingWithAllowedCharacters(.URLHostAllowedCharacterSet())
encodedParam.append(escapedStr!)
}
// sort the result
encodedParam.sortInPlace{ $0 < $1 }
return encodedParam
}
// do almost like stringByAddingPercentEncodingWithAllowedCharacters,

but take in account more character to be changed in %xyz
//!'();:@&=+$,/?%#[]
func escapeString(str: String) -> String {
let encoding: NSStringEncoding = NSUTF8StringEncoding
// !'();:@&=+$,/?%#[]
let charactersToBeEscaped = ":/?&=;+!@#$()',*" as CFStringRef
let charactersToLeaveUnescaped = "[]." as CFStringRef
let raw: NSString = str
let result =
CFURLCreateStringByAddingPercentEscapes(kCFAllocatorDefault, raw,
charactersToLeaveUnescaped, charactersToBeEscaped,
CFStringConvertNSStringEncodingToEncoding(encoding))
return result as String
}
public func encodeParameters(method: String, url: NSURL, parameters:

Dictionary<String, AnyObject>, credential: OAuthSwiftCredential) -> String {
    // algorithm used :

https://dev.twitter.com/oauth/overview/creating-signatures
// will add the oauth_signature parameter
    //define the oauth parameters
    var authorizationParameters = Dictionary<String, AnyObject>()
    authorizationParameters["oauth_version"] = OAuth.version
    authorizationParameters["oauth_consumer_key"] =

credential.consumer_key
authorizationParameters["oauth_timestamp"] =
String(Int64(NSDate().timeIntervalSince1970))
authorizationParameters["oauth_nonce"] = (NSUUID().UUIDString as
NSString).substringToIndex(8)
authorizationParameters["oauth_signature_method"] =
OAuth.signatureMethod
// add token is it exist
if (credential.oauth_token != ""){
authorizationParameters["oauth_token"] = credential.oauth_token
}
// add additionnal oauth (optionnal) parameters if not already
existing
// example, oauth_callback, defined when requesting the token
for (key, value) in parameters {
if key.hasPrefix("oauth_") {
authorizationParameters.updateValue(value, forKey: key)
}
}
    // escape all oauth parameter
    let encodedParam = getEscapedArrayParam(authorizationParameters)
    // convert it into a string, each param separated by &
    var outputStr:String = ""
    for v in encodedParam {
        outputStr += v + "&"
    }
    // remove last "&"
    outputStr.removeAtIndex(outputStr.endIndex.predecessor())
    // percent encode the oauth sorted, appened, parameters
    let percentOutput = self.escapeString(outputStr)

    // build the signature base string
    let urlPercented = self.escapeString(String(url)) //

String(url).stringByAddingPercentEncodingWithAllowedCharacters(.URLHostAllowedCharacterSet())
var signBaseString:String = method + "&" + urlPercented
signBaseString += "&" + percentOutput
    // build the signin key
    let signingKey = self.escapeString( credential.consumer_secret) +

"&" + self.escapeString(credential.oauth_token_secret)
    // Build the signature
    let sha1 = HMAC.sha1(key:

signingKey.dataUsingEncoding(NSUTF8StringEncoding)!, message:
signBaseString.dataUsingEncoding(NSUTF8StringEncoding)!)!
let oauth_signature = sha1.base64EncodedStringWithOptions([])
authorizationParameters.updateValue(self.escapeString(
oauth_signature) , forKey: "oauth_signature")
    // add the signature to the parameters
    //encodedParam.append("oauth_signature=" + oauth_signature )
    //encodedParam.sortInPlace{ $0 < $1 } // not sure it is useful r not

    // create an array, with escape before =
    let newrev = authorizationParameters.sort(){ $0.0 < $1.0 }

    var headerComponents = [String]()
    for (key, value) in newrev {
        headerComponents.append("\(key)=\"\(value)\"")
    }

    let finalUrl:String =  "OAuth " +

headerComponents.joinWithSeparator(",") //
headerComponents.joinWithSeparator(", ")
return finalUrl
}
PArt B:
public func makeRequest(urlString: String, method:
OAuthSwiftHTTPRequest.Method, parameters: [String: AnyObject] = [:],
headers: [String:String]? = nil) -> OAuthSwiftHTTPRequest? {
    let url: String = urlString
    let method: String = "POST"

if let url = NSURL(string: urlString) {
let request = OAuthSwiftHTTPRequest(URL: url, method: method,
parameters: parameters)
if self.credential.oauth2 {
request.headers = ["Authorization": "Bearer
(self.credential.oauth_token)"]
} else {
//let str = ["Authorization": self.encodeParameters(method,
url: url, parameters: parameters, credential: self.credential)]
//request.headers == str
//                var str = ["Authorization":
OAuthSwiftClient.authorizationHeaderForMethod(method, url: url, parameters:
parameters, credential: self.credential)]
var str2 = ["Authorization": self.encodeParameters(method,
url: url, parameters: parameters, credential: self.credential)]
request.headers =  str2
}
request.successHandler = success
request.failureHandler = failure
request.dataEncoding = dataEncoding
request.encodeParameters = true
        request.HTTPBody = image //

"(image)".dataUsingEncoding(NSUTF8StringEncoding)
request.contentType = "image/jpeg" //type
request.start()
    }

}

in OAuthSwift.swift
in makeRequest (it is now setupRequestForOAuth) :
      if (body != nil && contentType != nil) {

            for (key, value) in nonOAuthParameters {
                request.setValue(value as! String, forHTTPHeaderField:

key)
}
            request.setValue(contentType!, forHTTPHeaderField:

"Content-Type")
//request.setValue(body!.length.description,
forHTTPHeaderField: "Content-Length")
request.HTTPBody = body!
} else {
which replace this :
if let b = body {
request.HTTPBody = b
} else {
the other part of the code are different, but I believe that it is refactor
on your side, as I don't rememmber having updated so much that part.
smugmug need some parameter in the header, even for the upload, if I
remember well what was wrong.
Hope this help.
Olivier
2015-12-13 17:37 GMT+01:00 You shall not pass<EMAIL_ADDRESS>
@fewspider https://github.com/fewspider thank you to look for a
potential identical issue, but for this issue I wait for a sample code with
problem (api, version etc...) from @olivier38070
https://github.com/olivier38070
Some signature problem has been fixed last month
You can open a new issue with the version used of OAuthSwift, the code,
parameters, etc.. and I will test
—
Reply to this email directly or view it on GitHub
https://github.com/OAuthSwift/OAuthSwift/issues/115#issuecomment-164276216
.

@olivier38070  If you make a fork with your code, and say the provider(smugmug) and the method(post)
This is enough for me, no need to try to pull request, yes there is too many change

ok, I tried to commit and push, but I have an error with githib.
I don't see what I can do, as other githib are working fine
"You may not have permission to access OAuthSwift. Check Preferences to
make sure you’re still logged in."
2015-12-14 9:54 GMT+01:00 You shall not pass<EMAIL_ADDRESS>
@olivier38070 https://github.com/olivier38070 If you make a fork with
your code, and say the provider(smugmug) and the method(post)
This is enough for me, no need to try to pull request, yes there is too
many change
—
Reply to this email directly or view it on GitHub
https://github.com/OAuthSwift/OAuthSwift/issues/115#issuecomment-164381220
.

@olivier38070 are you trying to commit to this repo OAuthSwift/OAuthSwift, not your fork maybe?
I think you must not have the repo with last commit, go to the commit where you begin
for instance start with https://github.com/olivier38070/OAuthSwift/commit/c6cea89e66c101dccd20dced608b7f9625d36c36

I tried using sourcetree (which I know) instead of github desktop tool
provided by github....
seems to work.
2015-12-14 12:05 GMT+01:00 You shall not pass<EMAIL_ADDRESS>
@olivier38070 https://github.com/olivier38070 are you trying to commit
to this repo OAuthSwift/OAuthSwift, not your fork maybe?
I think you must not have the repo with last commit, go to the commit
where you begin
for instance start with olivier38070@c6cea89
https://github.com/olivier38070/OAuthSwift/commit/c6cea89e66c101dccd20dced608b7f9625d36c36
—
Reply to this email directly or view it on GitHub
https://github.com/OAuthSwift/OAuthSwift/issues/115#issuecomment-164410464
.

I was having some trouble with the Goodreads api and #168 didn't fix it, but this did. Is anything still missing for it to be merged? Maybe I could help.

@danielbmarques if I understand just adding bracket ( PR #168 for issue #174) do not fix your issues with Goodreads.
You take some code from oliver? you can make a PR if you can, I will read it for sure (here I ask to make a clean one, by merging code from OAuthSwift master)

Well, it seems I spoke too soon, this didn't actually fix my problem after all.
What I did was to replace my urlEncodedStringWithEncoding with the one in #165. Before I did that the POST request I was trying to make never worked. After I changed the code for Oliver's the request worked and I made my comment here, but now I found out it only works some of the time. Which is very weird… the string that gets signed is exactly the same except for the nonce and the timestamp parameters. Yet it works some of the time, and some of the time it doesn't.
The problem doesn't seem to be with Goodreads either, since the call works in their app (I think they use some objective C library).
I'll keep working on this and make a PR if I can fix it.
@phimage, if you have any ideas about how to debug this, I'm kind of clueless right now, and I don't know much about OAuth.
One thing that I noticed and might be worth mentioning is that the urlEncodedStringWithEncoding() gets called multiple times on the same strings. It gets called, for instance, for each parameter in the urlEncodedQueryStringWithEncoding function, and then those same parameters are passed to that function again when they're part of the whole url. If I remove the urlEncodedStringWithEncoding() calls from urlEncodedQueryStringWithEncoding, the resulting signatureBaseString is often exactly the same as it would be if the function was called, but even if it's the same, my api calls won't work (any of them whether they contain brackets or not). So it seems the encoding function interferes with more than just the signatureBaseString, but I only see it called from there (from signatureForMethod()). So the encoding changes something besides the signatureBaseString, but I can't figure out what, and I think that's where the problem is. Any ideas about this?
Thanks for the help and for the great work here.

I think I'm making some progress... #165 did solve a part of my problem, but I can't test it properly because of a different problem (I think). So I'll post another issue here describing what I found.

@phimage Do OAuthSwift requests send the Content-Length header? If they do, I couldn't find it in the code. And that header should be set, right?
There's a Goodreads forum post where their programmer says to "make sure the 'Content-Length' HTTP header is set to the actual post data length". So I'm thinking this is what could be causing my problem, couldn't it?
I thought of that because I started comparing the responses to my POST requests that work to the responses to those that don't work…
This is the response of a request that works:
<NSHTTPURLResponse: 0x7fa5f4042240> { URL: http://www.goodreads.com/user_status } { status code: 201, headers {
    "Cache-Control" = "max-age=0, private, must-revalidate";
    "Content-Encoding" = gzip;
    "Content-Length" = 302;
    "Content-Type" = "application/xml; charset=utf-8";

And this is the response of a request that doesn't (it does nothing even if it returns 200 – it should return 201):
<NSHTTPURLResponse: 0x7fa5f1ef52d0> { URL: https://www.goodreads.com/user_status } { status code: 200, headers {
    "Cache-Control" = "max-age=0, private, must-revalidate";
    "Content-Encoding" = gzip;
    "Content-Type" = "text/html; charset=utf-8";
    Date = "Sat, 05 Mar 2016 22:52:13 GMT";

I don't know if the fact that a Content-Length header is included in that response means it was included in the request, maybe it has nothing to do with it, but anyway, I thought it might be worth checking, and it doesn't seem like a Content-Length is set to most requests (is it set for some of them, though? could it have been set automatically or something for those that work?).

Well, I tried adding a Content-Length to the request myself, I don't know if I did it right, but it didn't fix my problem.
I don't know what could be causing it. :(
I'm comparing a request that works to a request that doesn't and the signatureBaseStrings are almost the same and the Authorization header too. The only difference in the header is the oauth_signature, and its characters change every time, and some of them are url encoded, so I think that change could account for the request working sometimes and not working other times, but comparing the signatures I can't fine anything wrong.
For instance, this is the oauth_signature of a request that works: "YkxSeZ65dMt7BknO2Y49JY4pnPQ%3D"
And these are signatures of requests that don't work: "n0P157G7EgHn%2BXemhbub2vpQPOU%3D", "V9U6IoENKngy3OviZNjUDSOWPis%3D", "9TWg3r2sGiL8aGAWrZEG%2F%2Fdr0jY%3D".
At the same time the only request that doesn't work is the one that uses brackets in the parameters, but I don't know what any of this has to do with the brackets… and fixing the urlEncodedStringWithEncoding function only made it work sometimes. :/

I've had similar problems with the signatures not matching.  In my case I'm using this on an iPhone app to connect to my own rails application.  From what I can tell I've found two separate issues.
The first problem I had was when sent a parameter in a POST or PATCH body that had a newline character \n.  I was getting an incorrect signature and the request would fail.  #165 fixed that problem for me.
The second problem is when sending a request with square brackets [].  Those requests were always generating bad signatures for me.  #168 did not fix that issue for me, and also still included the problem with the newline characters.
Since I have control over both ends of the API, I ended up just going with #165 and not sending any parameters with square brackets.  Square bracket parameters in rails are pretty much everywhere, so I'm not entirely happy with that solution, but it let me get something out the door for now.
For what it's worth, I did try checking things with the nouncer signature calculator, but I ended up getting 3 different signatures - one from that, one from this OAuthSwift code, and a third from my rails app using the simple_oauth gem.  At that point I figured it wasn't worth trying to track down the proper fix since I had no idea who was wrong.  Although I'd be inclined to believe that the rails gem is properly handling square brackets, since they're everywhere in rails and someone else would have discovered that issue over there by now.
Anyway, seems to me like the right way to proceed here would be to merge #165 and then see about a separate fix for #174 based off that.

Ok, I fixed other problems in my code and now it works with both #165 and #168.
It seems to me like the main difference between those two is that in #165 some of the excluded characters aren't mentioned explicitly: they're taken from URLQueryAllowedCharacterSet.
In both cases it's not clear what the allowed characters are. #165 uses URLQueryAllowedCharacterSet, whose contents aren't immediately clear, and #168 (as the original code) uses an invertedSet, which makes clear what characters are excluded, but not which characters are allowed.
The OAuth 1.0 specification, on the other hand, is very clear about what is allowed:

Characters in the unreserved character set as defined by [RFC3986], Section 2.3 (ALPHA, DIGIT, "-", ".", "_", "~") MUST NOT be encoded.
All other characters MUST be encoded.

So I think it would be a lot better to explicitly state the allowed characters.
I modified my code to do just that:
func urlEncodedStringWithEncoding(encoding: NSStringEncoding) -> String {
    let originalString: NSString = self
    let customAllowedSet =  NSCharacterSet(charactersInString:"abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789-._~")
    let escapedString = originalString.stringByAddingPercentEncodingWithAllowedCharacters(customAllowedSet)
    return escapedString! as String
}

Since the OAuth specification is so clear, I think this is pretty safe to do, and it works perfectly for me.
@phimage, should I make a PR with that change?

@danielbmarques I rebuilt my app using your changes - the newline problem didn't happen but I still had issues with the square brackets.  That leads me to believe that the problem is most likely on my end, not in this OAuthSwift code.
I'm in favor of your solution though since it matches the spec.  I'll post back here if I ever track down the issue on my end.

@danielbmarques thanks you to point out the info from RFC
I have just a doubt because one day I make a comment about information from RFC and the difference in OAuthSwift implemention. (Maybe two method must be created, one for the signature, one other to encode elsewhere, and maybe that"s why there is double encoding sometimes...)
I will look into PR and issues for this comment

@phimage, oh, it seems you're right. That info is actually from the OAuth 1.0 specification (from what I understand it's a little more strict than RFC3986), and it refers to both the signature base string and the encoding of parameters for the header.
It does add a caveat, though:

This method is different from the encoding scheme used by the "application/x-www-form-urlencoded" content-type (for example, it encodes space characters as "%20" and not using the "+" character).

I think that only applies for a request's HTTPBody, right? From what I saw in the code the parameters are added to the body in the setupRequestForOAuth() method of the OAuthSwiftHTTPRequest class, so we'd have to add a different method just for the encoding of the parameters that go in the body.
If that's the case, this is already broken with the current code, though. The spaces sent in the body, for instance, are currently encoded as "%20", not as "+".
So my code wouldn't really change anything about that and it would be safe to change it already.
From what I gather the code that goes in the body must be encoded differently, but at least for me it doesn't matter much: I'm using it as it is, with the spaces encoded incorrectly, for instance, but everything is working fine, even when I send spaces inside a parameter's value.
As for the double encoding, I looked into it further and I think that's actually right: http://stackoverflow.com/a/3292691/3539651
Anyway, let me know if/when you want me to make a PR for the changes in the urlEncodedStringWithEncoding() method… and I can also help writing a different one for the encoding of the body if you decide that's the way to go.

@cschulte22, from what I gathered from the Goodreads forums there are a lot of problems with the encoding of brackets in many different clients. Besides that, according to them servers aren't always consistent in the way they decode the brackets, and what works for one may not work for another.
I think it would be a good idea to try debugging your calls in another client, like Chrome's Postman extension, and if your requests work from there your problem is likely somewhere else on your code. If you can't get it to work in other clients, maybe your server decodes things differently… you could try it with the brackets encoded just once (as I said, they're encoded twice in the signature base string), I don't know.
Whether it's with #165, #168 or with my fix, I am now getting proper signatures that match the link Olivier mentioned in the original post, and my calls also match the ones made by the Postman extension and by Paw, a REST client for Mac.
Good luck. :)

@danielbmarques you can make a PR at anytime
ps: For Goodreads if a solution is found an option could be added

@phimage Done. :) #201
I'll add a Goodreads example to the demo when I have the time.

thanks you for your work on this
I keep this issue open to discuss about signature for the moment
And maybe when I have time make some tests on other services or just execute basic tests

Alright, finally tracked my issue down to code on my server.  Thanks @danielbmarques for the recommendations on Postman and Paw.  Turns out that OAuthSwift was generating proper signatures for me.
The problem on my server was that I was generating my signature using the parsed parameter hash (i.e. conversation[subject]=something was turning into {conversation: {subject: 'something'}}).  When I updated my server to use the decoded raw post data instead of the parameter hash it fixed the incorrect signatures.

This correspondence is correct.
\n is not be escaped is a problem, request in most of the service will fail.
I am using the c163d1d2050205f6ee4ed1c653c949472646f7d7 in their apps.
Most of the Library of the various language is compliant to RFC3986.
Perl https://metacpan.org/source/ETHER/URI-1.71/lib/URI/Escape.pm#L157
Ruby 2.2~ http://docs.ruby-lang.org/en/2.2.0/URI.html
python 3.3~ https://docs.python.org/3.3/library/urllib.parse.html
PHP http://rawurlencode.onlinephpfunctions.com/


aws-sso-util broken by incompatible botocore release

with pull request 2743 having been merged into botocore, there is a problem now:
aws-sso-util assumes it needs to return only the accessToken from the token loader:
https://github.com/benkehoe/aws-sso-util/blob/master/lib/aws_sso_lib/sso.py#L95
but now botocore expects to be given a dictionary by the token loader, and then extract the accessToken itself:
https://github.com/boto/botocore/blob/develop/botocore/credentials.py#L2167
The result of this is a "string indices must be integers" exception.

The perils of relying on internal botocore details. Thanks for this. This appears to be in botocore version 1.27.60 (boto3 version 1.24.60). I'll take a look.

This appears to only affect the get_credentials() function. I will fix it, but is there a reason you're using that instead of get_boto3_session()?

Thanks Ben! I'm calling aws-sso-util credential-process, which seems to always call get_credentials:
https://github.com/benkehoe/aws-sso-util/blob/master/cli/src/aws_sso_util/credential_process.py#L159

Hi folks, I just ran into this issue. Is there a quick workaround for the meantime?

This workaround worked for us:
~/.local/pipx/venvs/aws-sso-util/bin/python -m pip install botocore==1.27.58
It's a slightly older version of botocore from before the incompatible code change.

The newer version of botocore also doesn't support 3.6, I've been trying to figure out how to support both, I think I need to just deprecate 3.6 as well

This workaround worked for us: ~/.local/pipx/venvs/aws-sso-util/bin/python -m pip install botocore==1.27.58 It's a slightly older version of botocore from before the incompatible code change.

I just tried to verify this workaround after a fresh installation of aws-sso-util viapipx and found that I also needed to downgrade the boto3 package to avoid dependency conflicts. Thus, the following worked for me:
~/.local/pipx/venvs/aws-sso-util/bin/python -m pip install botocore==1.27.58 boto3==1.24.58

This should be fixed in aws-sso-lib v1.12 and aws-sso-util v4.29.


Snake FP: newApple is not checking that apple is not created on the snake's body

The solution also provides a good opportunity to show a custom recursive method (as distinct from using HoFs):

Solution (to be tested):

Fixed (in Resourced repo)


Add master node telemetry device

When running benchmarks on clusters with multiple nodes, the master node
could perform worse due to the additional responsibilities it needs to
handle.
To identify this, we add a telemetry device to record the master node.
We don't expect this to change during a typical benchmark, which is why
the default sample interval is 30s. To compensate this large sample
interval, we also record that information on benchmark start/end.
I tested this feature with a short sample interval in test mode, checking that the information was indeed recorded in the test store:
esrally race --distribution-version=7.14.1 --car="4gheap" --track=geonames --test-mode --telemetry master-node-stats --telemetry-params="master-node-stats-sample-interval: 0.1"
http localhost:9200/rally-metrics-2021-09/_search\?size=1000 | jq . | grep master-node -c

TODO:

[ ] #1324 mentioned that this device should be always on, how can I do that?
[x] Is it OK to use those APIs to retrieve the information, or should we revert to the cat API which is much less verbose? (Thanks @b-deam for the help on this!)
[ ] I recorded the master at the start and end of a benchmark, is that OK?

Is it OK to use those APIs to retrieve the information, or should we revert to the cat API which is much less verbose? (Thanks @b-deam for the help on this!)a_Run

@sethmlarson mentioned to me that we can use format="json" for the cat API, and it's stable, so maybe that's best in this case (only one call, less data to transfer). Will let you decide, switching is easy in any case.

Is it OK to use those APIs to retrieve the information, or should we revert to the cat API which is much less verbose? (Thanks @b-deam for the help on this!)a_Run

@sethmlarson mentioned to me that we can use format="json" for the cat API, and it's stable, so maybe that's best in this case (only one call, less data to transfer). Will let you decide, switching is easy in any case.

IMO that cat approach is certainly easiest. If @sethmlarson says it's stable, then I'd trust him ;-), in either case I'd probably want to avoid extraneous API calls (e.g. to Node Info or others).

I addressed all review comments, please take another look

ℹ A seemingly unrelated test fails in a deterministic way on CI, both for Python 3.8 and Python 3.9, but I don't reproduce the failure locally yet.


bump up k8s api to v0.29.0

depends bot fails with k8s api related go vendor's update. Hence I make it manually.

for k8s, packages below are cross reference... hence somehow, we need to bump up the version at same time, which is out of depends bot's feature for now.
	k8s.io/api v0.29.0
	k8s.io/apimachinery v0.29.0
	k8s.io/client-go v0.29.0
	k8s.io/klog/v2 v2.110.1


introduce a finalize! method

There are a few variables (settings) which are only needed while defining API, once everything is set up those variables
could be cleaned up to save memory.
So, the finalize! method was added to the API and Instance classes, I found only few variables which might be cleaned up, over time there might be more.
The finalize! method could be a part of the compile! method, however, compile! must be called on a child level (for example, Twitter::API.compile!) while finalize! must be called on Grape::API to find all inheritors and clean them up.
This PR isn't finished, I would like to get early feedback before going deeper.
TODO:

[ ] better tests
[ ] document this feature in README.md
[ ] add a section to UPGRADING.md
[x] update changelog

There are a few variables (settings) which are only needed while defining API

The whole namespace_stackable/inheritable thing is needed only while defining API.
After the setup you should have a router and bunch of configured API handlers(they should be "flat" in memory/there is no need for inheritance/stack/tree structure in runtime). The "alternative inheritance" mechanism provided by DSL is only needed  for proper configuration of those handlers on setup.
In that sense, I would re-write the framework from scratch =)

I've measured RAM before and after the change. A following tiny API was used:
class API < Grape::API
  version 'v1', using: :path

  helpers do
    params :address_params do
      requires :address, type: Hash do
        requires :street, type: String
        requires :postal_code, type: Integer
        optional :city, type: String
        optional :tags, type: Array[String]
      end
    end
  end

  namespace :addresses do
    params do
      use :address_params
    end

    post '/' do
      # some logic
    end

    patch '/' do
      # some logic
    end
  end
end

Before calling finalize!
Total allocated: 380742 bytes (3676 objects)
Total retained:  63946 bytes (494 objects)

After calling finalize!
Total allocated: 381118 bytes (3681 objects)
Total retained:  62658 bytes (483 objects)

1,25 KB was saved in this case

How do we really feel about introducing a new API for something so minor in terms of memory allocation gain? In many ways I feel that we're doing GC's work.
Should we be finding a way to finalize automatically in the first request?

A good time to call finalize! might be after mounting the API / Endpoint. Probably we should be cleaning up the set up objects after the endpoints are mounted (I think to remember we had a callback for on_mount in a few of these places)

@myxoh There are cases when an endpoint is a top one. In that case, setup steps will stay as they are. Actually, after measuring memory, I am not sure whether it is worth. :thinking:

The profit of this change is too low. I am not going to continue.


Unable to add mapster via nuget

Hello,
I'm trying to add mapster via nuget, But every time it failed to install with error

Object reference not set to an instance of an object.

I'm using VS2015 and .Net 4.5.
Here is screenshot.

Hi @Amit1150,
Sorry for late reply, I cannot reproduce.

I will close this first, since it shouldn't relate to the library.


Add AVM

great project, here is my commit.
(kbw stands for Kabel BW, it's internal an extra model)

Thanks


Authentication with Third Party URL and Msal

Please follow the issue template below. Failure to do so will result in a delay in answering your question.

Library

[x]<EMAIL_ADDRESS>or<EMAIL_ADDRESS>[ ]<EMAIL_ADDRESS>[ ]<EMAIL_ADDRESS>[ ]<EMAIL_ADDRESS>[ ]<EMAIL_ADDRESS>
Important: Please fill in your exact version number above, e.g<EMAIL_ADDRESS>Framework
I am using nodejs only, with msal libraries
Description
Error : Protocol Error
OpenIdConnectAuthenticationHandler: message.State is null or empty.
Screenshot Attached below

Security
Is this issue security related?
Yes it seems so, it's related to login.
Regression
Did this behavior work before?
Configuration
Please provide your MSAL configuration options.
https://portal.azure.com/#blade/Microsoft_AAD_RegisteredApps/ApplicationMenuBlade/Quickstart/appId/4b7e6470-0036-4d46-a7ec-86b67e0ef12c/isMSAApp/
redirect_uri:
https://auth.connectwise.com/f3b925df-97fc-41d0-a567-6488e98ede6c/signin-oidc
// Provide configuration values here.
// For Azure B2C issues, please include your policies.

Reproduction steps

I followed steps from here.
https://portal.azure.com/#blade/Microsoft_AAD_RegisteredApps/ApplicationMenuBlade/Quickstart/appId/4b7e6470-0036-4d46-a7ec-86b67e0ef12c/isMSAApp/
redirect_uri: https://auth.connectwise.com/f3b925df-97fc-41d0-a567-6488e98ede6c/signin-oidc
It takes you to login screen of microsoft but as login done it redirect to auth.connectwise.com and through the error.
It thrw state null message as screenshot attached.

Expected behavior
It should successfully logged in into the auth.connectwise.com without giving state error.
Browsers
Issue is in every browser

Can you provide us with some more information? My initial guess is that something is configured incorrectly.
I can't see your configuration in the portal link you provided as we can't access those sorts of things in your stead. Can you provide a screenshot of your portal configuration, and also a snippet of the code your using, specifically the configuration your passing into msal and how it's being used afterward?

Hello,
Attaching here the sample code I am using.
https://bit.ly/2X1eXXK
https://bit.ly/2JwyiIz
Can you please let me know what other configuration I need to share?

@harshek Can you try setting the redirectUri to a page in your local application and see if you are getting a similar error? Can you also remove state from your auth config? This should only be optionally passed to the request object.

Closing due to inactivity and we don't believe this is an issue with msal. Let us know if you still need help.


Make site offline compatible

Commit

This issue will be closed as there are some problems with sw.js...

UPDATE: I'll be tracking this with #36.


Frequency Chat socket failure warning

Seeing this message in the log quite often (once every few seconds) after server restart
[info] - application - Chatsocket rejected

it's from this part of the code:
https://github.com/patsonluk/airline/blob/d4ab9fde31598b1f2789487b70c1d2b5b63b4502/airline-web/app/controllers/ChatApplication.scala#L35
Which i believe is referencing the similar main socket code as in:
https://github.com/patsonluk/airline/blob/d4ab9fde31598b1f2789487b70c1d2b5b63b4502/airline-web/app/controllers/WebsocketApplication.scala#L16
However i do not observe the same frequent failure on the websocket application.
I believe the chat socket has certain code in the front end and attempts to reconnect? Would it be possible that it tries to reconnect even though the request session is no longer valid? If so, perhaps the front-end script should stop retrying if the server failed the request with a reason that the server session is no longer valid?
@daprof1  any thoughts please ? :)

Done and pushed


Fully Managed PaaS solution

Is the Azure team considering turning ELK into a fully managed PaaS solution like what AWS has?

Azure has a managed PaaS service in the form of Azure Search so I wouldn't imagine another search service appearing, although that's a question best answered by Microsoft.
I can tell you that we're working on bringing Elastic Cloud to Azure, to offer the best Elasticsearch as a Service experience. No definite timeline for when this will be ready as yet but stay tuned.