Consumer group partitions always wait for slowest partition

We ran into an issue where our consumer group started falling behind on some topics. We had one consumer group servicing multiple topics. We found that sarama suffers from "naughty neighbor" between all partitions the consumer is subscribed to. If there is one slower partition (either because it is slower, or because it just has more work), all partitions must wait for the slow partition to finish. This is because subscriptionConsumer() waits for all partition consumers to finish before asking for more data from the server.
This is especially problematic if you are using consistently hashed partitions. Data is not spread randomly, so it is easier to get temporary hot spots where one partition has a lot more work, or perhaps has a type of work that takes a lot longer to process. While that issue persists, all your partitions start to back up instead of just one.
Do you have any thoughts on how to address this? Maybe a min/max/maxwaittime approach for waiting for the partitions to be done?
We have already switched to separate consumers for each topic, which helped for now. Another workaround would be to just spin up multiple consumers for each topic in the same process.

Are you looking for the Consumer.MaxProcessingTime config? I believe lowering that does what you want.

Thank you for responding. We already have MaxProcessingTime configured appropriately. The issue in sarama stems from the fact there is a single event loop per consumer per broker (pasted below).
It fetches data for all partitions/topics, kicks off processing in separate goroutines, waits for all to finish, then handles errors and fetches the next batch of data. Consider what happens if you have 8 partitions and each piece of work takes 1 second to process. If one of the partitions gets a huge influx of 1000 items and each of the other 7 partitions has only 1 item to process, all the partitions will wait 1000 seconds for the slow partition before consumer more data. This will cause all partitions to back up instead of just the single partition.
func (bc *brokerConsumer) subscriptionConsumer() {
	<-bc.wait // wait for our first piece of work

	// the subscriptionConsumer ensures we will get nil right away if no new subscriptions is available
	for newSubscriptions := range bc.newSubscriptions {
		bc.updateSubscriptions(newSubscriptions)

		if len(bc.subscriptions) == 0 {
			// We're about to be shut down or we're about to receive more subscriptions.
			// Either way, the signal just hasn't propagated to our goroutine yet.
			<-bc.wait
			continue
		}

		response, err := bc.fetchNewMessages()

		if err != nil {
			Logger.Printf("consumer/broker/%d disconnecting due to error processing FetchRequest: %s\n", bc.broker.ID(), err)
			bc.abort(err)
			return
		}

		bc.acks.Add(len(bc.subscriptions))
		for child := range bc.subscriptions {
			child.feeder <- response
		}
		bc.acks.Wait()
		bc.handleResponses()
	}
}

I think I'm observing something similar to this issue. In a setup with multiple brokers and 10 partitions, adding more consumers to a consumer group (CG) doesn't result in any improvement in overall throughput (i.e., the total number of messages processed by all consumers in a CG). Instead, the load seems to be spread among the consumers almost equally.

Now, that effect seems to change based on the processing time. When the processing time is short (50ms), adding consumers actually leads to higher overall throughput. When the processing time is higher (e.g., 250 or 500ms), however, not only the throughput of each individual consumer is slower (as expected), but adding more consumers to the CG doesn't result in any increases in overall throughput. It's as if slow consumers were holding back the other consumers in the same CG.
We've tested up to 10 consumers (1 per partition) and the overall throughput is capped at basically the same as what we see with 2 consumers. Other info about our setup:

We have plenty of messages to be processed (millions);
The consumers run the exact same code and consume from the same topics;
The consumers are running in independent processes (VMs) and are not in any way close to their limit;
The brokers are also fine; adding more consumers to different consumer groups results in much higher throughput.


Cannot find symbol when using library from Java

I am on Kotlin 1.4.10 and 2.0.0-alpha.
When I try to access properties of DateTimeTz I get an error.
final double nowOffsetInMillis = DateTimeTz.Companion.nowLocal().getOffset();
But then the compiler complains:
error: cannot find symbol
final double nowOffsetInMillis = DateTimeTz.Companion.nowLocal().getOffset();
^
symbol:   method getOffset()
location: class DateTimeTz
It basically happens on any property, not only 'offset'
Any ideas?

Also tried 2.0.0-rc2 but the issue is still there.

Did some more experimentation.....getMinutes() works and any other that returns a basic type like Int work. But the ones that returns some other com.soywiz.klock class don't work

@weliem Klock is not intended to be consumed from Java. The main problem here is that we are using inline classes that is not yet available on Java until they implement value types with Valhalla.
Kotlin creates a special mangled names for functions using inlined types that probably won't be callable from Java. What Kotlin does is to mangle the function name + instead of receiving/returning for example TimeSpan, it receives/returns a Double, and the inline class functions are static.
This makes consuming the API from Java hard, or directly impossible. Once Java supports Valhalla, it is likely that Kotlin supports generating code using that stuff without having to mangling function names and stuff, and you will be able to consume it from modern Java versions directly, but for now, it is not possible.

We choose Kotlin for it's interoperability with Java and other platforms. We have been using Klock successfully for the last year and calling Klock from Java has never been a problem. But now that we are upgrading from 1.8.3 to 2.0.0, our code is broken. So did something change in Klock? Did older versions not use inline classes?
Anyway, perhaps I can write some extentions to fix the interoperability issues....

I was not aware that some people was using it with Java, and directly assumed it was not possible. Inline classes are experimental, so I think that's something unreliable at this point. You could check the exported class methods on both versions of the library, maybe the signature change because inline classes are experimental.
I'd suggest you to stick to 1.8.3 for now. On Kotlin new compiled versions are needed because Kotlin/Native doesn't have binary compatibility of the libraries, but with the JVM you should be safe to stick with older versions.

I think our use case is very common: we have a library written in Kotlin Multi Platform which we use in our iOS, Android, and JVM applications. We use Klock in our library and it is also used in the external interface of the library. Hence interoperability with Java and Swift is essential...
Will try to find a workaround for now, but we would greatly appreciate it if future versions of your library would have improved interoperability.

I'll investigate if we can do something regarding to binary compatibility, but if Kotlin started to use synthetic mangled methods with different signature on 1.4 because inline classes are experimental and there are no guarantees for them, I'm not sure if we could do anything. In any case, this library is opensource and public domain, if you have workarounds you want to include here, just make a PR with your proposals 👍


Marathon destroys docker containers when suspended

I am using docker containers as apps to be deployed in mesos. The docker containers run apace storm components. From marathon UI, when one of the app is suspended, first issue the restart link does not work. So i tried scaling the application instead of restarting. Scaling actually creates a new container. So the data within the old container is lost. I can do a workaround using docker volumes to host, but then I would need to pin the apps to host.
Is this an expected behavior of restarting an app using marathon?

Clicking on Suspend is actually a shortcut for scaling an app down to 0 instances.
I agree that this might not be very intuitive.
cc/ @leemunroe

Thanks for the response. Does scaling down to 0 means, the docker container will be destroyed? If yes, what's the best way to stop and restart an app that's based on docker?

It is confusing. We should at least update our warning.
@srshanmugam So I understand correctly, your expectation was that the current state would be saved on suspend, and restarting would have brought it back to that state?

Thanks again, yes, at very minimum, not destroy the containers. Destroying the container deletes the state of the container, especially when volumes are not used.

Any updates or suggestions on handling docker container apps restart with Marathon?

@srshanmugam @Ieemunroe  I also appeared this problem,  when one of the app is suspended,
Restart the app does not work,If use Scale my app can  run, But  the App data is new.

My marathon logs , Marathon's version  1.1.3
Sep 21 17:05:54 localhost marathon[11650]: [2016-09-21 17:05:54,417] INFO <IP_ADDRESS> - - [21/Sep/2016:09:05:54 +0000] "GET //<IP_ADDRESS>:8080/v2/queue HTTP/1.1" 200 12 "http://<IP_ADDRESS>:8080/ui/" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.108 Safari/537.36"  (mesosphere.chaos.http.ChaosRequestLog$$EnhancerByGuice$$dc32d379:qtp713707020-38)
Sep 21 17:05:54 localhost marathon[11650]: [2016-09-21 17:05:54,417] INFO <IP_ADDRESS> - - [21/Sep/2016:09:05:54 +0000] "GET //<IP_ADDRESS>:8080/v2/apps//web-01?embed=app.taskStats&embed=app.readiness HTTP/1.1" 200 1090 "http://<IP_ADDRESS>:8080/ui/" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.108 Safari/537.36"  (mesosphere.chaos.http.ChaosRequestLog$$EnhancerByGuice$$dc32d379:qtp713707020-33)
Sep 21 17:05:55 localhost marathon[11650]: [2016-09-21 17:05:55,146] INFO Upgrade group id:/ version:2016-09-21T09:05:55.146Z with force:false (mesosphere.marathon.state.GroupManager:marathon-akka.actor.default-dispatcher-6074)
Sep 21 17:05:55 localhost marathon[11650]: [2016-09-21 17:05:55,147] INFO [/web-01]: restart detected for app (oldVersion FullVersionInfo(2016-09-21T09:04:34.315Z,2016-09-21T09:04:34.315Z,2016-09-21T07:45:13.750Z)) (mesosphere.marathon.upgrade.GroupVersioningUtil$:ForkJoinPool-2-worker-11)
Sep 21 17:05:55 localhost marathon[11650]: [2016-09-21 17:05:55,155] INFO Computed new deployment plan:
Sep 21 17:05:55 localhost marathon[11650]: DeploymentPlan 2016-09-21T09:05:55.146Z
Sep 21 17:05:55 localhost marathon[11650]: step 1:
Sep 21 17:05:55 localhost marathon[11650]:  * Restart(App(/web-01, image="docker.io/nginx")))
Sep 21 17:05:55 localhost marathon[11650]: (mesosphere.marathon.state.GroupManager:ForkJoinPool-2-worker-11)
Sep 21 17:05:55 localhost marathon[11650]: [2016-09-21 17:05:55,155] INFO Deploy plan with force=false:
Sep 21 17:05:55 localhost marathon[11650]: DeploymentPlan 2016-09-21T09:05:55.146Z
Sep 21 17:05:55 localhost marathon[11650]: step 1:
Sep 21 17:05:55 localhost marathon[11650]:  * Restart(App(/web-01, image="docker.io/nginx")))
Sep 21 17:05:55 localhost marathon[11650]:  (mesosphere.marathon.MarathonSchedulerService$$EnhancerByGuice$$cefd15e2:ForkJoinPool-2-worker-11)
Sep 21 17:05:55 localhost marathon[11650]: [2016-09-21 17:05:55,156] INFO [/web-01] storing new app version 2016-09-21T09:05:55.146Z (mesosphere.marathon.state.GroupManager:ForkJoinPool-2-worker-11)
Sep 21 17:05:55 localhost marathon[11650]: [2016-09-21 17:05:55,473] INFO Deployment of / successful (mesosphere.marathon.MarathonSchedulerActor:marathon-akka.actor.default-dispatcher-6067)
Sep 21 17:05:55 localhost marathon[11650]: [2016-09-21 17:05:55,474] INFO Removing a82f054a-9497-497e-9698-f85b46f1b25a from list of running deployments (mesosphere.marathon.upgrade.DeploymentManager:marathon-akka.actor.default-dispatcher-6067)
Sep 21 17:05:55 localhost marathon[11650]: [2016-09-21 17:05:55,729] INFO Updated groups/apps according to deployment plan a82f054a-9497-497e-9698-f85b46f1b25a (mesosphere.marathon.state.GroupManager:ForkJoinPool-2-worker-5)
Sep 21 17:05:55 localhost marathon[11650]: [2016-09-21 17:05:55,729] INFO Deployment acknowledged. Waiting to get processed:
Sep 21 17:05:55 localhost marathon[11650]: DeploymentPlan 2016-09-21T09:05:55.146Z
Sep 21 17:05:55 localhost marathon[11650]: step 1:
Sep 21 17:05:55 localhost marathon[11650]:  * Restart(App(/web-01, image="docker.io/nginx")))
Sep 21 17:05:55 localhost marathon[11650]: (mesosphere.marathon.state.GroupManager:ForkJoinPool-2-worker-11)

There is no way to stop and resume the same docker container instance using Marathon. Suspending an app destroys the related containers.

Note: This issue has been migrated to https://jira.mesosphere.com/browse/MARATHON-2212. For more information see https://groups.google.com/forum/#!topic/marathon-framework/khtvf-ifnp8.


Custom end date before start date still returns a result!

If I select the start date: 22 Jun 2020
And end date 21 Jun 2019
The registry returned results, where it should have thrown an error. See screenshot.
The fields should really be named "Start date" and "End date" as well.

Disable selection of nonsensical values and rename fields

Fixed, the date fields are restricted by the selection of a date in the opposite field. The reset button added for the possibility of a value removing.

Yes look good to me now. I think it would also be helpful if the user is not allowed to select a date in the future. Is this possible?


[Issue]: Segmentation faults when using dual GPUs (RX 7900 GRE)

Problem Description
#2804
When using dual RX 7900 GRE setup, accessing the second GPU yields seg faults.
I tried reinstalling Ubuntu or ROCm but to no avail
Operating System
NAME="Ubuntu" VERSION="24.04 LTS (Noble Numbat)"
CPU
AMD Ryzen 9 7900 12-Core Processor
GPU
AMD Radeon RX 7900 XT
ROCm Version
ROCm 6.2.0
ROCm Component
No response
Steps to Reproduce

Have a dual GPU setup that works as PCIe 5.0 X8 for each (ProArt X670E-CREATOR WIFI)
Clean install of Ubuntu 24.04
Install ROCm according to the official document
Do anything that works with the second GPU, and it will give segmentation fault

For pytorch,
`

import torch
X = torch.rand(2,3).cuda(0)
Y = torch.rand(2,3).cuda(1)
Segmentation fault (core dumped)
[For rocHPCG](https://github.com/ROCm/rocHPCG/issues/80),
Memory access fault by GPU node-1 (Agent handle: 0x2308090) on address 0x77525e362000. Reason: Page not present or supervisor privilege. Aborted (core dumped)
`

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support
ROCk module version 6.8.5 is loaded
HSA System Attributes
Runtime Version:         1.14
Runtime Ext Version:     1.6
System Timestamp Freq.:  1000.000000MHz
Sig. Max Wait Duration:<PHONE_NUMBER>3709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model:           LARGE
System Endianness:       LITTLE
Mwaitx:                  DISABLED
DMAbuf Support:          YES
==========
HSA Agents

Agent 1

Name:                    AMD Ryzen 9 7900 12-Core Processor
Uuid:                    CPU-XX
Marketing Name:          AMD Ryzen 9 7900 12-Core Processor
Vendor Name:             CPU
Feature:                 None specified
Profile:                 FULL_PROFILE
Float Round Mode:        NEAR
Max Queue Number:        0(0x0)
Queue Min Size:          0(0x0)
Queue Max Size:          0(0x0)
Queue Type:              MULTI
Node:                    0
Device Type:             CPU
Cache Info:
L1:                      32768(0x8000) KB
Chip ID:                 0(0x0)
ASIC Revision:           0(0x0)
Cacheline Size:          64(0x40)
Max Clock Freq. (MHz):   5482
BDFID:                   0
Internal Node ID:        0
Compute Unit:            24
SIMDs per CU:            0
Shader Engines:          0
Shader Arrs. per Eng.:   0
WatchPts on Addr. Ranges:1
Memory Properties:
Features:                None
Pool Info:
Pool 1
Segment:                 GLOBAL; FLAGS: FINE GRAINED
Size:                    98562356(0x5dff134) KB
Allocatable:             TRUE
Alloc Granule:           4KB
Alloc Recommended Granule:4KB
Alloc Alignment:         4KB
Accessible by all:       TRUE
Pool 2
Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
Size:                    98562356(0x5dff134) KB
Allocatable:             TRUE
Alloc Granule:           4KB
Alloc Recommended Granule:4KB
Alloc Alignment:         4KB
Accessible by all:       TRUE
Pool 3
Segment:                 GLOBAL; FLAGS: COARSE GRAINED
Size:                    98562356(0x5dff134) KB
Allocatable:             TRUE
Alloc Granule:           4KB
Alloc Recommended Granule:4KB
Alloc Alignment:         4KB
Accessible by all:       TRUE
ISA Info:

Agent 2

Name:                    gfx1100
Uuid:                    GPU-71be6bf20576cf7f
Marketing Name:          Radeon RX 7900 GRE
Vendor Name:             AMD
Feature:                 KERNEL_DISPATCH
Profile:                 BASE_PROFILE
Float Round Mode:        NEAR
Max Queue Number:        128(0x80)
Queue Min Size:          64(0x40)
Queue Max Size:          131072(0x20000)
Queue Type:              MULTI
Node:                    1
Device Type:             GPU
Cache Info:
L1:                      32(0x20) KB
L2:                      6144(0x1800) KB
L3:                      65536(0x10000) KB
Chip ID:                 29772(0x744c)
ASIC Revision:           0(0x0)
Cacheline Size:          64(0x40)
Max Clock Freq. (MHz):   1927
BDFID:                   768
Internal Node ID:        1
Compute Unit:            80
SIMDs per CU:            2
Shader Engines:          6
Shader Arrs. per Eng.:   2
WatchPts on Addr. Ranges:4
Coherent Host Access:    FALSE
Memory Properties:
Features:                KERNEL_DISPATCH
Fast F16 Operation:      TRUE
Wavefront Size:          32(0x20)
Workgroup Max Size:      1024(0x400)
Workgroup Max Size per Dimension:
x                        1024(0x400)
y                        1024(0x400)
z                        1024(0x400)
Max Waves Per CU:        32(0x20)
Max Work-item Per CU:    1024(0x400)
Grid Max Size:<PHONE_NUMBER>(0xffffffff)
Grid Max Size per Dimension:
x<PHONE_NUMBER>(0xffffffff)
y<PHONE_NUMBER>(0xffffffff)
z<PHONE_NUMBER>(0xffffffff)
Max fbarriers/Workgrp:   32
Packet Processor uCode:: 232
SDMA engine uCode::      21
IOMMU Support::          None
Pool Info:
Pool 1
Segment:                 GLOBAL; FLAGS: COARSE GRAINED
Size:                    16760832(0xffc000) KB
Allocatable:             TRUE
Alloc Granule:           4KB
Alloc Recommended Granule:2048KB
Alloc Alignment:         4KB
Accessible by all:       FALSE
Pool 2
Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
Size:                    16760832(0xffc000) KB
Allocatable:             TRUE
Alloc Granule:           4KB
Alloc Recommended Granule:2048KB
Alloc Alignment:         4KB
Accessible by all:       FALSE
Pool 3
Segment:                 GROUP
Size:                    64(0x40) KB
Allocatable:             FALSE
Alloc Granule:           0KB
Alloc Recommended Granule:0KB
Alloc Alignment:         0KB
Accessible by all:       FALSE
ISA Info:
ISA 1
Name:                    amdgcn-amd-amdhsa--gfx1100
Machine Models:          HSA_MACHINE_MODEL_LARGE
Profiles:                HSA_PROFILE_BASE
Default Rounding Mode:   NEAR
Default Rounding Mode:   NEAR
Fast f16:                TRUE
Workgroup Max Size:      1024(0x400)
Workgroup Max Size per Dimension:
x                        1024(0x400)
y                        1024(0x400)
z                        1024(0x400)
Grid Max Size:<PHONE_NUMBER>(0xffffffff)
Grid Max Size per Dimension:
x<PHONE_NUMBER>(0xffffffff)
y<PHONE_NUMBER>(0xffffffff)
z<PHONE_NUMBER>(0xffffffff)
FBarrier Max Size:       32

Agent 3

Name:                    gfx1100
Uuid:                    GPU-2a5814ccdc518719
Marketing Name:          Radeon RX 7900 GRE
Vendor Name:             AMD
Feature:                 KERNEL_DISPATCH
Profile:                 BASE_PROFILE
Float Round Mode:        NEAR
Max Queue Number:        128(0x80)
Queue Min Size:          64(0x40)
Queue Max Size:          131072(0x20000)
Queue Type:              MULTI
Node:                    2
Device Type:             GPU
Cache Info:
L1:                      32(0x20) KB
L2:                      6144(0x1800) KB
L3:                      65536(0x10000) KB
Chip ID:                 29772(0x744c)
ASIC Revision:           0(0x0)
Cacheline Size:          64(0x40)
Max Clock Freq. (MHz):   1927
BDFID:                   1792
Internal Node ID:        2
Compute Unit:            80
SIMDs per CU:            2
Shader Engines:          6
Shader Arrs. per Eng.:   2
WatchPts on Addr. Ranges:4
Coherent Host Access:    FALSE
Memory Properties:
Features:                KERNEL_DISPATCH
Fast F16 Operation:      TRUE
Wavefront Size:          32(0x20)
Workgroup Max Size:      1024(0x400)
Workgroup Max Size per Dimension:
x                        1024(0x400)
y                        1024(0x400)
z                        1024(0x400)
Max Waves Per CU:        32(0x20)
Max Work-item Per CU:    1024(0x400)
Grid Max Size:<PHONE_NUMBER>(0xffffffff)
Grid Max Size per Dimension:
x<PHONE_NUMBER>(0xffffffff)
y<PHONE_NUMBER>(0xffffffff)
z<PHONE_NUMBER>(0xffffffff)
Max fbarriers/Workgrp:   32
Packet Processor uCode:: 232
SDMA engine uCode::      21
IOMMU Support::          None
Pool Info:
Pool 1
Segment:                 GLOBAL; FLAGS: COARSE GRAINED
Size:                    16760832(0xffc000) KB
Allocatable:             TRUE
Alloc Granule:           4KB
Alloc Recommended Granule:2048KB
Alloc Alignment:         4KB
Accessible by all:       FALSE
Pool 2
Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
Size:                    16760832(0xffc000) KB
Allocatable:             TRUE
Alloc Granule:           4KB
Alloc Recommended Granule:2048KB
Alloc Alignment:         4KB
Accessible by all:       FALSE
Pool 3
Segment:                 GROUP
Size:                    64(0x40) KB
Allocatable:             FALSE
Alloc Granule:           0KB
Alloc Recommended Granule:0KB
Alloc Alignment:         0KB
Accessible by all:       FALSE
ISA Info:
ISA 1
Name:                    amdgcn-amd-amdhsa--gfx1100
Machine Models:          HSA_MACHINE_MODEL_LARGE
Profiles:                HSA_PROFILE_BASE
Default Rounding Mode:   NEAR
Default Rounding Mode:   NEAR
Fast f16:                TRUE
Workgroup Max Size:      1024(0x400)
Workgroup Max Size per Dimension:
x                        1024(0x400)
y                        1024(0x400)
z                        1024(0x400)
Grid Max Size:<PHONE_NUMBER>(0xffffffff)
Grid Max Size per Dimension:
x<PHONE_NUMBER>(0xffffffff)
y<PHONE_NUMBER>(0xffffffff)
z<PHONE_NUMBER>(0xffffffff)
FBarrier Max Size:       32
*** Done ***
Additional Information
I've also

disabled integrated GPU
enabled resizable bar
enabled above 4G decoding

When I launch some kernels by myself, it works without (seemingly) no problems.
I confirmed that torch nightly works without the above issue.
So it might not be the problem with the rocm itself, but maybe a compatibility issue

Hi @han-minhee, you mentioned launching kernels by yourself and testing using the ROCm 6.2  PyTorch wheel was successful. How were you initially installing torch when you ran into this error?
import torch
X = torch.rand(2,3).cuda(0)
Y = torch.rand(2,3).cuda(1)
Segmentation fault (core dumped)
Memory access fault by GPU node-1 (Agent handle: 0x2308090) on address 0x77525e362000. Reason: Page not present or supervisor privilege. Aborted (core dumped)

As my initial report says, I'm using ROCm 6.2 with Ubuntu 24.04. It was just that I was able to install torch with ROCm 6.1 without any apparent problem.

@han-minhee, apologies for the confusion. To use torch with your ROCm 6.2.0 installation, it would be best to continue with the 6.2 nightly build. The documentation will also be updated to reflect the new installation command shortly.
pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/rocm6.2/

The mismatch in your original configuration between the 6.2 installed on your system and the 6.1 torch installation is the most likely culprit for the seg faults as I was not able to reproduce them in my testing. If you do encounter them again, please leave a comment and I will re-open this issue to further investigate.
I have added the AMD Radeon RX 7900 GRE to the issue template, thanks for the heads up.


🚸 Update metadata like description upon re-running

@timtreis ran into this

I agree with Tim. We should consider to update metadata like this silently. The danger is that another user might have wanted a different description, but I agree that’s rather unlikely.

If the hash is the same, updating the description and not uploading a new version sounds reasonable to me

Made the change here:

https://github.com/laminlabs/lamindb/pull/1588


401 Auth error connecting to enterprise instance

I have an enterprise instance of GH v2.4 and I'm unable to get my pull request builds to work.  I have configured the access token and I am able to verify the credentials in the master jenkins settings.  I can even post a comment to an open pull request.
However, when the pull request is created or a comment/commit is added, Jenkins throws a 401.
The corporate Jenkins is behind HTTPS.
Here's a piece of the log:
Oct 30, 2015 4:08:49 PM WARNING org.jenkinsci.plugins.ghprb.GhprbPullRequest obtainAuthorEmail

Couldn't obtain author email.
java.io.IOException: Server returned HTTP response code: 401 for URL: http://github.company.com/api/v3/users/username

Closing this for now due to inactivity.


Timeout issue

Enter ProcessID: 12664
Injecting into remote process: Success!
Waiting for remote thread IPC:
Remote thread wait timeout: Unable to find target thread
Press any key to continue . . .
I've tried in calculator and multiple UWP Apps I just get a timeout, this is using a compiled version of the build x64.
OS Name	Microsoft Windows 10 Pro
Version	10.0.16299 Build 16299
System Type	x64-based PC

I am getting this same issue. Did you find a solution?

nope

Not sure why you closed this issue immediately after opening it though. I figured it meant "nvm fixed it" or something.
I'm doing some work on #16 that might help with this.

any update?

nope
try with a compiled version of #16

UWPDumper.zip

this issue occurs with the new game pass for pc titles

this issue occurs with the new game pass for pc titles

the issue may be a mismatched platform. Try the x86 version

nope

try with a compiled version of #16
UWPDumper.zip

I'm experiencing the timeout issue with some UWP apps and wanted to try this but this UWPDumper is compiled against the debug version of the VC++ dlls which aren't included in the VC++ redistributable so I can't run it. Any chance of a version compiled against the public runtime library?

this issue occurs with the new game pass for pc titles

the issue may be a mismatched platform. Try the x86 version

This worked for me on some titles! Remember, even if you're running 64 bit windows, some software binaries may be 32-bit.

It may be worth closing this issue


New color scheme for those who are color blind

Could accompany colors with a symbol...

added thumb icon


Test all links and code examples in docs for GA/Beta.

Test all doc links and examples for...

[x] BigQuery (results)

[x] Links/references (#3196)
[x] Examples/Snippets

[x] Datastore 1 anchor reference error, (results)

[x] Links/references (#3186)
[x] Examples/Snippets

[ ] Storage (results)

[x] Links/references (#3192)
[ ] Examples/Snippets (generate_upload_policy snippet appears to be broken) (results)

[ ] Logging (results)

[x] Links/references (#3187)
[ ] Examples/Snippets

[ ] Natural Language

[x] Links/references (#3197)
[ ] Examples/Snippets

[ ] Speech

[x] Links/references (#3198)
[ ] Examples/Snippets (#3206) (script)

[ ] Translation

[x] Links/references (#3199)
[ ] Examples/Snippets

[ ] Vision

[ ] Links/references (#3202)
[ ] Examples/Snippets

/cc @lukesneeringer @omaray

This is in some sense the grandchild of #212. I vote for the approach used in #2738.

Moving discussion from https://github.com/GoogleCloudPlatform/google-cloud-python/pull/3206#issuecomment-289135807 to this thread.
@lukesneeringer so here's the current status of snippets for GA.

Datastore snippet tests work/pass and are the most current architecture.
BigQuery snippet tests work/pass but they are the old architecture and need to be updated.
Storage snippet tests do not work and are the old architecture.
Logging snippet tests do not exists.

The Right Thing™ to do is probably update the BigQuery and Storage snippets to the same methodology that Datastore is using. I think that can be done in reasonable time. Logging however, is pretty huge and may take a fair bit of work.
I thought this was already a known, and that's why manual testing was the preferred route for now.

@daspecster FYI: https://github.com/GoogleCloudPlatform/google-cloud-python/blob/1bcff68a53435301e1f309d634420703ec8c2464/docs/logging_snippets.py

that's why manual testing was the preferred route for now.

There is an inflection point where doing things "by hand" starts to become longer than doing it the right way

@dhermes logging_snippets.py looks fairly outdated.
╰─$ python docs/logging_snippets.py
instantiate_client       : Instantiate client.
  ERROR: ValueError('This library only supports credentials from google-auth-library-python. See https://google-cloud-python.readthedocs.io/en/latest/google-cloud-auth.html for help on authentication with this library.',)
client_list_entries      : List entries via client.
  ERROR: TypeError('Can not find message descriptor by type_url: type.googleapis.com/google.cloud.audit.AuditLog.',)
logger_usage             : Logger usage.
metric_crud              : Metric CRUD.
sink_storage             : Sink log entries to storage.
sink_bigquery            : Sink log entries to bigquery.
sink_pubsub              : Sink log entries to pubsub.
logging_handler          : None
  ERROR: TypeError('logging_handler() takes exactly 1 argument (2 given)',)
setup_logging            : None
  ERROR: TypeError('setup_logging() takes exactly 1 argument (2 given)',)

I am unaware of how up to date it is, just posted as a response to "Logging snippet tests do not exists."

@dhermes yeah thanks! I updated the status in the list.

@dhermes the Storage snippet for generate_upload_policy seems to be broken. I looked at the docs but I don't see where the breakdown is.
Here's the output when trying to upload with the generated HTML form.
<Error>
InvalidArgument
<Message>Invalid argument.</Message>
Cannot create buckets using a POST.
</Error>

Any ideas?

@jonparrott Wrote the snippet.

It's probably fine, it's just that the bucket has to exist.

@jonparrott in my test, I was/am using a bucket that exists.

Weird. That error message indicates otherwise!

@jonparrott haha I know! 😕

Can you try manually verifying it?

@jonparrott yeah that's how I'm getting the error. I'm not just running the snippets. The storage snippets need some updating to work.
bucket = client.bucket('my-bucket24356uytrew')
conditions = [
    ['starts-with', '$key', ''],
    {'acl': 'public-read'}]

policy = bucket.generate_upload_policy(conditions)

# Generate an upload form using the form fields.
policy_fields = ''.join(
    '<input type="hidden" name="{key}" value="{value}">'.format(
        key=key, value=value)
    for key, value in policy.items()
)

upload_form = (
    '<form action="http://{bucket_name}.storage.googleapis.com"'
    '   method="post" enctype="multipart/form-data">'
    '<input type="text" name="key" value="">'
    '<input type="hidden" name="bucket" value="{bucket_name}">'
    '<input type="hidden" name="acl" value="public-read">'
    '<input name="file" type="file">'
    '<input type="submit" value="Upload">'
    '{policy_fields}'
    '<form>').format(bucket_name=bucket.name, policy_fields=policy_fields)

print(upload_form)

Hmm. I'm at a a bit of a loss.

I was able to get the logging snippets running somewhat.
instantiate_client       : Instantiate client.
  ERROR: ValueError('This library only supports credentials from google-auth-library-python. See https://google-cloud-python.readthedocs.io/en/latest/google-cloud-auth.html for help on authentication with this library.',)
client_list_entries      : List entries via client.
  ERROR: TypeError('Can not find message descriptor by type_url: type.googleapis.com/google.cloud.audit.AuditLog.',)
logger_usage             : Logger usage.
metric_crud              : Metric CRUD.
sink_storage             : Sink log entries to storage.
sink_bigquery            : Sink log entries to bigquery.
sink_pubsub              : Sink log entries to pubsub.
logging_handler          : None
setup_logging            : None

@daspecster can you file a separate bug about the storage policy document stuff and assign it to me? Thanks!

@jonparrott sure thing!
https://github.com/GoogleCloudPlatform/google-cloud-python/issues/3222

Ok, just a few PR's to merge and I think we're pretty much there.

#3219
#3220
#3206
#3227

Ok all the PRs are merged.
I'm not saying the docs are perfect but they should be a lot closer.
I'm going to close this issue since all the outlined tasks are completed.

:tada:, thanks.


pizza.pl

bas2perl.pl is a simple perl script to convert BASIC code to Perl code, pizza.pl is the example output.

Hey excellent! 🍕


Add files via upload

This is my personal Protofolio-Website using HTML5,CSS3,JavaScript
https://ubiquitous-starlight-096b53.netlify.app/

https://ubiquitous-starlight-096b53.netlify.app/


Sahilll15 Profile

Description
I have created a pull request to add my profile to the website
Related Issues
Checklist

[x] I have read and followed the Contribution Guidelines.
[x] All new and existing tests passed.
[x] I have updated the documentation to reflect the changes I've made.
[x] My code follows the code style of this project.
[x] The title of my pull request is a short description of the requested changes.

@Sahilll15 Thank you so much for your contribution! Your efforts are greatly appreciated and will go a long way in improving our project. Please feel free to share it with others and help us grow even further!


FT: Set one pair of accessKey/secretKey with env variables

@ironman-machine

Hello @nicolas2bert
Starting end to end procedure using the following payload:
{"REPO_NAME": "S3", "DEFAULT_BRANCH": "master", "SCALITY_S3_BRANCH": "ft/keyenvvar"}
Please follow http://ci.ironmann.io/gh/scality/Integration/10166 for CI status.

E2E passed http://ci.ironmann.io/gh/scality/Integration/10174#build-parameters


PT-PT translation + fix typos

This PR includes:

European portuguese (pt-pt) translation;
Fix typos in translation.json files.

@benawad In the language selector, I suggest that Brazilian Portuguese be represented as pt-br (can change that if needed), since there are now two portuguese dialects (pt-br + pt-pt).

yeah if you could change that, that would be great

@benawad Done


terINSUF_FEE_B

I get from time to time a "terINSUF_FEE_B" error. But I am certain I have enough XRP to pay the fee. It happens when several transactions are sent at the same time.
I suspect it sums up all transactions for this account that are in the open ledger and on basis of that determines the possible balance ... So if I place an order to sell 30.000 XRP and also an order to sell 20.000 XRP in the same open ledger and I have 40.000 XRP in my account, then I would receive this error...... Is this correct and 'works as intended'?
Not completely sure, but this "total XRP spent" seems to be determined per orderbook. I.e. "if the total of XRP sell orders of one orderbook > XRP balance" then I receive the terINSUF_FEE_B. Strangely enough I can place 1 order manually where the XRP amount to be sold is larger then the balance.
I think I can work around it if I have to, but it seems non-intended.
Below an example trace of my logs. At the time, the account had +/- 85.000 XRP
[24/10/2017 06:54:37] Creating GTC order for Address: xxx - Sequence: 849786

= 849786
{
  "direction": "sell",
  "quantity": {
    "currency": "XRP",
    "value": "15000.00000000"
  },
  "totalPrice": {
    "currency": "BTC",
    "counterparty": "rchGBxcD1A1C2tdxF6papQYZ8kjRKMYcL",
    "value": "0.54636318"
  }
}

[24/10/2017 06:54:37] GTC order '849786' will cancel order '849780'

= 849786
{
  "resultCode": "terQUEUED",
  "resultMessage": "Held until escalated fee drops."
}

[24/10/2017 06:54:37] Creating GTC order for Address: xxx - Sequence: 849788

= 849788
{
  "direction": "sell",
  "quantity": {
    "currency": "XRP",
    "value": "36222.61903100"
  },
  "totalPrice": {
    "currency": "BTC",
    "counterparty": "rchGBxcD1A1C2tdxF6papQYZ8kjRKMYcL",
    "value": "1.35156036"
  }
}

[24/10/2017 06:54:37] GTC order '849788' will cancel order '849782'

= 849788
{
  "resultCode": "terQUEUED",
  "resultMessage": "Held until escalated fee drops."
}

[24/10/2017 06:54:38] Creating GTC order for Address: xxx - Sequence: 849790

= 849790
{
  "direction": "sell",
  "quantity": {
    "currency": "XRP",
    "value": "36222.61903100"
  },
  "totalPrice": {
    "currency": "BTC",
    "counterparty": "rchGBxcD1A1C2tdxF6papQYZ8kjRKMYcL",
    "value": "1.48028039"
  }
}

[24/10/2017 06:54:38] GTC order '849790' will cancel order '849784'

= 849790
{
  "resultCode": "terINSUF_FEE_B",
  "resultMessage": "Account balance can't pay fee."
}

Not my area of expertise, but I may be able to offer you some guidance.

The fee charged by the XRP Ledger is dynamic inside of a given ledger, not just from ledger-to-ledger.  As transactions are added to the ledger beyond a threshold the fee goes up exponentially.  Under unusual circumstances the fee has jumped as high as 16,000 XRP (see https://xrpcharts.ripple.com/#/metrics).  So an 85 XRP balance truly may not be enough to cover the required fee.

Here is more information on the dynamic nature of fees: https://ripple.com/build/transaction-cost/.  There are a number of network-wide and local-to-the-client metrics that determine the fee you must pay for your transaction to get onto the network.  In particular, submitting several transactions at once can increase the load on an individual ledger and cause the immediate fee to increase.  FWIW, the fee is not influenced by the order book you are using.

If I'm understanding your logs correctly, you are not specifying the maximum fee that you are willing to pay.  It might be prudent to specify that maximum, given the potential volatility of fees.

Do you understand that the error code returned when a transaction is submitted is not the final disposition of the transaction?  In particular, a ter code means the transaction will be automatically retried later.  So your log is not telling you whether the transaction actually got into the ledger.  To do so you'd want to see if the hash for that transaction is present in a validated ledger.  For robust transaction handling you cannot just send-and-forget a transaction; you must check back to locate the transaction in a validated ledger.

To prevent your transaction from getting stuck in an unknown state forever you should set an expiration time for the transaction using a LastLedgerSequence field.

Here is a thorough writeup on how to robustly submit transactions to the network: https://ripple.com/build/reliable-transaction-submission/.  Take it to heart.  Smart people have lost money if they have not followed all of the advice.

In summary, I believe all of the effects you are seeing are fully intended by the designers of the XRP Ledger.  Best of luck.

Hi Scott, first of all, thanks for taking the time for your thorough answer.
The first thing to make clear is the amount of XRP. It is not 85, but 85 thousand!  I find it unlikely that the fee rises to that level. As matter of fact, I have more bots running and this error only occurs on orderbooks dealing with XRP.
I do not set the max fee, but calculate the fee for every transaction, to a certain max level. The logs you see are from ripple-lib, they might differ a bit compared to the rippled json objects.
The error code is indeed 'ter', which has some side effects for the other trades I make (the sequence numbering becomes out of sync in my program.) I am aware that the transaction might be validated in the next ledger. That is not really an issue. I am placing orders as a market maker. If the order doesn't make it in this ledger then the next ledger is ok. And I replace older orders. And yes, I also use LastLedgerSequence field. But that is beside the point :-) no need to explain my program in all details.
Main issue is I think, that the amount of XRP in my wallet was 85 thousand, not 85 ...

The terINSUF_FEE_B error means that the transaction engine found some combination of pending or previous transactions that left your account balance insufficient to pay the fee. If you didn't impose any fee limit at all (never a good idea) it is possible that an instantaneous spike in what the server thought was the required fee caused the error.
Are you choosing the fee or are you letting the server choose it? If the latter, are you imposing any fee limits? You should never let the server choose the fee without imposing a fee limit. If you do, a server that has even momentarily poor synchronization to the network can recommend an absurdly high fee. The server relies on the client for fee sanity checking.
We do not recommend you let the server sign the transaction. And when you get an error like this, logging the exact transaction you submitted is helpful. If the fee is absurdly high, that's your problem. If the fee is low, then we know that for some reason the server thought your balance would be low, and we can investigate/troubleshoot that.

As said, I use ripple-lib, so it might be a little different then what you are used to in rippled. I set the fee as follows:
let prepared = await api.prepareOrder(address, order, { "sequence": mySequence, "fee": bots.fee } )
where bots.fee is a certain amount I set (max 0.2). Setting this fee sets the exact amount of fee to be used according to documentation. I could also set the maxFee next to this one, but it seems a little redundant, because of the exact setting of the fee.
The rest of my program is as follows:
const {signedTransaction} = api.sign(prepared.txJSON, secret[address]);
let result = await api.submit(signedTransaction)

I will see if I can capture the 'prepared' field next time in the log, hope it clears things up

That is odd. If you are specifying the exact fee and get terINSUF_FEE_B, that means that the server really did think that your XRP balance would be insufficient to pay the fee. That is the only condition that returns this exact message, unless ripple-lib issues it itself for some reason.
It could be that you had an offer to sell XRP and the server thought that a transaction would claim that offer. But offers being claimed by others can't push you below your reserve, and it seems that your reserve should have been enough to pay the fee on all outstanding transactions.

Problem is I can't put the log level of my rippled to 'trace' for a long time. But I can create more log from within my nodejs program. When it has occured again I'll be back with some more detailed log

I see what's going on. The balance predictor estimates the maximum cost of each transaction independently and doesn't fully take into account the fact that a transaction can't cost into the reserve. So if an offer is in the fee queue, it is assumed that it could be fully satisfied when it executes.

The key here is that the first two transactions are offers to sell XRP (if I'm reading this correctly), and they both got queued (terQUEUED).
Because those first two transactions are presumably still in the queue, when you send the third one, the queue checks that you have sufficient XRP to cover the maximum possible amount of XRP that the queued transactions could spend plus the fee of the third. Because you're offering to sell XRP, the queue assumes that your offers are completely taken when checking if more transactions are queued. Note this is only relevant to the queue, because we don't want to queue and broadcast a transaction that may or may not be able to claim its fee. Once the transaction gets out of the queue into the open ledger, then we know what the actual balance is likely to be, so further transactions can then be processed.
So 15000 + 36222.619031 = 51222.619031 plus whatever fees you're paying. Since there's no information in that output about the fees paid, I can't tell how close you're getting to your balance, but it does seem like you may be paying some pretty high fees.

@ximinez Yeah, that's what's going on. The maximum possible cost calculation is very worst case here and doesn't take into account the fact that the reserve is protected. I'm not sure whether it's worth adding code to fix that, but it's fairly simple and would not require an amendment.

ok. The amount was +/- 85000. When adding 15000 + 36222 + again 36222 then that would take me over the 85000. That was the pattern I thought to see occuring every time. The fees shouldn't be higher then 0.2 XRP
So I just checked my logs again (no added loginfo yet, sorrie) to find out it also occured on order for orderbook BTC->XRP which is not what I first expected, but I thought better share it maybe you do see logic in it :)
It concerns the following order :
[25/10/2017 07:57:33] BTC.Bitstamp_XRP - Create GTC Order: Sell 0.526465 BTC for 16413.303803 XRP(Price: 31176.438705, fee: 0.000012)

[25/10/2017 07:57:33] Creating GTC order for Address: xxx - Sequence: 854860

# 854860
{
  "direction": "sell",
  "quantity": {
    "currency": "BTC",
    "counterparty": "rvYAfWj5gh67oV6fW32ZzP3Aw4Eubs59B",
    "value": "0.52646500"
  },
  "totalPrice": {
    "currency": "XRP",
    "value": "16413.30380259"
  }
}

[25/10/2017 07:57:33] GTC order '854860' will cancel order '854851'

# 854860
{
  "resultCode": "terINSUF_FEE_B",
  "resultMessage": "Account balance can't pay fee."
}

And the other related orders sent just before that:
[25/10/2017 07:57:33] Creating GTC order for Address: xxx - Sequence: 854854

# 854854
{
  "direction": "sell",
  "quantity": {
    "currency": "BTC",
    "counterparty": "rvYAfWj5gh67oV6fW32ZzP3Aw4Eubs59B",
    "value": "0.40000000"
  },
  "totalPrice": {
    "currency": "XRP",
    "value": "11115.07814684"
  }
}

[25/10/2017 07:57:33] GTC order '854854' will cancel order '854845'

# 854854
{
  "resultCode": "terQUEUED",
  "resultMessage": "Held until escalated fee drops."
}

[25/10/2017 07:57:33] Creating GTC order for Address: xxx - Sequence: 854857

# 854857
{
  "direction": "sell",
  "quantity": {
    "currency": "BTC",
    "counterparty": "rvYAfWj5gh67oV6fW32ZzP3Aw4Eubs59B",
    "value": "0.52646500"
  },
  "totalPrice": {
    "currency": "XRP",
    "value": "14986.05999367"
  }
}

[25/10/2017 07:57:33] GTC order '854857' will cancel order '854848'
[25/10/2017 07:57:33] MGN:0.050000 IOC:28465.444035/5.00 GTC:27787.695367/0.40(3) BAL:0.526465 BID:D/26963.552610

# 854857
{
  "resultCode": "terQUEUED",
  "resultMessage": "Held until escalated fee drops."
}

The account contains 71000 XRP and 0.526 BTC.
Well, that might be not the whole story, as also other trades were made, where XRP was sold for something else. The complete list of orders for that account at that particular moment:
[25/10/2017 07:57:33] XRP_BTC.Fifth - Create GTC Order: Sell 15000.000000 XRP for 0.567135 BTC(Price: 0.000038, fee: 0.000012)
[25/10/2017 07:57:33] BTC.Bitstamp_XRP - Create GTC Order: Sell 0.400000 BTC for 11115.078147 XRP(Price: 27787.695367, fee: 0.000012)
[25/10/2017 07:57:33] XRP_BTC.Bitstamp - Create GTC Order: Sell 15000.000000 XRP for 0.567135 BTC(Price: 0.000038, fee: 0.000012)
[25/10/2017 07:57:33] XRP_BTC.Fifth - Create GTC Order: Sell 21220.103650 XRP for 0.821879 BTC(Price: 0.000039, fee: 0.000012)
[25/10/2017 07:57:33] BTC.Bitstamp_XRP - Create GTC Order: Sell 0.526465 BTC for 14986.059994 XRP(Price: 28465.444035, fee: 0.000012)
[25/10/2017 07:57:33] XRP_BTC.Bitstamp - Create GTC Order: Sell 21220.103650 XRP for 0.821879 BTC(Price: 0.000039, fee: 0.000012)
[25/10/2017 07:57:33] XRP_BTC.Fifth - Create GTC Order: Sell 21220.103650 XRP for 0.900154 BTC(Price: 0.000042, fee: 0.000012)
[25/10/2017 07:57:33] BTC.Bitstamp_XRP - Create GTC Order: Sell 0.526465 BTC for 16413.303803 XRP(Price: 31176.438705, fee: 0.000012)
[25/10/2017 07:57:33] XRP_BTC.Bitstamp - Create GTC Order: Sell 21220.103650 XRP for 0.900154 BTC(Price: 0.000042, fee: 0.000012)

So could be maximum cost calculation, which hit in this case a BTC->XRP order.
My work-around would be to keep track of how much XRP I place in an XRP sell order and then subtract that from the 'available XRP' for the next trade in that same ledger

That seems like a reasonable workaround if you're only doing 3 transactions at a time. Good luck!

@scottschurr thanks for the links anyways. As it turns out, the answer is in there: https://ripple.com/build/transaction-cost/#queued-transactions:

Queuing Restrictions
The rippled server uses a variety of heuristics to estimate which transactions are "likely to be included in a ledger." The current implementation uses the following rules to decide which transactions to queue:
Transactions must be properly-formed and authorized with valid signatures.
Transactions with an AccountTxnID field cannot be queued.
A single sending address can have at most 10 transactions queued at the same time. In order for a transaction to be queued, the sender must have enough XRP to pay all the XRP costs of all the sender's queued transactions including both the Fee fields and the sum of the XRP that each transaction could send. New in: rippled 0.32.0
If a transaction affects how the sending address authorizes transactions, no other transactions from the same address can be queued behind it. New in: rippled 0.32.0
If the transaction includes a LastLedgerSequence field, the value of that field must be at least the current ledger index + 2.

The question in this case is imo, should an order for trading XRP account the same as a definite sending XRP

@ximinez, thanks, I will implement the workaround then. Can you let me know the verdict, will there become a 'fix'?

Are you talking about @JoelKatz's fix described above? I'll look into it, and get something going, but the soonest any change could reasonably be released is 0.90.

Yes. No problem, it's not a critical issue for me, and I can work around. Fix only if you are sure it should be fixed and that the fix is correct.

That's how we do all of our fixes. 😀

@jnr101 Thank you for reporting this issue. It's been a while, but a fix is included in the just-release version 1.2.0

Thanks @ximinez!

You're welcome @jnr101!


not working correctly with C.H.I.P. computer

Very cool project! Unfortunately I am getting weird results when I try to use it with the C.H.I.P. computer. When I type, only sometimes do the characters appear on the screen, and not even the same characters work. It seems to depend on how fast I type and what the previous letter was that I typed. Even when I type fast the first character doesn't appear.
I assembled my ESP32 with a level shifter because I'm not comfortable giving it 5V signals directly. Any thoughts on what I can examine? I tried to enable debugging based on your instruction in the README.md, but I have no idea what SOMETHING should be and it won't compile.
I am hoping to use this to solve my need for a PS/2 keyboard for this CHIP computer, as I don't have that kind of keyboard. I don't have any other devices capable of using PS/2 to test with either, so I don't know if it's just this specific combination of devices or if it's something odd with my implementation of this, like the level shifter.

I experimented some more and it seems like it's just eating the first character of a series typed very fast. If you type slow, all characters are eaten. If you type a word really fast just the first letter is eaten. The timeframe a sequence of characters has to be typed successfully is extremely fast, then it eats the next character.  If you type the same character as the first character, even if very rapidly, it never appears.
I was able to type a word successfully if I type each character I want very fast after some other character to account for the first character not appearing. To get the enter key to work I have to do the same thing...type some other character and then the enter key very fast.
This seems like a timing issue of some sort, but no idea which end of the connection is responsible.
I also noticed the caps-lock light does not illuminate on my keyboard even though the characters are capitalized appropriately.

Hi there! Thanks for trying out my project!
Yes, it seems like a timing issue of some sort. At least it seems your system is accepting the module correctly as you can at least get some control codes to reach the host, so the setup at post is working.
First of all, we need to check if the software is doing some weird timing. For this, try to turn on debug messages so you can see the serial console and check for feedback as soon as you press a key. You'll be looking for two things:
1- That the code is receiving the key correctly (it should echo the exact key you pressed).
2- That the echo message appears as soon as you press the key with no delay.
This will discard any code problems. I experienced some timing issues and it was due to the FreeRTOS ticking rate (make sure you're using the provided config file that includes this parameter!).
In main.cpp there are many logging/debug function calls that provide the info you need. More specifically, at line 379 you have what you are looking for:
https://github.com/Hamberthm/esp32-bt2ps2/blob/e4298b657bce084e72884f15ce24aac141fd48a8/main/main.cpp#L379
This call will echo the USB HID code received from the keyboard in the serial monitor, before passing it to the keyboard.keyHid_send() function to be translated to the PS2 protocol Control Code.
For v0.4 I turned the verbosity level down to avoid spamming the serial console, as it is a pretty stable build. To be able to see this debug message, you need to elevate the ESP-IDF logging verbosity to "debug" level or higher.  This is done via the ESP-IDF Project Configuration page. For more info Google "ESP-IDF project configuration on Visual Studio Code" (I think it was done by typing "config" on the search bar on top of VS Code, enter the Project Configuration page, then search for the verbosity setting) or look here.
This is the easiest first step to try. Once we are sure the code is working OK with no lag, then we can start checking other things.
P.S.: Caps lock/num lock/scroll lock LED light up function is not supported yet and will be in the future. Caps should work regardless.
Kind regards
Humberto

Here is what I get in the monitor console:
Pressing the A key once:
D (270079) event: running post ESP_HIDH_EVENTS:2 with handler 0x400dbef8 and context 0x3ffd7aac on loop 0x3ffd7810
0x400dbef8: esp_ble_hidh_event_handler_wrapper at C:/Users/edge9/esp/v5.2.2/esp-idf/components/esp_hid/src/ble_hidh.c:621

D (270080) BTKeyboard: 00 04 00 00 00 00 00
D (270083) KEYS PASSED TO MAIN: : 00 00 00 00 00
D (270177) event: running post ESP_HIDH_EVENTS:2 with handler 0x400dbef8 and context 0x3ffd7aac on loop 0x3ffd7810
0x400dbef8: esp_ble_hidh_event_handler_wrapper at C:/Users/edge9/esp/v5.2.2/esp-idf/components/esp_hid/src/ble_hidh.c:621

D (270178) BTKeyboard: 00 00 00 00 00 00 00
D (270181) KEYS PASSED TO MAIN: : 00 00 00 00 00

Pressing A, then S in very rapid succession:
D (343626) event: running post ESP_HIDH_EVENTS:2 with handler 0x400dbef8 and context 0x3ffd7aac on loop 0x3ffd7810
0x400dbef8: esp_ble_hidh_event_handler_wrapper at C:/Users/edge9/esp/v5.2.2/esp-idf/components/esp_hid/src/ble_hidh.c:621

D (343627) BTKeyboard: 00 04 00 00 00 00 00
D (343630) KEYS PASSED TO MAIN: : 00 00 00 00 00
D (343675) event: running post ESP_HIDH_EVENTS:2 with handler 0x400dbef8 and context 0x3ffd7aac on loop 0x3ffd7810
0x400dbef8: esp_ble_hidh_event_handler_wrapper at C:/Users/edge9/esp/v5.2.2/esp-idf/components/esp_hid/src/ble_hidh.c:621

D (343675) BTKeyboard: 00 04 16 00 00 00 00
D (343678) KEYS PASSED TO MAIN: : 16 00 00 00 00
D (343683) BTKeyboard: Down key: 16
D (343725) event: running post ESP_HIDH_EVENTS:2 with handler 0x400dbef8 and context 0x3ffd7aac on loop 0x3ffd7810
0x400dbef8: esp_ble_hidh_event_handler_wrapper at C:/Users/edge9/esp/v5.2.2/esp-idf/components/esp_hid/src/ble_hidh.c:621

D (343725) BTKeyboard: 00 16 00 00 00 00 00
D (343729) KEYS PASSED TO MAIN: : 00 00 00 00 00
D (343733) BTKeyboard: Up key: 16
D (343775) event: running post ESP_HIDH_EVENTS:2 with handler 0x400dbef8 and context 0x3ffd7aac on loop 0x3ffd7810
0x400dbef8: esp_ble_hidh_event_handler_wrapper at C:/Users/edge9/esp/v5.2.2/esp-idf/components/esp_hid/src/ble_hidh.c:621

D (343775) BTKeyboard: 00 00 00 00 00 00 00
D (343779) KEYS PASSED TO MAIN: : 00 00 00 00 00

I think I see where your problem is.
It seems your keyboard is reporting six key bytes and the code is only registering 5. Then you get output only when you press two or more keys at the same time (the first one gets ignored). That's what really happening with "quick writing".
That's not my code right there but I can certainly fix it.
Please stand by and I will look at it today or tomorrow!

Thanks! Looking forward to trying out your fix.
I am using a Logitech MX Keys Wireless Keyboard, model YR0073. It's been an excellent keyboard for several years.

Looking at it, it seems the push_key function is being given a byte length of 5 as a parameter.
Could you please ramp up the debug verbosity level to "VERBOSE" and pass me the same log? I need to check what's happening to the length paramater.

Verbose log of same tests:
Pressing A once:
D (94164) event: running post ESP_HIDH_EVENTS:2 with handler 0x400dc578 and context 0x3ffd78c0 on loop 0x3ffd78a4
0x400dc578: esp_ble_hidh_event_handler_wrapper at C:/Users/edge9/esp/v5.2.2/esp-idf/components/esp_hid/src/ble_hidh.c:621

V (94164) BTKeyboard: f5:0e:80:7e:ba:de INPUT: KEYBOARD, MAP:  0, ID:   1, Len: 7, Data:
D (94171) BTKeyboard: 00 04 00 00 00 00 00
D (94176) KEYS PASSED TO MAIN: : 00 00 00 00 00
D (94314) event: running post ESP_HIDH_EVENTS:2 with handler 0x400dc578 and context 0x3ffd78c0 on loop 0x3ffd78a4
0x400dc578: esp_ble_hidh_event_handler_wrapper at C:/Users/edge9/esp/v5.2.2/esp-idf/components/esp_hid/src/ble_hidh.c:621

V (94315) BTKeyboard: f5:0e:80:7e:ba:de INPUT: KEYBOARD, MAP:  0, ID:   1, Len: 7, Data:
D (94322) BTKeyboard: 00 00 00 00 00 00 00
D (94326) KEYS PASSED TO MAIN: : 00 00 00 00 00

Pressing A, then S in rapid succession:
D (121090) event: running post ESP_HIDH_EVENTS:2 with handler 0x400dc578 and context 0x3ffd78c0 on loop 0x3ffd78a4
0x400dc578: esp_ble_hidh_event_handler_wrapper at C:/Users/edge9/esp/v5.2.2/esp-idf/components/esp_hid/src/ble_hidh.c:621

V (121090) BTKeyboard: f5:0e:80:7e:ba:de INPUT: KEYBOARD, MAP:  0, ID:   1, Len: 7, Data:
D (121098) BTKeyboard: 00 04 00 00 00 00 00
D (121102) KEYS PASSED TO MAIN: : 00 00 00 00 00
D (121165) event: running post ESP_HIDH_EVENTS:2 with handler 0x400dc578 and context 0x3ffd78c0 on loop 0x3ffd78a4
0x400dc578: esp_ble_hidh_event_handler_wrapper at C:/Users/edge9/esp/v5.2.2/esp-idf/components/esp_hid/src/ble_hidh.c:621

V (121165) BTKeyboard: f5:0e:80:7e:ba:de INPUT: KEYBOARD, MAP:  0, ID:   1, Len: 7, Data:
D (121173) BTKeyboard: 00 04 16 00 00 00 00
D (121177) KEYS PASSED TO MAIN: : 16 00 00 00 00
D (121182) BTKeyboard: Down key: 16
D (121217) event: running post ESP_HIDH_EVENTS:2 with handler 0x400dc578 and context 0x3ffd78c0 on loop 0x3ffd78a4
0x400dc578: esp_ble_hidh_event_handler_wrapper at C:/Users/edge9/esp/v5.2.2/esp-idf/components/esp_hid/src/ble_hidh.c:621

V (121217) BTKeyboard: f5:0e:80:7e:ba:de INPUT: KEYBOARD, MAP:  0, ID:   1, Len: 7, Data:
D (121225) BTKeyboard: 00 16 00 00 00 00 00
D (121229) KEYS PASSED TO MAIN: : 00 00 00 00 00
D (121234) BTKeyboard: Up key: 16
D (121267) event: running post ESP_HIDH_EVENTS:2 with handler 0x400dc578 and context 0x3ffd78c0 on loop 0x3ffd78a4
0x400dc578: esp_ble_hidh_event_handler_wrapper at C:/Users/edge9/esp/v5.2.2/esp-idf/components/esp_hid/src/ble_hidh.c:621

V (121267) BTKeyboard: f5:0e:80:7e:ba:de INPUT: KEYBOARD, MAP:  0, ID:   1, Len: 7, Data:
D (121275) BTKeyboard: 00 00 00 00 00 00 00
D (121279) KEYS PASSED TO MAIN: : 00 00 00 00 00

Unrelated to this issue, I found something today that confirms ESP32 is 5V tolerant on the inputs as long as the ESP32 is powered with 3.3V. I may experiment with removing the level shifter to simplify the circuit. Is the clock pin driven by ESP32 or by the host? Is the data pin bidirectional or driven by ESP32 or by the host?

Yes, I'm sure you'll have no problems at all deleting the logic level converter. You'll then have a pretty neat standalone, and small, device you can attach to your system.
I'm currently working on your bug and it seems to be that your keyboard does a special 7-byte key report.
May I ask for another log? This time pressing a modifier key like CTRL, ALT or SHIFT. Then at the same time a letter.
Like CTRL, and then CTRL+A
This is to be sure we are getting the correct report for the modifier key. In my Logitech K380 the reports are as follows:
V (2571533) BTKeyboard: f4:73:35:86:e2:5a INPUT: KEYBOARD, MAP:  0, ID:   1, Len: 8, Data:
D (2571541) BTKeyboard: 01 00 00 00 00 00 00 00
D (2571546) KEYS PASSED TO MAIN: : 00 00 00 00 00 00
D (2571551) BTKeyboard: Down key: LCTRL
This is a standard 8-byte report with the first byte containing the modifier (LCTRL), the second being undefined (always 0) and the 6 remaining cotaining the pressed keys.
Your keyboard seems to be using a different 7-byte report. I need to be sure the first byte is still being used for the modifier key.

It looks like first byte is the modifier and second byte is the regular key. I noticed if I press a third key while holding a modifier and a regular key the third byte is the third key. Based on what you explained, it seems like my keyboard is omitting the undefined byte you are used to getting in the 8-byte sequence.
Log with modifier key:
Control key pressed and released:
D (13119770) event: running post ESP_HIDH_EVENTS:2 with handler 0x400dc578 and context 0x3ffd78c0 on loop 0x3ffd78a4
0x400dc578: esp_ble_hidh_event_handler_wrapper at C:/Users/edge9/esp/v5.2.2/esp-idf/components/esp_hid/src/ble_hidh.c:621

V (13119770) BTKeyboard: f5:0e:80:7e:ba:de INPUT: KEYBOARD, MAP:  0, ID:   1, Len: 7, Data:
D (13119778) BTKeyboard: 01 00 00 00 00 00 00
D (13119783) KEYS PASSED TO MAIN: : 00 00 00 00 00
D (13119787) BTKeyboard: Down key: LCTRL
D (13119895) event: running post ESP_HIDH_EVENTS:2 with handler 0x400dc578 and context 0x3ffd78c0 on loop 0x3ffd78a4
0x400dc578: esp_ble_hidh_event_handler_wrapper at C:/Users/edge9/esp/v5.2.2/esp-idf/components/esp_hid/src/ble_hidh.c:621

V (13119895) BTKeyboard: f5:0e:80:7e:ba:de INPUT: KEYBOARD, MAP:  0, ID:   1, Len: 7, Data:
D (13119903) BTKeyboard: 00 00 00 00 00 00 00
D (13119907) KEYS PASSED TO MAIN: : 00 00 00 00 00
D (13119912) BTKeyboard: Up key: LCTRL

Control pressed and held, A pressed, A released, Control released:
D (13139471) event: running post ESP_HIDH_EVENTS:2 with handler 0x400dc578 and context 0x3ffd78c0 on loop 0x3ffd78a4
0x400dc578: esp_ble_hidh_event_handler_wrapper at C:/Users/edge9/esp/v5.2.2/esp-idf/components/esp_hid/src/ble_hidh.c:621

V (13139471) BTKeyboard: f5:0e:80:7e:ba:de INPUT: KEYBOARD, MAP:  0, ID:   1, Len: 7, Data:
D (13139479) BTKeyboard: 01 00 00 00 00 00 00
D (13139483) KEYS PASSED TO MAIN: : 00 00 00 00 00
D (13139488) BTKeyboard: Down key: LCTRL
D (13139796) event: running post ESP_HIDH_EVENTS:2 with handler 0x400dc578 and context 0x3ffd78c0 on loop 0x3ffd78a4
0x400dc578: esp_ble_hidh_event_handler_wrapper at C:/Users/edge9/esp/v5.2.2/esp-idf/components/esp_hid/src/ble_hidh.c:621

V (13139796) BTKeyboard: f5:0e:80:7e:ba:de INPUT: KEYBOARD, MAP:  0, ID:   1, Len: 7, Data:
D (13139804) BTKeyboard: 01 04 00 00 00 00 00
D (13139808) KEYS PASSED TO MAIN: : 00 00 00 00 00
D (13139994) event: running post ESP_HIDH_EVENTS:2 with handler 0x400dc578 and context 0x3ffd78c0 on loop 0x3ffd78a4
0x400dc578: esp_ble_hidh_event_handler_wrapper at C:/Users/edge9/esp/v5.2.2/esp-idf/components/esp_hid/src/ble_hidh.c:621

V (13139994) BTKeyboard: f5:0e:80:7e:ba:de INPUT: KEYBOARD, MAP:  0, ID:   1, Len: 7, Data:
D (13140002) BTKeyboard: 01 00 00 00 00 00 00
D (13140006) KEYS PASSED TO MAIN: : 00 00 00 00 00
D (13141219) event: running post ESP_HIDH_EVENTS:2 with handler 0x400dc578 and context 0x3ffd78c0 on loop 0x3ffd78a4
0x400dc578: esp_ble_hidh_event_handler_wrapper at C:/Users/edge9/esp/v5.2.2/esp-idf/components/esp_hid/src/ble_hidh.c:621

V (13141219) BTKeyboard: f5:0e:80:7e:ba:de INPUT: KEYBOARD, MAP:  0, ID:   1, Len: 7, Data:
D (13141227) BTKeyboard: 00 00 00 00 00 00 00
D (13141231) KEYS PASSED TO MAIN: : 00 00 00 00 00
D (13141236) BTKeyboard: Up key: LCTRL

Hi there, yesterday I worked on the issue for many hours. The problem seems to be the so called "Protocol Mode" as defined by the USB HID specification. It can either be "REPORT" or "BOOT". Report mode doesn't use a null byte in the key reports and BOOT mode is a fixed 8 byte with the second byte always zero. They did that for old BIOS compatibility or something.
I'm working in how to reliably identify the current mode the keyboard is using so to not break compatibility. It may take me a little longer.
In the meanwhile, I don't want to make you wait. You can try the following temporal fix on your build so you can test your machine. Please change the following lines in bt_keyboard.cpp:
https://github.com/Hamberthm/esp32-bt2ps2/blob/e4298b657bce084e72884f15ce24aac141fd48a8/main/bt_keyboard.cpp#L1098
Change to: uint8_t max = (size > MAX_KEY_COUNT + 1) ? MAX_KEY_COUNT : size - 1;
and
https://github.com/Hamberthm/esp32-bt2ps2/blob/e4298b657bce084e72884f15ce24aac141fd48a8/main/bt_keyboard.cpp#L1102
Change to: inf.keys[i] = keys[i + 1];

If I'm right, that should offset the byte count and include the missing one so you can test your machine and in the meanwhile, I can think on a more universal fix. If it still fails, come back with a log as usual.
Cheers!
Humberto

Thanks! I really appreciate your work on this. I just confirmed the temporary changes fix my situation and I am able to type normally now. Looking forward to seeing what you do for the protocol mode. I haven't dug into keyboard protocols before, so this is interesting.

Hi there!
I've been working on a new release. I implemented a workaround for the BOOT/REPORT mode issue.
Turns out Espressif does provide a function to identify the current protocol mode, but it's only supported on BLE and not on BT Classic. Also, there's no apparent way to know the protocol mode by examining the report parameters because that info doesn't get passed to the callback function on the user side.
Even more, when on BT Classic the stack auto-generates a fake entry on the Report Map for a supposed "BOOT" mode report with the exact same ID as the REPORT mode one, it just duplicates it and changes the mode.
Anyway, that's way too technical and into the stack's code so don't worry about it. The new push_key() function is as follows:
void BTKeyboard::push_key(uint8_t *keys, uint8_t size)
{
  uint8_t offset = 2; // default to BOOT mode report, second byte is always 0

  if (keys[1]) // some keyboards use REPORT mode and the second byte of the report is the first one
  {
    offset = 1;
  }

  KeyInfo inf;
  inf.modifier = (KeyModifier)keys[0];

  uint8_t max = (size > MAX_KEY_COUNT + offset) ? MAX_KEY_COUNT : size - offset;
  // inf.keys[0] = inf.keys[1] = inf.keys[2] = 0; // No need to do this?
  for (int i = 0; i < max; i++)
  {
    inf.keys[i] = keys[i + offset];
  }
  if (max < MAX_KEY_COUNT) // End flag in case our keyboard reports less keys than our max count
    inf.keys[max] = 0;
  ESP_LOG_BUFFER_HEX_LEVEL("KEYS PASSED TO MAIN: ", inf.keys, max, ESP_LOG_DEBUG);
  xQueueSend(event_queue, &inf, 0);
}

As you can see, I replaced the offset with a variable, so it dynamically checks if a key is reported on the second byte after the modifier, like in your case, and adjusts the offset accordingly.
That's not the ideal fix, because the second byte being reserved on BOOT mode doesn't forbid the manufacturers from putting a non-zero constant value on there and thus reporting a fake key, but I'm willing to bet that would be an extremely rare case. Most or nearly all the keyboards following BOOT mode will give a zero value there, I'm sure.
So, if you want to test it, here you have the new source code and a compiled binary for fast flashing and testing (Use FLASH.BAT). Pins are set to 22 and 23 like the code on the repository!
Binary with flasher: https://1drv.ms/u/s!AlKre4_rNmpJiO06mvjtyg04xExK7w?e=XfPOVz
Source: https://1drv.ms/u/s!AlKre4_rNmpJiPIjbqgqmQaozdPbtw?e=tK98xB
Keep in mind this will be a major release and there's changes on many files, not only this function.
Will leave this open for a while in case you come back with feedback :)
Thanks a lot!
Humberto

Great!! I will give it a try soon and let you know how it goes. I appreciate the details :)

I finally got a chance to try out your new code. I had to build from source because I am using different pins. Normal typing looks good, but I think the control key isn't working right. I am using this adapter on a homebrew computer kit and I don't remember for certain if control-C used to work but it seems like it did when I needed to break out of a BASIC program. The shift and caps-lock keys are working fine. I reviewed the debug output and the control and alt keys are signaling correctly from bluetooth. Maybe there is something odd with handling the key modifiers out to PS/2 in the latest changes?

Hi @ktownsend-personal !
I think in reality your 'C' key isn't working! That's a big bug I just discovered fixed today! It drove me really mad. Explanation is really long, but if you want it fixed right away please change the key section in main.cpp to:
                // KEY SECTION (always tested)
                // release the keys that have been just released
                for (int i = 0; i < BTKeyboard::MAX_KEY_COUNT; i++)
                {
                    if (!infoBuf.keys[i])
                    {                                                     // Detect END FLAG
                        break;
                    }
                    for (int j = 0; (info.keys[j]) && (j < BTKeyboard::MAX_KEY_COUNT); j++)
                    {
                        if (infoBuf.keys[i] == info.keys[j])
                        {
                            found = true;
                            break;
                        }
                    }
                    if (!found)
                    {
                        ESP_LOGD(TAG, "Up key: %x", infoBuf.keys[i]);
                        keyboard.keyHid_send(infoBuf.keys[i], false);
                        gpio_set_level(GPIO_NUM_2, 1);
                    }
                    else
                    {
                        found = false;
                    }
                }

                // press the keys that have been just pressed
                for (int i = 0; i < BTKeyboard::MAX_KEY_COUNT; i++)
                {
                    if (!info.keys[i])
                    {                                                   // Detect END FLAG
                        break;
                    }
                    for (int j = 0; (infoBuf.keys[j]) && (j < BTKeyboard::MAX_KEY_COUNT); j++)
                    {
                        if (info.keys[i] == infoBuf.keys[j])
                        {
                            found = true;
                            break;
                        }
                    }
                    if (!found)
                    {
                        ESP_LOGD(TAG, "Down key: %x", info.keys[i]);
                        keyboard.keyHid_send(info.keys[i], true);
                        gpio_set_level(GPIO_NUM_2, 0);
                    }
                    else
                    {
                        found = false;
                    }
                }

                infoBuf = info;                 // Now all the keys are handled, we save the state
                typematicLeft = typematicDelay; // Typematic timer reset
            }

The only changes are the conditions for the for loops:
(info.keys[j]) && (j < BTKeyboard::MAX_KEY_COUNT)
Basically, the array never gets fully initialized in the bt_keyboard.cpp module, it only gets populated with the pressed keys. As there may be garbage bytes already in memory, and my code was checking the full array until MAX_KEY_COUNT, it encountered those garbage bytes and interpreted them like already-pressed keys.
Now it checks only while there's a key code and not 0 (info.keys[j]), because the bt_keyboard module inserts a 0 end indicator when there's no more keys. This wasn't bein checked before.
Expect a big release before the weekend with mouse support and this bug fixed!
Cheers!

Interesting. I will give it a try. When I was trying it earlier the normal C character was appearing on the screen of the kit computer, just not modified by the control key.

No change. I just get the normal C character displayed instead of control-C.

Sorry for the delay, had to sleep a little 🤪
Well, that's really strange! I just tested it on my Toshiba Satellite 205CDS and LCTRL is working fine with CTRL-C/CTRL-V and so on.
Also, the modifier is a single byte sent always in front of the report. For LCTRL, the first bit is set so it should read 0x1 on the modifier byte with that key pressed alone.
It could be an initialization problem just like above but it already gets initialized. Anyway it is a single byte and it always get overwritten by the buffer save (infoBuf = info;) at the end of the key section. So if it were an initialization problem it should fail the first press but start working afterwards.
Anyway, just to be sure you can try changing the initialization setting at the beggining of the loop at line:
   ` info.modifier = infoBuf.modifier = (BTKeyboard::KeyModifier)0;`

And change the 0 to something like 2 or 3. But I'm not sure that's the problem.
Anyway, if you have time a Verbose log chunk like you posted above would be useful. Also, can you confirm previous builds were working for you?
Thanks!

I added the logging but not the initialization. Figured worth seeing what we get without fiddling first. It looks like it sees the modifier at this point in the logic.
D (37782) BTKeyboard: 01 00 00 00 00 00 00
D (37785) KEYS PASSED TO MAIN: : 00 00 00 00 00
D (37790) bt_keyboard: MODIFIER: 0x1
D (37794) BTKeyboard: Down key: LCTRL
D (39530) event: running post ESP_HIDH_EVENTS:2 with handler 0x400dd0c8 and context 0x3ffd79c4 on loop 0x3ffd7944
0x400dd0c8: esp_ble_hidh_event_handler_wrapper at C:/Users/edge9/esp/v5.2.2/esp-idf/components/esp_hid/src/ble_hidh.c:621

D (39530) BTKeyboard: 01 06 00 00 00 00 00
D (39534) KEYS PASSED TO MAIN: : 06 00 00 00 00 00
D (39539) bt_keyboard: MODIFIER: 0x1
D (39542) BTKeyboard: Down key: 6
D (39980) event: running post ESP_HIDH_EVENTS:2 with handler 0x400dd0c8 and context 0x3ffd79c4 on loop 0x3ffd7944
0x400dd0c8: esp_ble_hidh_event_handler_wrapper at C:/Users/edge9/esp/v5.2.2/esp-idf/components/esp_hid/src/ble_hidh.c:621

D (39980) BTKeyboard: 01 00 00 00 00 00 00
D (39984) KEYS PASSED TO MAIN: : 00 00 00 00 00
D (39988) bt_keyboard: MODIFIER: 0x1
D (39992) BTKeyboard: Up key: 6
D (40655) event: running post ESP_HIDH_EVENTS:2 with handler 0x400dd0c8 and context 0x3ffd79c4 on loop 0x3ffd7944
0x400dd0c8: esp_ble_hidh_event_handler_wrapper at C:/Users/edge9/esp/v5.2.2/esp-idf/components/esp_hid/src/ble_hidh.c:621

D (40655) BTKeyboard: 00 00 00 00 00 00 00
D (40659) KEYS PASSED TO MAIN: : 00 00 00 00 00
D (40663) bt_keyboard: MODIFIER: 0x0
D (40667) BTKeyboard: Up key: LCTRL

Good, as per the posted log it seems that LCTRL is being processed correctly and passed to the PS/2 module (Down and Push key logs appear).
So, maybe we have a (very strange) problem with PS2dev module and PS/2 commands.
May I suggest:

I tried LCTRL commands on v0.6 and they're working totally fine to me. Try it out and see if it gets solved: https://github.com/Hamberthm/esp32-bt2ps2/discussions/14

If it still doesn't work, try to edit a line at 'esp32-ps2dev.h':
Change  const uint32_t BYTE_INTERVAL_MICROS = 500;  to  const uint32_t BYTE_INTERVAL_MICROS = 200;. This will make byte intervals shorter for PS/2 commands, more faithful to the standard. It's set longer on purpose to try boost compatibility.

When possible, also try other systems to discard any particular problem to the one you are using.

I'm always here to help.
Humberto

cool, I will try out v0.6 later tonight or tomorrow. I wish I had another PS/2 host I could try but unfortunately I don't. That's really the whole reason I'm using this...because I don't have anything PS/2 except this CHIP computer kit and hence no PS/2 keyboard laying around LOL

I was just browsing the documentation for the computer you're using and came across this on the manual:
One thing to note: To stop a program press the "pause/break" key on the keyboard.
You mentioned above you were trying to use CTRL-C instead to break out of BASIC programs!
Maybe that's it?

Oh!  I forgot all about that! It must not handle control codes, which seems weird but certainly possible. Sorry for having you chase this non-bug!

Haha! Great! That's a relief!
No worries, you did more work than me on this one. Also big, big thanks to you! Your certainly helped me on the boot/report issue and collaborated to make the project even more universally compatible.
That system you're using seems like fun! I'm a collaborator at a local retro computer museum in my city, so I know you're into some good old "home computer" fun 😊
Once you tested it further, remember to turn down the logging to "Info" level, as debugging messages affect performance a lot.
Let me close this one so I can enjoy 0-issues for once haha!
Please reopen if issues with your system continue, and in case of any other.
Cheers!
Humberto

I didn't mind helping out at all... this project fits my needs perfectly and lets me avoid having a second keyboard floating around my office. Win-win!
I installed v0.6-beta and it's working great. I disabled boot pairing to get faster startup and my keyboard is operational in 10 seconds from power on now. It was quite a lot slower before. My ESP32 module doesn't have a boot button, so I'll have to remember to ground GPIO0 if I ever need to pair again.
It took me some time to figure out updating esp-idf to 5.3 since I have not used it for any of my personal projects (I use platform.io; check out my RetroPhone project). After I finally figured out how to update the version the build wouldn't work and wanted me to run idf.py fullclean, which had me stumped for a bit because I couldn't find any path where idf.py was recognized, but eventually found the fullclean command in the extension UI when I stopped banging my head on the console long enough to remember there is a UI. Toolchain woes LOL
I did a compare and saw how much you changed...quite a lot! I like the mouse option and multiple paring option. Can't really use it for my particular case, but good features. I went ahead and mapped the mouse pins to something sensible for my board in case I decide to add the extra mouse PS/2 cable in the future. I might try finding a PS/2 cable with all 6 wires and put all the signals on one cable, then use a mouse/keyboard splitter adapter for hardware that needs it.
Thanks for providing this project and all the time you put into it so people like me can benefit!

Thanks a lot for the feedback! Glad it worked!!  😁
Pretty interesting that phone project! Going to check it out for sure!


NEXT-5469 Pagination of properties in SW6 admin is missing

EN:
At the moment its not possible to list more than 25 properties in the SW6 Admin (Catalogues -> Properties) because the pagination of the properties is missing.
DE:
Aktuell ist es nicht möglich mehr als 25 Eigenschaften im SW6 Admin (Katalog -> Eigenschaften) zu listen, da die Paginierung der Eigenschaften hier fehlt.

interested

@tbkruse it seems like this was implemented by one of our developers. Maybe this ticket is da duplicate.
I'll close it then :)

Was this issue related to this one? https://github.com/shopware/platform/issues/971
It's on different parts of the UI but maybe the same mistake was made for the pagination?


Make changing dev server port easier

Probably shouldn't be in @config on second thought, since this might bleed to the app and be discoverable to the end user, which it probably shouldn't be.

Better to just use PORT=1234 npm run dev. A dev can easily set an alias for themselves if they want to do this often, probably good enough.


Update Archiving code page

[x] Review page to see if we can make this any clearer
[x] Fred approved page
[x] James approved page
[x] Naushin approved page
[x] Rebecca approved page
[x] Share page with Exeter if relevant

Definition of done

[ ] Team reviewed and approved page?
[ ] Add changelog to page
[ ] Assign Fred to update the Schematron if relevant

@fred-atherden Has the schematron been updated for the tests on this page?

@naushinthomson, please can you remind me what change this ticket is capturing?

@fred-atherden Sorry I missed this! I think some changes were made to the schematron test wordings (you can see the history here) and I was just wondering if this was reflected in schematron links etc.

Thanks @naushinthomson. I think we're all good.


Specify API version when making requests

Apparently you can prefix calls to the Remote API with version information... '/1.17/whatever'
We should take advantage of this to guard against Docker's ever evolving API.
Not sure if the docker-api gem makes this easy/possible.. simply making a note of it here.
-s

Good idea. This would help with keeping this cookbook stable through docker
upgrades.
I'll take a look at docker-api gem and see what we can do.
On Tuesday, November 10, 2015, Sean OMeara<EMAIL_ADDRESS>wrote:

Apparently you can prefix calls to the Remote API with version
information... '/1.17/whatever'
We should take advantage of this to guard against Docker's ever evolving
API.
Not sure if the docker-api gem makes this easy/possible.. simply making a
note of it here.
-s
—
Reply to this email directly or view it on GitHub
https://github.com/someara/chef-docker/issues/552.

Thanks!

Looked into the docker-api gem and it is already passing version 1.16 for the api calls.
https://github.com/swipely/docker-api/blob/master/lib/docker/connection.rb#L83
https://github.com/swipely/docker-api/blob/master/lib/docker/version.rb#L6

Welp.


Improve array loading times for large snapshots

In a large snapshot for customer: money there's significant time spent on array deserialization in the snapshot. This area seems like it could be improved to either do less work or do its work a bit faster.
From discussion with @a-siva and @rmacnak-google:

can the array of class and function references in the dictionary of the library object be eliminated (except for those classes, functions that have the entry point pragma)
consider placing arrays that only have references to RO Data objects in a special cluster and use offsets instead of ids in the reference table
consider placing arrays that only have smis in a special cluster and use the actual smi values instead of ids in the reference table

Some more assets are available at http://shortn/_09Yx1TBIBc (google only, sorry)

can the array of class and function references in the dictionary of the library object be eliminated (except for those classes, functions that have the entry point pragma)

Here is the breakdown of where arrays are coming from in terms of snapshot sizes. This was computed from heapsnapshot generated by gen_snapshot:
total: 1499344
%shared: 13 0.00%
ArtificialRoot.canonical_type_arguments: 15 0.00%
ArtificialRoot.canonical_type_parameters: 15 0.00%
ArtificialRoot.canonical_function_types: 15 0.00%
ArtificialRoot.canonical_types: 15 0.00%
ArtificialRoot.symbol_table: 15 0.00%
Class.functions_hash_table_: 1428 0.10%
Function.data_: 2810 0.19%
ArtificialRoot.symbol-table: 2819 0.19%
TypeParameters.names_: 6057 0.40%
FunctionType.named_parameter_names_: 7220 0.48%
ExceptionHandlers.handled_types_data_: 21562 1.44%
Class.functions_: 28170 1.88%
Class.fields_: 36647 2.44%
Class.interfaces_: 55533 3.70%
ArtificialRoot.libraries_map: 61223 4.08%
FunctionType.parameter_types_: 82887 5.53%
GrowableObjectArray.data_: 84556 5.64%
Library.dictionary_: 105438 7.03%
Class.invocation_dispatcher_cache_: 129244 8.62%
Class.constants_: 179990 12.00%
const Array: 693672 46.27%

Some of this is fairly trivial to prune:

Library.dictionary_ 105438 7.03% drops non entry-points entirely
GrowableObjectArray.data_: 84556 5.64% seems to come from Library.used_scripts_: 51471 99.98%. This field should just be dropped entirely in product AOT snapshots AFAIK.
Class.constants_ I think this should be possible to drop as well (at least have a mode for that). I think it should only be needed for canonicalization of arriving messages sent between two different isolate groups. Maybe we can just have a special flag which disables multiple isolate groups in Flutter applications? (This is not properly supported by tooling anyway). /cc @mkustermann @aam

Class.invocation_dispatcher_cache_: 129244 8.62%

This is caused by a bug which adds a lot of duplicated entries to these caches. I am going to fix this:
-[ ] https://github.com/dart-lang/sdk/issues/48914

Maybe it would be better to simply prune the constants: Exclude any constant based on user-defined class/function and any constant transitively referencing such? We can measure and see what % we could remove.
@aam Interested in looking at this?

Sure, created https://github.com/dart-lang/sdk/issues/48928 for that.

Related:

706e62a1bf9f8c11053330821155ca1feae95dcf - [vm] Canonicalize empty Class::interfaces_.
72c153b93084ead3e9d2079b2f3c9568d8fa2d09 - [vm] Keep snapshot cursor etc in registers during ReadFill.
5748add35c24f87cc271fd563e01a37d9a46f92e - [vm] Use a big-endian encoding for snapshot ref ids.

Here is the data for the GPay AOT snapshot from using the flag print_array_optimization_candidates
INFO: From Flutter, aot, snapshotting //java/com/google/nbu/paisa/flutter/gpay/app:android_release_android_library_flutter_artifacts_android_aot:
Arrays
  total:  17416,  221765 elements
  smi-only:264,  16179 elements
  ro-only:1986 ,  18883 elements
  empty:160
Arrays
  total:  61449,  289799 elements
  smi-only:0,  0 elements
  ro-only:1247 ,  1879 elements
  empty:13288
VMIsolate(CodeSize): 3180
Isolate(CodeSize): 6029491
ReadOnlyData(CodeSize): 9108896
Instructions(CodeSize): 23826104
Total(CodeSize): 38967671
VMIsolate(HeapSize): 16684
Isolate(HeapSize): 14336996
Total(HeapSize): 14353680

With https://github.com/dart-lang/sdk/commit/706e62a1bf9f8c11053330821155ca1feae95dcf - [vm] Canonicalize empty Class::interfaces_
the empty array references are gone reducing the number of array references by 13k
INFO: From Flutter, aot, snapshotting //java/com/google/nbu/paisa/flutter/gpay/app:android_release_android_library_flutter_artifacts_android_aot:
Arrays
total:  17416,  221765 elements
smi-only:264,  16179 elements
ro-only:1986 ,  18883 elements
empty:160
Arrays
total:  48161,  289799 elements
smi-only:0,  0 elements
ro-only:1247 ,  1879 elements
empty:0
VMIsolate(CodeSize): 3180
Isolate(CodeSize): 5963051
ReadOnlyData(CodeSize): 9108896
Instructions(CodeSize): 23826104
Total(CodeSize): 38901231
VMIsolate(HeapSize): 16684
Isolate(HeapSize): 14124388
Total(HeapSize): 14141072

Another related fix https://dart-review.googlesource.com/c/sdk/+/243709 (Trim user-defined entries from classes constants table)

https://dart-review.googlesource.com/c/sdk/+/242862 (Avoids duplicate entries in invocation dispatcher cache).

https://dart-review.googlesource.com/c/sdk/+/242864 (Removes used_scripts field from AOT product builds)
This reduces the Arrays count from 45359 to 37990.

https://dart-review.googlesource.com/c/sdk/+/244303 is the CL in progress to remove dictionary entries (The CL still has some failing tests that need investigation). A preliminary run of building the AOT snapshot including this CL shows the following stats
Arrays
total:  29632,  96654 elements


Row class

Adds the ability to set a class on the <tr> elements. This parameter also accepts a function, so you can for example highlight rows based on some logic.
Usage:
<table-component :data="data" :row-class="highlight">

data() {
	return {
		highlightedRow: undefined,  // update this using click, checkbox, or whatever
	};
},
methods: {
	highlight(row) {
		if (row.data) return this.highlightedRow === row.data.vueTableComponentInternalRowId ? 'highlight' : '';
		return '';
	},
}

Is there any reason this PR is not merged? This is also a useful feature.

This PR was replaced by #101, which was then auto-closed by a bot due to lack of time of the repository creators...


MMCA-4906 | Implement search error without search paramater on Cash Account dashboard

Task:

Add error validation for search input
Update errors when system is down/timeouts

You have conflicts with main, it'll be 4959 that's just merged earlier today.


Obtain OIDC certification for SDKs

Is your feature request related to a problem? Please describe.
OIDC Authentication SDKs need to be OIDC cetified.

This issue is being closed due to extended inactivity. Please feel free to reopen it if further attention is needed. Thank you for helping us keep the issue list relevant and focused!


update cs mapping configmap

What this PR does / why we need it:
Update mapping configmap template
common-service-maps.yaml:
- requested-from-namespaces:
  - cp4iforFord
  map-to-common-service-namespace: common-service-ford
map-default: ibm-common-servies

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #
Special notes for your reviewer:

Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

Release note:

I will create PR to common service operator after this PR is merged

I will create PR to common service operator after this PR is merged


MBT low-friction maintenance

Crate
primarily ibc
Problem Summary
We wish to avoid reaching a situation where we have to disable MBT tests because "no one understands what's happening there," and minimize the risk of these tests becoming easily outdated.
We can close this issue if we manage to answer this question: how can an ibc-rs developer do changes to the codebase without facing significant challenges with respect to MBT?
Context
MBT is progressing very nicely for modules (#701, #601).
Before adding additional code and features to this testing method, it would be helpful to understand how we plan to keep MBT aligned with the codebase. This is a concern because there are several major changes slated for the next few months (hopefully). Here are some:

add proof verification (which may need an actual IAVL/kv store)
introduce capabilities & permissions (#406)
refactor error-handling (see the concrete example below)
add more clients (e.g., for solo machine #705)
the "modifying handshake" Hermes user story (no issue yet for this)

Concrete example
To frame the discussion in a concrete example, we can look at error handling and how the codebase will (probably) evolve in that respect. Not sure if this is the best example, but its one of the simplest..
For refactoring error-handing in the modules, the discussion in #68 provides a good idea of what we'd like to do (see in particular this comment) and almost did (#493 but it was too much work, and not a priority at that point).
A typical modification here involves changing error types. An error is currently an anomaly::Error<Kind>, so we'd like to throw away the anomaly part and retain just the enum Kind, which we can rename into enum Error. Consequently we would change the return types pretty much everywhere, for instance:
@@ -496,13 +496,13 @@ impl ClientDef for AnyClient {
         client_id: &ClientId,
         proof: &CommitmentProofBytes,
         client_state_on_counterparty: &AnyClientState,
-     ) -> Result<(), Box<dyn std::error::Error>> {
+     ) -> Result<(), Error> {
         match self {
             Self::Tendermint(client) => {
                 let client_state = downcast!(
                     client_state => AnyClientState::Tendermint
                 )
-                 .ok_or_else(|| Kind::ClientArgsTypeMismatch(ClientType::Tendermint))?;
+                 .ok_or_else(|| Error::ClientArgsTypeMismatch(ClientType::Tendermint))?;

Question: would a modification as above break MBT, because the tests assume specific error types, names? If yes, we should try to make it so that it's relatively simple to get the tests to run again.

For Admin Use

[X] Not duplicate issue
[X] Appropriate labels applied
[X] Appropriate milestone (priority) applied
[X] Appropriate contributors tagged
[X] Contributor assigned/self-assigned

That is a great question to ask, Adi (@adizere)! I completely share your concerns. Here is my current view:
No code works without tests. Unit tests are a must, and It seems to be a consensus within Tendermint-rs & IBC-rs devs that property-based tests are also needed (see TM-rs#792, TM-rs#709, or TM-rs#815). Let's work from this assumption.
To properly write both unit tests and PB-tests, the key, imho, is abstraction -- the ability to generate your concrete datastructures from abstract inputs, with the possibility to omit a lot of details, and bring them back only when needed. Testgen is a good example of such abstraction for TM-rs types. It probably should live closer to the datastructures but that's a different story. Another example of abstraction you may find in Vitor's PR #701, see executor/mod.rs. Interestingly, PB-tests also require this to write good strategies, so let's agree that abstraction is needed.

Those two are very important to have in any project -- they will warn you in case if what you write fundamentally breaks something. In particular, these tests will tell you "sorry, you error type has changed, I will not compile".
But as you can see from PR #701, it is exactly this abstraction that we need also for MBT! And it is the key component to enable it. Everything else can be automated away, and we are working towards such automation. The first version will be available by the end of Q1, and by the end of Q2 we will have a more or less complete one for Rust.
Here is a preview of what we'll have. MBT will consist of

MBT-core, that encapsulates as much as possible of the target-language independent functionality, e.g. manipulating the models, or generating, storing, and navigating the tests.
Plugins for target languages; the first one is going to be MBT-Rust. Those will be quite thin layers that interact with MBT-core, and only execute the provided tests in the target language.

Both MBT-core and MBT-Rust will be crates and executables external to IBC-rs, so no code to maintain here. What will be in IBC-rs:

abstractions: those are also needed for unit- and PB-tests; MBT will simply reuse them. We will help to formulate the interfaces such that the reuse will happen with least friction.
TLA+ model & TLA+ tests: those need to be manually written, but we are going to have some degree of automation here. In particular a lot of boilerplate TLA+ will be taken away from you.
MBT config: a pretty minimal configuration, which will tell MBT what abstractions, models and tests to use, and what to test based on them.

So, you see, the maintenance should be low. As long as your unit- and PB-tests work, MBT will also work. As for the current PR #701, it is still within the old approach when all code lives in the repo, but this will change after Q1, when we release the first version of MBT-core + MBT-Rust. I would just go ahead and merge it, with the understanding that we will refactor it in around a month from now.

Here are a couple of diagrams.
Old-style MBT

New MBT


[Feature request] Add AppleTV+ URL scraping

Seems like there's AppleTV and there's AppleTV+ that looks similar but isubrip doesn't work with the latter, and I couldn't get a working iTunes URL from the page source either. Please see the example below:
https://tv.apple.com/ca/movie/cha-cha-real-smooth/umc.cmc.6loo9fah7vwj2zpp7ehactlpd
Maybe it's not related to iTunes in any way and therefore not doable, no idea.

There's an iTunes link in the page source which can be extracted.
def get_itunes_url(appletv_url):
    req = requests.get(url=appletv_url)
    search = re.search('itmss://(itunes.apple.com[^"]+)"', req.text)
    if search:
        return 'https://' + search[1].strip('\\')


can you provide finetuning code? error

the original finetune_on_pregenerated.py from hugging face/pytorch-transformers repository actually has some bugs:
optimizer = AdamW(optimizer_grouped_parameters,
lr=args.learning_rate,
warmup=args.warmup_proportion,
t_total=num_train_optimization_steps)
actually "warmup,t_total" shouldn't be used.
Besides, I first run "Pregenerating training data", then the "Training on pregenerated data" as the readme from "huggingface," said. but the final files didn't improve the performance
from the code in the original finetune_on_pregenerated.py：
"      output_model_file = os.path.join(args.output_dir, WEIGHTS_NAME)
output_config_file = os.path.join(args.output_dir, CONFIG_NAME)
torch.save(model_to_save.state_dict(), output_model_file)
model_to_save.config.to_json_file(output_config_file)
tokenizer.save_vocabulary(args.output_dir)

"
the final finetuning model files are different from your provides;

can you provide finetuning code? appreciated.

Hi.
Thanks for your request.
I will have a look at it.

Hey @zdlpsina
I could reproduce your error. It should work now:
Check it out:
https://github.com/deepopinion/domain-adapted-atsc/tree/master/finetuning_and_classification
I also checked the final files, they should be the same as in the models I released.
I will close this issue, let me know if everything works for you.