does hystrix acquire some resources to start up it’s thread pool ?

we have set the minimum thread pool size as 20 and Max is 100.
but at the start of request processing , we are getting below exception:
The first few exceptions said:
Task java.util.concurrent.FutureTask@1ad6194b rejected from java.util.concurrent. ThreadPoolExecutor@6a085fce[Running, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 0]
next few exceptions were about
GroupService:GetGroupMembers fallback execution rejected
And finally, the circuit was opened and we started getting these errors:
Hystrix circuit short-circuited and is OPEN
we are confused as even though we had core size is 20, but actual pool size was 1 in the exception logs.

The first command to execute in a threadpool will set it up - there shouldn't be a period where it's set up improperly.
That log makes it appear like the threadpool is only configured with a single thread.  How are you configuring it?  If you have the config stream enabled (https://github.com/Netflix/Hystrix/wiki/Metrics-and-Monitoring#configuration-stream), can you post the relevant output of how this threadpool is configured?

We have configured thread pool with multithreading behaviour. As of now I am providing hystrix.stream data for thread pool , please let me know if this would be fine to have regarding this issue?
             data: {"type":"HystrixThreadPool","name":"group-service-thread-pool","currentTime":1493747155308,"currentActiveCount":0,"currentCompletedTaskCount":97065375,"currentCorePoolSize":20,"currentLargestPoolSize":100,"currentMaximumPoolSize":100,"currentPoolSize":20,"currentQueueSize":0,"currentTaskCount":97065375,"rollingCountThreadsExecuted":1616,"rollingMaxActiveThreads":5,"rollingCountCommandRejections":0,"propertyValue_queueSizeRejectionThreshold":5,"propertyValue_metricsRollingStatisticalWindowInMilliseconds":10000,"reportingHosts":1}

@JagmohanSharma , are you initiating a significant number of requests before the thread pools are "warmed"" up?

@bltb Yes we are handling good amount of requests with hystrix protected way. And this issue occur when we deploy a new version of our service and activate that. So it gets all requests in a good numbers at that moment and throws these error for period of 4 seconds. and after that things get normal with thread pool.

@JagmohanSharma, when you say "good numbers", how many requests are you
receiving in 4 seconds?

@bltb  it was around ~150 rps on a single node when we activated the new version for receiving the requests.

I just ran a test which generally simulates your case: thread pool with min=20/core = 100 and then I gave it 100 concurrent calls.
1494370006900 : main constructing the pool to launch commands from concurrently
1494370006902 : main about to launch the 100 commands concurrently
1494370006920 : main about to start the await
1494370007150 : hystrix-Unit-7 starting
1494370007150 : hystrix-Unit-2 starting
...
1494370007164 : pool-1-thread-99 done with 98
1494370007162 : pool-1-thread-92 done with 91
1494370007164 : main done with await

is some output from a sample run on my Macbook Pro.
The implication is that it took around 250ms to get all of the Hystrix data structures set up for the first command execution, but there was no cost beyond the first request.
If you're able to provide a similar test that demonstrates your issue, that'd be valuable, as I can't currently replicate the symptoms you've described.

@mattrjacobs this can be reproduced in hystrix enabled services. As when I start the server and just hit the endpoints which are protected by hystrix with these properties for commandKey
hystrix.threadpool.getData.coreSize=20
hystrix.threadpool.getData.allowMaximumSizeToDivergeFromCoreSize=true
hystrix.threadpool.getData.maximumSize=100.
I got exceptions like below
2017-05-25 18:05:09.282 ERROR 20543 --- [io-8080-exec-26] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is com.netflix.hystrix.exception.HystrixRuntimeException: getData could not be queued for execution and fallback failed.] with root cause
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5aedf205 rejected from java.util.concurrent.ThreadPoolExecutor@189572bb[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
and next few request got failed with this.
2017-05-25 18:05:09.283 ERROR 20543 --- [io-8080-exec-16] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is com.netflix.hystrix.exception.HystrixRuntimeException: getData could not be queued for execution and fallback failed.] with root cause
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@448b9a6e rejected from java.util.concurrent.ThreadPoolExecutor@189572bb[Running, pool size = 3, active threads = 3, queued tasks = 0, completed tasks = 0]
I think when we just start the server under very high load (200+ req/sec), the deployed instance of service doesn’t get enough time to initialise its thread thus leading to this issue.

does my added information make sense ? or it is confusing and perplexed after reading this?

@mattrjacobs In these situation like bursting of requests as we can use queue or increase the pool(maximum pool size) , do we have any other option to accommodate this by Hystrix thread pool?
As Queues can also be used in cases when burst of requests need to be accommodated which is also being used at netflix in these kind of usecases. But do we have any other option or planned enhancement for handling such pre initialisation of hystrix thread pool if required?

We use queues very sparingly (I believe in 1 thread pool out of ~100) in the application I help operate.
One idea would be to build a version of Hystrix that prestarts all core threads and see how that performs.

This discussion seems similar to #1596 and also https://github.com/Netflix/Hystrix/issues/1554#issuecomment-298792781.
Having done some reasonably in-depth experiments using "queues" and Hystrix; it was discovered that, in order to handle bursty traffic, the Apache Tomcat ThreadPoolExecutor implementation worked better than the Java default. We just extended HystrixConcurrencyStrategy.

Thanks for the tip @bltb !  Will point people towards that if that comes up again

you can see JDK ThreadPoolExecuter.addWorker(Runnable firstTask, boolean core) method.
because:

reject is using ctl
But print ThreadPool.toString is using workers.size()

ctl increase happend before workers.add
So it's just a concurrency issue when printing ThreadPool.
i think:
The essence of the problem is that the performance of the application is poor when it is first started, because various resources are not ready.
You can try to warm up the application

@mattrjacobs, we had similar concerns on how Hystrix handles bursty traffic beyond coreSize threads without a queue/buffer. Assuming a scenario where queueSize (10) is full and maximumSize (100) is large with no queueSize (0), does Hystrix gracefully handle all requests upto maximumSize or reject any at all while new thread creations take time? Was looking for a test case around this scenario.


FZ editor: splitted zones inaccurate positioning

There is a vertical shift from the splitter position and the actual new zone location:
Position the slipper near the editor dialog as a reference:

The result is shifter vertically:

Same problem with vertical splitter, in that case the result is shifted on the left.

Fixed in v.0.17 https://github.com/microsoft/PowerToys/releases


bug in parse-mongo-url causing promised-mongo unusable

A bug in readPreference value detection in parse-mongo-url, fixed at https://github.com/sorribas/parse-mongo-url/pull/2 but not merged, is rendering promised-mongo unusable.
Any workarounds without forking?
Update: This is not fixed in the underlying repo. I have sent a https://github.com/gordonmleigh/promised-mongo/pull/33 with the updated package.json

Is that causing "auth failed" errors with Mongo 3.0? I'm currently ripping my hair out trying to figure out why my production servers are all crashed. :P

nope.. it doesn't even connect ... citing something like "Invalid readPreference value"


Add abstraction of requests and responses in Relic

Relic is Serverpod's built-in web server. Adding an abstraction layer on top of dart:io could be beneficial for a number of reasons. It would make it easier to work with Relic and make the interface future-proof if we want to move away from dart:io.
The shelf package has done a pretty good job with this, but it has some shortcomings. We could create a new package based on shelf (to preserve its licence, and give them proper credit). We can then tighten up some of its shortcomings. The new package could be named serverpod_relic_helpers or if we come up with a better name. From the shelf package we can copy and modify the following classes: Message, Request, Response, and Body.
The most significant issues with the types from shelf are that they are largely untyped (my guess because of legacy), and they are using List<int> to represent data (which is much less efficient than using Uint8List). Adding type-safety and using Uint8List will improve performance and memory usage (especially for large requests).
Modifications to the shelf Body class
Instead of having a generic constructor that takes any type of object, we could have three distinct, typed constructors:
Body.fromString(String body, {Encoding? encoding});

Body.fromDataStream(Stream<Uint8List> body);

Body.fromData(Uint8List body);

The underlying stream can be accessed through either the read method, which should return a Stream<Uint8List> (instead of the Stream<List<int>>) or readAsString, which decodes the full stream into a String.
Modifications to the shelf Request class
We can take the opportunity to add the parameters that are parsed by the new Router (#2489) to the Request class, making them easy to access:
Map<String, String> pathParameters;

Modifications to Relic's Route class
To support the new abstractions, the handleRequest method of the Route class can be structured like this:
Future<Response> handleRequest(Session session, Request request);

My understanding is that the biggest shortcoming of shelf is that it does not provide type-safety and uses List<int>´ instead of Uint8List`.
How big of an undertaking is it to introduce these changes? Is there any possibility for us to add these changes as contributions directly to shelf instead of creating a fork?
We kind of end up needing to copy or mimic the classes regardless if we want an abstraction layer, but it could help us make it smaller.


Add JSON to result file. closes #635

Including tests and docs. the option name is --jsonReporter and the API is very similiar to --XUNIT
UPDATE this closes #647 and not #635 . but I can't fix that from my side. sorry for the typo
UPDATE: fixed to implement @n1k0 suggestions about code style etc, Also more DRY.

@alonisser any updates on this PR?

Archiving PR using ARCHIVE label as CasperJS, like PhantomJS, is not actively maintained for now. These PRs are from 1-5 years ago but there just hasn't been maintainers to continue the work.


Help - Writing Floats to 16bit integers

I'm submitting a ...  (check one with "x")
Support/Advice

support/advice request => Please do not submit support requests here, instead use the link to our support page, please!

Feature

( ) feature request

What is the motivation / use case for changing the behavior?

Simple question to get some hint

( ) simple question => please visit the Wiki, the YouTube tutorials and the Info tab by clicking on the node inside Node-RED before
(x) help wanted => you read all information and watched relevant YouTube tutorials, but you need more help

Problem

( ) bug report => please search github for a similar issue or PR and test Modbus problems with the modbus-serial package before submitting
( ) unexpected behavior => please visit the Wiki before

Current behavior

Expected behavior

Minimal reproduction of the problem with instructions

Your Environment
Please tell us about your environment:

(x ) I am using just the Node-RED package and got modbus-serial package installed with it
( ) I have installed modbus-serial package global from source (using git clone)
( ) I have installed modbus-serial as a global package (using npm i -g)

Yes, modbus-serial is the name but it does all, TCP and Serial.
The node-modbus package is just for the Modbus Server node.

Device :

( ) macOS: v
( ) Linux: v
( ) Windows: v
(x ) Raspbian: v
( ) Other:

Name:
Version:

Description of the connecting Modbus system:

Name   :
Version:
Vendor :
Link   : https://

node-red-contrib-modbus version: v5.13.3

Node: (min. 10.x)
node --version = v12.X.X

Help - I'm using the server as a Modbus slave and populating the registers via various systems (some fixed values) which is then queried by a Tridium Jace8000 (acting as Modbus Master). One of the values is a large number (1766229), so I have the flow below converting it to 16bit and writing to 2 registers.
When I read these registers in the Modbus master, the resulting float value is incorrect. The value drifts as it gets larger. For instance, at 10, it is correct, at 100, the Modbus master reads 100.4 and at 1776229 it reads 1767680.
I cant figure out if the problem is at the node-red end or at the Modbus master.
Thanks
[{"id":"1dd9ce49.b1647a","type":"function","z":"80b83cac.1c4e7","name":"","func":"function float32ToUint16(value) {  \n  var buffer = new ArrayBuffer(4);\n  var intView = new Uint16Array(buffer);\n  var floatView = new Float32Array(buffer);\n\n  floatView[0] = value;\n \n\n  return [intView[0], intView[1]];\n}\n\n//var incoming = JSON.parse(msg.payload);\n\n//Compose ModBus message (Holding Registers) READ/WRITE\nvar modbusHR = {\n payload : {\n 'value' : [\n float32ToUint16(msg.payload)[0], \n float32ToUint16(msg.payload)[1],\n ],\n 'fc' : 16,\n 'unitid' : 1,\n 'address' : 4,\n 'quantity' : 2\n }\n};\n\nnode.send([modbusHR]);","outputs":1,"noerr":0,"initialize":"","finalize":"","x":460,"y":180,"wires":[["4c60cf8d.36e49","447eebe8.03b0fc"]]},{"id":"447eebe8.03b0fc","type":"debug","z":"80b83cac.1c4e7","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":610,"y":240,"wires":[]},{"id":"4c60cf8d.36e49","type":"modbus-flex-write","z":"80b83cac.1c4e7","name":"","showStatusActivities":true,"showErrors":true,"server":"1dd4b889.c286f7","emptyMsgOnFail":false,"keepMsgProperties":false,"x":710,"y":180,"wires":[[],["ee02d018.f56d8"]]},{"id":"ee02d018.f56d8","type":"modbus-response","z":"80b83cac.1c4e7","name":"","registerShowMax":20,"x":970,"y":180,"wires":[]},{"id":"da2856fb.2af07","type":"inject","z":"80b83cac.1c4e7","name":"1766229","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"1766229","payloadType":"num","x":260,"y":180,"wires":[["1dd9ce49.b1647a"]]},{"id":"1dd4b889.c286f7","type":"modbus-client","z":"","name":"local","clienttype":"tcp","bufferCommands":true,"stateLogEnabled":false,"queueLogEnabled":false,"tcpHost":"<IP_ADDRESS>","tcpPort":"10502","tcpType":"DEFAULT","serialPort":"/dev/ttyUSB","serialType":"RTU-BUFFERD","serialBaudrate":"9600","serialDatabits":"8","serialStopbits":"1","serialParity":"none","serialConnectionDelay":"100","unit_id":1,"commandDelay":1,"clientTimeout":1000,"reconnectOnTimeout":false,"reconnectTimeout":2000,"parallelUnitIdsAllowed":false}]

// Use Uint16 Registers to Write your Variables.
var dim     = 10;  // Size of Registers you want to Write on Modbus Buffer
var Uint16  = new Uint16Array(dim);
var Uint8   = new Uint8Array(dim*2);
// Uint16 Assignation
Uint16[0] = Var1;
Uint16[1] = Var2;
Uint16[2] = Var3;
// Pass UInt16 Register to Uint8 Registers
for(var i=0; i<dim*2; i+=2){
Uint8[i]    = Uint16[parseInt(i/2)]>>8;
Uint8[i+1]  = Uint16[parseInt(i/2)];
}
// Assign payload
msg.payload = {
'value': Uint8,
'register': 'holding',
'address': 0
};
return msg;

Hi, please give Buffers a try to fill them first and then create y new UINT16Array from that Buffer ;)
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Typed_arrays
https://nodejs.org/en/knowledge/advanced/buffers/how-to-use-buffers/


TypeNotFound suppressions

Hello,
Since version 1.17.1, TypeNotFound errors are not thrown but returned as warnings (https://github.com/PowerShell/PSScriptAnalyzer/pull/957), which is very nice.
But is there a way to suppress them?
I tried adding a SuppressMessage attribute:
[Diagnostics.CodeAnalysis.SuppressMessageAttribute("TypeNotFound", "", Justification = "")]

And also excluding this rule in the settings file:
ExcludeRules = @('TypeNotFound')

But it's not working.
Thank you

Initially, I wanted to just log a warning but during the PR it was suggested/decided to rather return an object of type DiagnosticRecord to still alert the user about it in some way. It is therefore a bit unfortunate/misleading because it is not a rule per se and can therefore not be suppressed. However, because it is PowerShell you can filter out those DiagnosticRecords from the list of returned results. I myself am not 100% happy with this solution but this is what it is at the moment

Ok that's what I though but I wasn't completely sure.
I agree that ideally we should be able to treat it like any other rule violation.

Thank you for your quick answer!


chore(release): 28.0.3

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 94.792%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
1120

Relevant Lines:
1147

💛  - Coveralls


light weight backbone

would you support some light weight backbones such as RepVGG which is GPU friendly?

Sure thing! We hid it inside backbone.py but now we're bringing it to light (command-line). Use timm to replace your backbone. Try
python train.py -p bdd100k -bb repvgg_b0
This is basically what we do under the hood:
if backbone_name:
            self.encoder = timm.create_model(backbone_name, pretrained=True, features_only=True, out_indices=(2,3,4))  # P3,P4,P5

Hello, after switching to repvgg_b0, the following error occurs：

@datvuthanh @xoiga123

@ForestWang Have you encountered the same problem?

I have the same problem @MartinPR307, any suggestion to solve that @datvuthanh @xoiga123.
Thank you.


Oopsy anonymous mode

A way to possibly change names to just the job, or maybe initials of the player.

What's the goal of initials instead of names? I'm not sure I quite follow what the underlying intent is here.

To have it on stream, but to not outright name people. Mainly for memeing

The goal of oopsy is to make information more obvious so it's faster to analyze what's going wrong, not to trash on people for messing up.  I think this is not something I plan on implementing, and so am going to close this feature request.  Sorry!


Documentation: Refer back between name and initial-cluster options

This is a common problem I've seen in IRC, people don't match their name to the value in initial-cluster when using static bootstrapping. This should help those who are referring back to the documentation when setting up a cluster.

@robszumski How's this look?

LGTM.
/cc @yichengq

LGTM. I just linked the "static clustering" text to the clustering example doc.


TPU VM delete

Environment details

OS type and version: linux and macOs
Python version: 3.9.7
google-cloud-tpu version: 1.3.2

Steps to reproduce
After creating a TPU VM thanks to https://stackoverflow.com/a/71244709/7433183
there is an issue when deleting it through the same python library.
(previously mentioned here: https://www.googlecloudcommunity.com/gc/AI-ML/Create-TPU-Node-Malformed-Name/td-p/397566)
Code example
NAME = f"projects/{manifest.tpu.gcpProject}/locations/{manifest.tpu.gcpZone}/nodes/{manifest.name}"

client = tpu_v2alpha1.TpuClient()

request = tpu_v2alpha1.DeleteNodeRequest(
    name=NAME,
)

# Make the request
operation = client.delete_node(request=request)

logging.info("Waiting for operation to complete...")
response = operation.result()

Stack trace
The TPU VM is successfully deleted, but the python code eventually fails:
Cleaning TPU
Waiting for operation to complete...
Traceback (most recent call last):
File "/argo/staging/script", line 29, in <module>
response = operation.result()
File "/root/.local/lib/python3.9/site-packages/google/api_core/future/polling.py", line 132, in result
self._blocking_poll(timeout=timeout, **kwargs)
File "/root/.local/lib/python3.9/site-packages/google/api_core/future/polling.py", line 110, in _blocking_poll
retry_(self._done_or_raise)(**kwargs)
File "/root/.local/lib/python3.9/site-packages/google/api_core/retry.py", line 283, in retry_wrapped_func
return retry_target(
File "/root/.local/lib/python3.9/site-packages/google/api_core/retry.py", line 190, in retry_target
return target()
File "/root/.local/lib/python3.9/site-packages/google/api_core/future/polling.py", line 88, in _done_or_raise
if not self.done(**kwargs):
File "/root/.local/lib/python3.9/site-packages/google/api_core/operation.py", line 170, in done
self._refresh_and_update(retry)
File "/root/.local/lib/python3.9/site-packages/google/api_core/operation.py", line 159, in _refresh_and_update
self._set_result_from_operation()
File "/root/.local/lib/python3.9/site-packages/google/api_core/operation.py", line 130, in _set_result_from_operation
response = protobuf_helpers.from_any_pb(
File "/root/.local/lib/python3.9/site-packages/google/api_core/protobuf_helpers.py", line 65, in from_any_pb
raise TypeError(
TypeError: Could not convert Any to Node

Hello @remidebette,
Thanks for reporting this issue and providing a stack trace. I've reached out to the API team to propose an update to the proto source file for the python client library. Googlers see cl/436800147. I'll update this issue once I have more information.

The internal bug for this issue is b/226550050.

I'm going to close this bug as the issue is on the backend and there are no improvements for the client library itself. Googlers see the internal bug b/226550050 for progress updates.

Hi,
I am still seeing the same issue with the google-cloud-tpu lib at the latest version 1.5.1
W
2022-10-05 15:58:33,820 [INFO] Cleaning TPU
2022-10-05 15:58:34,491 [INFO] Waiting for operation to complete...
Traceback (most recent call last):
  File "/argo/staging/script", line 29, in <module>
    response = operation.result()
  File "/usr/lib/python3.9/site-packages/google/api_core/future/polling.py", line 132, in result
    self._blocking_poll(timeout=timeout, **kwargs)
  File "/usr/lib/python3.9/site-packages/google/api_core/future/polling.py", line 110, in _blocking_poll
    retry_(self._done_or_raise)(**kwargs)
  File "/usr/lib/python3.9/site-packages/google/api_core/retry.py", line 283, in retry_wrapped_func
    return retry_target(
  File "/usr/lib/python3.9/site-packages/google/api_core/retry.py", line 190, in retry_target
    return target()
  File "/usr/lib/python3.9/site-packages/google/api_core/future/polling.py", line 88, in _done_or_raise
    if not self.done(**kwargs):
  File "/usr/lib/python3.9/site-packages/google/api_core/operation.py", line 170, in done
    self._refresh_and_update(retry)
  File "/usr/lib/python3.9/site-packages/google/api_core/operation.py", line 159, in _refresh_and_update
    self._set_result_from_operation()
  File "/usr/lib/python3.9/site-packages/google/api_core/operation.py", line 130, in _set_result_from_operation
    response = protobuf_helpers.from_any_pb(
  File "/usr/lib/python3.9/site-packages/google/api_core/protobuf_helpers.py", line 65, in from_any_pb
    raise TypeError(
TypeError: Could not convert Any to Node


Crash-protocol witness for XCGLogger.DestinationProtocol.process(logDetails: XCGLogger.LogDetails) -> () in conformance XCGLogger.BaseDestination : XCGLogger.DestinationProtocol in XCGLogger

Crash in DestinationProtocol.process. Can anyone check this?
XCGLogger
protocol witness for XCGLogger.DestinationProtocol.process(logDetails: XCGLogger.LogDetails) -> () in conformance XCGLogger.BaseDestination : XCGLogger.DestinationProtocol in XCGLogger :0
XCGLogger
XCGLogger.XCGLogger.logln(: XCGLogger.XCGLogger.Level, functionName: Swift.String, fileName: Swift.String, lineNumber: Swift.Int, userInfo: [Swift.String : Any], closure: () -> Any?) -> () XCGLogger.swift:335
XCGLogger
XCGLogger.XCGLogger.logln(: XCGLogger.XCGLogger.Level, functionName: Swift.StaticString, fileName: Swift.StaticString, lineNumber: Swift.Int, userInfo: [Swift.String : Any], closure: () -> Any?) -> () XCGLogger.swift:313
XCGLogger
merged XCGLogger.XCGLogger.verbose(: @autoclosure () -> Any?, functionName: Swift.StaticString, fileName: Swift.StaticString, lineNumber: Swift.Int, userInfo: [Swift.String : Any]) -> ()
XCGLogger
XCGLogger.XCGLogger.debug(: @autoclosure () -> Any?, functionName: Swift.StaticString, fileName: Swift.StaticString, lineNumber: Swift.Int, userInfo: [Swift.String : Any]) -> ()
Sample
Sample.SyncManager.getLine(line: Swift.String) -> Swift.String SyncManager.swift:759

Might be the same bug we've seen:


Fix problem with culture concerning floating values

Fix problem with culture concerning floating values corresponding to issue #49

Thanks, @YohskDista.


Getting pending requests too

Some of our requests seem to stay in pending for a long time.
Is there a way to get the pending requests also returned in getRequests()?

I see the includePending option.  I will try it out and close the ticket.


Alpha contract address deployments

looks like there's some extra stuff in this pr - did you mean to change membership price code here?

yes, this is all required and part of this PR


Wrongly assigned sulfur charge when reading molecule from mol2 format

I am using rdkit 2017.03.3
When I read the following molecule containing sulfur from mol2 format(via MolFromMol2Block), I get sulfur carrying a positive charge.
However, if I read this molecule from mol2 using oechem (the oechem molecule has sulfur with -ve charge) and write out a .sdf file, then reading in this molecule (in sdf format) using rdkit (via SDMolSupplier), I have a sulfur with negative charge assigned. Any idea why this is the case?
This is the molecule in mol2:
@<TRIPOS>MOLECULE
DrugBank_3448
   33    33     0     0     0
SMALL
NO_CHARGES

@<TRIPOS>ATOM
      1 C5         -4.3466    4.1816    0.2815 C.ar      1 <0>         0.0000
      2 C4         -3.9099    2.9125    0.5675 C.ar      1 <0>         0.0000
      3 C8         -3.3447    3.1722   -4.6728 C.2       1 <0>         0.0000
      4 C7         -2.4608    2.3604   -4.0850 C.2       1 <0>         0.0000
      5 C9         -3.2878    3.3582   -6.1350 C.2       1 <0>         0.0000
      6 C10        -0.0437    1.9460   -2.2852 C.2       1 <0>         0.0000
      7 C1         -3.0484   -0.1138   -0.7992 C.3       1 <0>         0.0000
      8 C3         -1.6160    1.7811    0.2114 C.3       1 <0>         0.0000
      9 C6         -1.3563    1.1749   -2.2680 C.3       1 <0>         0.0000
     10 C2         -1.7158    0.6446   -0.8484 C.3       1 <0>         0.0000
     11 N2         -3.3304    4.9005   -0.2843 N.ar      1 <0>         0.0000
     12 N3         -2.2650    4.1106   -0.3610 N.ar      1 <0>         0.0000
     13 N1         -2.6146    2.9190    0.1501 N.pl3     1 <0>         0.0000
     14 N4         -2.3663    2.0783   -2.7946 N.pl3     1 <0>         0.0000
     15 O1         -4.0728    4.0898   -6.7263 O.2       1 <0>         0.0000
     16 O3          0.0640    3.1447   -2.0577 O.2       1 <0>         0.0000
     17 O4         -0.8807   -0.9925    0.9655 O.2       1 <0>         0.0000
     18 O5         -0.6563   -1.6201   -1.4937 O.2       1 <0>         0.0000
     19 O2          1.0198    1.1601   -2.5945 O.3       1 <0>         0.0000
     20 S1         -0.3916   -0.6212   -0.4046 S.o2      1 <0>         0.0000
     21 H7         -5.3232    4.6089    0.4513 H         1 <0>         0.0000
     22 H6         -4.3858    2.0487    1.0092 H         1 <0>         0.0000
     23 H11        -4.1069    3.7031   -4.1151 H         1 <0>         0.0000
     24 H10        -1.7041    1.8365   -4.6594 H         1 <0>         0.0000
     25 H12        -2.4989    2.8017   -6.6682 H         1 <0>         0.0000
     26 H1         -3.1043   -0.9099   -1.5500 H         1 <0>         0.0000
     27 H2         -3.2154   -0.5998    0.1688 H         1 <0>         0.0000
     28 H3         -3.8899    0.5649   -0.9748 H         1 <0>         0.0000
     29 H4         -1.7176    1.3435    1.2136 H         1 <0>         0.0000
     30 H5         -0.6040    2.2112    0.1980 H         1 <0>         0.0000
     31 H8         -1.2754    0.3457   -2.9820 H         1 <0>         0.0000
     32 H9         -3.0242    2.5111   -2.1539 H         1 <0>         0.0000
     33 H13         0.8146    0.2149   -2.7570 H         1 <0>         0.0000
@<TRIPOS>BOND
     1    1    2 2
     2    1   11 1
     3    2   13 1
     4    3    4 2
     5    3    5 1
     6    4   14 1
     7    5   15 2
     8    6    9 1
     9    6   16 2
    10    6   19 1
    11    7   10 1
    12    8   10 1
    13    8   13 1
    14    9   10 1
    15    9   14 1
    16   10   20 1
    17   11   12 2
    18   12   13 1
    19   17   20 2
    20   18   20 2
    21    1   21 1
    22    2   22 1
    23    3   23 1
    24    4   24 1
    25    5   25 1
    26    7   26 1
    27    7   27 1
    28    7   28 1
    29    8   29 1
    30    8   30 1
    31    9   31 1
    32   14   32 1
    33   19   33 1

This is the sdf molecule:
DrugBank_3448
  -OEChem-09221720343D

 33 33  0     1  0  0  0  0  0999 V2000
   -4.3466    4.1816    0.2815 C   0  0  0  0  0  0  0  0  0  0  0  0
   -3.9099    2.9125    0.5675 C   0  0  0  0  0  0  0  0  0  0  0  0
   -3.3447    3.1722   -4.6728 C   0  0  0  0  0  0  0  0  0  0  0  0
   -2.4608    2.3604   -4.0850 C   0  0  0  0  0  0  0  0  0  0  0  0
   -3.2878    3.3582   -6.1350 C   0  0  0  0  0  0  0  0  0  0  0  0
   -0.0437    1.9460   -2.2852 C   0  0  0  0  0  0  0  0  0  0  0  0
   -3.0484   -0.1138   -0.7992 C   0  0  0  0  0  0  0  0  0  0  0  0
   -1.6160    1.7811    0.2114 C   0  0  0  0  0  0  0  0  0  0  0  0
   -1.3563    1.1749   -2.2680 C   0  0  1  0  0  0  0  0  0  0  0  0
   -1.7158    0.6446   -0.8484 C   0  0  2  0  0  0  0  0  0  0  0  0
   -3.3304    4.9005   -0.2843 N   0  0  0  0  0  0  0  0  0  0  0  0
   -2.2650    4.1106   -0.3610 N   0  0  0  0  0  0  0  0  0  0  0  0
   -2.6146    2.9190    0.1501 N   0  0  0  0  0  0  0  0  0  0  0  0
   -2.3663    2.0783   -2.7946 N   0  0  0  0  0  0  0  0  0  0  0  0
   -4.0728    4.0898   -6.7263 O   0  0  0  0  0  0  0  0  0  0  0  0
    0.0640    3.1447   -2.0577 O   0  0  0  0  0  0  0  0  0  0  0  0
   -0.8807   -0.9925    0.9655 O   0  0  0  0  0  0  0  0  0  0  0  0
   -0.6563   -1.6201   -1.4937 O   0  0  0  0  0  0  0  0  0  0  0  0
    1.0198    1.1601   -2.5945 O   0  0  0  0  0  0  0  0  0  0  0  0
   -0.3916   -0.6212   -0.4046 S   0  5  0  0  0  0  0  0  0  0  0  0
   -5.3232    4.6089    0.4513 H   0  0  0  0  0  0  0  0  0  0  0  0
   -4.3858    2.0487    1.0092 H   0  0  0  0  0  0  0  0  0  0  0  0
   -4.1069    3.7031   -4.1151 H   0  0  0  0  0  0  0  0  0  0  0  0
   -1.7041    1.8365   -4.6594 H   0  0  0  0  0  0  0  0  0  0  0  0
   -2.4989    2.8017   -6.6682 H   0  0  0  0  0  0  0  0  0  0  0  0
   -3.1043   -0.9099   -1.5500 H   0  0  0  0  0  0  0  0  0  0  0  0
   -3.2154   -0.5998    0.1688 H   0  0  0  0  0  0  0  0  0  0  0  0
   -3.8899    0.5649   -0.9748 H   0  0  0  0  0  0  0  0  0  0  0  0
   -1.7176    1.3435    1.2136 H   0  0  0  0  0  0  0  0  0  0  0  0
   -0.6040    2.2112    0.1980 H   0  0  0  0  0  0  0  0  0  0  0  0
   -1.2754    0.3457   -2.9820 H   0  0  0  0  0  0  0  0  0  0  0  0
   -3.0242    2.5111   -2.1539 H   0  0  0  0  0  0  0  0  0  0  0  0
    0.8146    0.2149   -2.7570 H   0  0  0  0  0  0  0  0  0  0  0  0
  1  2  2  0  0  0  0
  1 11  1  0  0  0  0
  2 13  1  0  0  0  0
  3  4  2  0  0  0  0
  3  5  1  0  0  0  0
  4 14  1  0  0  0  0
  5 15  2  0  0  0  0
  6  9  1  0  0  0  0
  6 16  2  0  0  0  0
  6 19  1  0  0  0  0
  7 10  1  0  0  0  0
  8 10  1  0  0  0  0
  8 13  1  0  0  0  0
  9 10  1  0  0  0  0
  9 14  1  0  0  0  0
 10 20  1  0  0  0  0
 11 12  2  0  0  0  0
 12 13  1  0  0  0  0
 17 20  2  0  0  0  0
 18 20  2  0  0  0  0
  1 21  1  0  0  0  0
  2 22  1  0  0  0  0
  3 23  1  0  0  0  0
  4 24  1  0  0  0  0
  5 25  1  0  0  0  0
  7 26  1  0  0  0  0
  7 27  1  0  0  0  0
  7 28  1  0  0  0  0
  8 29  1  0  0  0  0
  8 30  1  0  0  0  0
  9 31  1  0  0  0  0
 14 32  1  0  0  0  0
 19 33  1  0  0  0  0
M  CHG  1  20  -1
M  END
$$$$

The RDKit really only correctly supports mol2 files that use corina atom types. What's the source of this file?

Hi Greg, the mol2 file I am using adopts sybyl atom types.
For the purpose of the open force field project, I need to read in partial charge information in order to fully parameterise molecules. This makes mol2 type files a suitable choice. Is there any way to generate a mol2 file with Corina atom types with an open tool? Or would you suggest some other ways to enable me to process molecules with both coordinates and charges info?

The problem with "Sybyl" atom types (and the reason the RDKit doesn't support them) is that their meaning differs between tools from the company formerly known as Tripos. The semantics of those atom types is poorly defined.
If you can generate an SDF from whichever tool you are using to generate the partial charges, I would suggest adding a new SDF field called something like PARTIAL_CHARGES and putting comma-separated charges in there.


Canvas image API should support animated images

From @collinjackson on June 2, 2015 19:19
This will require us to change the Sky engine API to support indexing into individual image frames
Copied from original issue: domokit/mojo#189

Maybe the canvas image API doesn't have to, if the Image API does it itself and just fires a callback every time the image changes (so we can schedule a new paint).


[benchmark] Benchmark using different madvise() settings for MMAP

Goal: benchmark impact different flags of mmap and compare them
not only rand, sequential, many more

No further contribution.


feat - Added show/hide functionality for integration tokens.

User description

Closes #
📑 Description

✅ Checks

[ ] My pull request adheres to the code style of this project
[ ] My code requires changes to the documentation
[ ] I have updated the documentation as required
[ ] All the tests have passed

☢ Does this introduce a breaking change?

[ ] Yes
[ ] No

ℹ Additional Information

Description

Introduced show/hide functionality for integration tokens to enhance user experience.
Updated input fields to use type="password" for security, with toggle visibility icons.
Improved form validation and submission process using jQuery.
Adjusted CSS for better alignment of input group elements.

Changes walkthrough 📝
Relevant filesEnhancement

integrations.phpImplement show/hide functionality for integration tokens

src/integrations.php
Added show/hide functionality for integration tokens.  Updated input fields to use type 'password' for sensitive data.  Included Font Awesome for visibility toggle icons.  Enhanced form validation and submission handling.

+144/-94

Formatting

user.cssStyle adjustments for input groups

src/user.css

Added styles for input group text alignment.

+5/-0

Configuration changes

settings.jsonUpdate spell check settings

.vscode/settings.json

Updated spell check words for new token names.

+5/-1

💡 Penify usage:
Comment /help on the PR to get a list of all available Penify tools and their descriptions

Summary by CodeRabbit

New Features

Enhanced security for sensitive input fields by changing API key inputs to password types with visibility toggle icons.
Added new entries to the spell-checking configuration for improved text accuracy.

Style

Introduced a new CSS class .input-group-text for improved layout and design consistency.

Bug Fixes

Improved form validation before submission to ensure proper user input handling.

@straccini codacy bypass


tag-self-close in documentation is actually tags-self-close

If you have "tag-self-close": "never" in your configuration and <img> in your HTML, then it returns an error: void element should close itself.
However, if you have "tags-self-close": "never" (plural tags instead of singular tag) in your configuration, then it works as expected and <img> won't return an error.
The documentation says that the key is supposed to be tag-self-close even though that doesn't work. Either the documentation should be updated, or tags-self-close should be changed to tag-self-close in the code.

The option "always" for tag-self-close specifies that void elements like <img> must close themselves with a trailing /, like <img... />, while "never" means that they must close themselves with a </img> closing tag. If you want to turn this rule off entirely, use the value false, which is the default.
tags-self-close simply doesn't do anything, so that the default false value for tag-self-close is used. I assume the error with 'tag-self-close':'never' comes from an html4-style <img.../> tag; if you want to allow these tags and disallow the alternative, use the value "always".
It's very difficult to explain this option both concisely and clearly. I may try to rewrite the wiki entry again, but no guarantees that it will come out any better.


Add p2p ip address whitelisting

Change summary:

Add p2p ip address whitelisting for on-premise nodes

Related issues:

https://github.com/ainblockchain/ain-blockchain/issues/1285

Thanks for the review! @minho-comcom-ai Now merging..


Change readme

@Jackie-Brown -
Did you mean to include changes to the STIS_Coronagraphic_Observation_Feasibility.ipynb notebook in this PR?

The HTML pages should be included in the table.  E.g.:
https://spacetelescope.github.io/hst_notebooks/notebooks/STIS/cross-correlation/cross-correlation.html

@Jackie-Brown, I removed the files that were unintentionally included in your PR.

fits --> FITS
raw --> RAW
flt --> FLT
x1d --> X1D
"the Acquisition notebook" --> "the target_acquisition notebook"

@haticekaratay @mgough-970 -
Do we have to do something to rebuild the HTML README page using this update?
https://spacetelescope.github.io/hst_notebooks/notebooks/STIS/README.html


README.rst is not correctly rendered

I am not sure whether this is a warehouse bug or a readme_renderer one.
My README.rst file is not rendered on pypi.org. However, although  readme_renderer is described as "[rendering] long_description/README files in Warehouse", it does not find any error in my file:
$ python setup.py check --restructuredtext
running check

Now, I have no idea how to debug this…
Additional notes:

I am using python 3.5.4 (on Debian testing) with an up-to-date version of readme_renderer (version 17.2);
restructured-lint does not see any error either.

-- Louis

Hi @paternal,
When I run a check on your repo, here's the output I get:
$ python setup.py check --restructuredtext
running check
warning: Check: :2: (WARNING/2) Title underline too short.

warning: Check:

warning: Check: papersize 📏 Paper size related tools

warning: Check: ====================================

warning: Check: Invalid markup which will not be rendered on PyPI.

Thanks! I eventually found the problem: it works (readme_renderer finds an error in my README.rst) with python2.7 and python3.6, but it does not with python3.5. I reported an issue to readme_renderer: pypa/readme_renderer#57
Sorry for the noise,
-- Louis

No worries, thanks for filing that issue!


Compile error on MSVC2019 with any macro which involves stringification of std::string (asserts, INFO, etc.) when  isn't included

Description
If I compile doctest v2.3.8 on MSVC 2019,
I don't know if this issue was caused by updating doctest or MSVC. I didn't experience the issue on doctest v2.3.4 with an older MSVC, but newer MSVC and latest C++ mode breaks older doctest. And when trying to downgrade doctest and build using C++17 mode, Qt Creator crashed and I'm trying to debug that.
This issue seems somewhat related to https://github.com/onqtam/doctest/issues/356, but I'm getting a compilation error rather than linking error.
Steps to reproduce
#include <doctest.h>
#include <string>

TEST_CASE("test") {
    std::string foo;
    INFO(foo);
}

Resulting errors:
https://gist.github.com/nyanpasu64/4c93792cdbf02b5c02700fddf7c64df5
Extra information

doctest version: 2.3.8
Operating System: Windows 10 x64
Compiler+version: MSVC 2019 x64 14.26.28801 or 16.6.30114.105 or 19.26.28805 (not sure)

I think this would again be fixed by just including either <ostream> or <iostream> - MSVC 2019 changed their internal header structure and I remember filing a bug about that (but it won't be fixed because the change has been intentional). Unfortunately I can't find the bug or the other doctest issue which linked to that bug...

https://github.com/onqtam/doctest/commit/1ea91445686768a1216e03815433962cd4b8681d was found as soon as 2.3.3. But I upgraded doctest from 2.3.4 (which already had the fix) to 2.3.8, and MSVC at the same time, causing my compile issue.
I ended up including , but why does that fix not work?

oh - because for that you'd need to define DOCTEST_CONFIG_USE_STD_HEADERS before the doctest header

// `#define INFO` from DOCTEST
// conflicts with macros from other libs
#define DOCTEST_CONFIG_NO_SHORT_MACRO_NAMES

@blockspacer is this related to the issue?

@blockspacer is this related to the issue?

I got similar error, also with INFO macro. fixed by DOCTEST_CONFIG_NO_SHORT_MACRO_NAMES

@blockspacer I think these are 2 separate issues - you just have conflicting macros with other libraries and indeed the right approach for you is to use DOCTEST_CONFIG_NO_SHORT_MACRO_NAMES. This issue however is completely different - it's related to the way doctest stringifies expressions and detects operator<<(std::ostream&...) for types.

This has been fixed in dev. :)


gen_context script usage

To create a context from an RDFS/OWL definition using the script/gen_context script several parameters are mandatory required, but no example is documented.
Attempted to do with the ontology https://d-nb.info/standards/elementset/gnd.rdf, but got the following error:
$ ruby gen_context.rb --body --initial gnd_20221010.rdf --language en --output gnd_context --prefix "https://d-nb.info/standards/elementset" --vocab "https://d-nb.info/standards/vocab/gnd"

        from /home/rails5/.rbenv/versions/3.2.2/lib/ruby/3.2.0/json/common.rb:216:in `parse'
        from gen_context.rb:69:in `block in <main>'
        from /home/rails5/.rbenv/versions/3.2.2/lib/ruby/3.2.0/getoptlong.rb:863:in `block in each'
        from /home/rails5/.rbenv/versions/3.2.2/lib/ruby/3.2.0/getoptlong.rb:860:in `loop'
        from /home/rails5/.rbenv/versions/3.2.2/lib/ruby/3.2.0/getoptlong.rb:860:in `each'
        from gen_context.rb:66:in `<main>'

What am I doing wrong here?

This is mostly an internal tool, but there is some documentation:
ruby gen_context.rb --help

Usage: script/gen_context [options] URL ...
  --body          Include the vocabulary definition in the body of the document
  --hierarchical  Create a hierachy using 'children' as reverse of rdfs:subClassOf
  --initial       Initial context to load, overrides default
  --language      Default language for vocabulary
  --output, -o    Output to the specified file path
  --prefix        space-separated prefix uri combination
  --quiet         Supress most output other than progress indicators
  --verbose       Detail on execution
  --vocab         Set context @vocab to specified URI
  --help, -?      This message

The --initial option expects a JSON-LD context, while you're providing the oncology itself.
There was a bug in some of the necessary code, which I've updated on the develop branch and will be out in the next release. If you clone the gem on your machine, do a bundle install, then run bundle exec ruby script/gen_context --language en --prefix "https://d-nb.info/standards/elementset" --vocab "https://d-nb.info/standards/vocab/gnd" https://d-nb.info/standards/elementset/gnd.rdf. You can add a --output option, but the next argument is treated as the output file.

./json-ld/script/gen_context --language en --prefix "https://d-nb.info/standards/elementset" --vocab "https://d-nb.info/standards/vocab/gnd" https://d-nb.info/standards/elementset/gnd.rdf --output context.json runs without error now, thank you!

Personally I am using this script and it would be useful to be available after gem installation.


fit-trackee: 0.7.31 -> 0.8.5

This upgrade will require database migration updates
Description of changes
Diff: https://github.com/SamR1/FitTrackee/compare/v0.7.31...v0.8.5
Changelog: https://github.com/SamR1/FitTrackee/blob/v0.8.5/CHANGELOG.md
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Result of nixpkgs-review pr 328942 run on x86_64-linux 1

2 packages built:

fit-trackee
fit-trackee.dist

Result of nixpkgs-review pr 328942 run on aarch64-darwin 1

2 packages failed to build:

fit-trackee
fit-trackee.dist

Result of nixpkgs-review pr 328942 run on x86_64-darwin 1

2 packages failed to build:

fit-trackee
fit-trackee.dist

Result of nixpkgs-review pr 328942 run on aarch64-darwin 1

2 packages failed to build:

fit-trackee
fit-trackee.dist

Result of nixpkgs-review pr 328942 run on x86_64-darwin 1

2 packages failed to build:

fit-trackee
fit-trackee.dist

Result of nixpkgs-review pr 328942 run on aarch64-linux 1

2 packages built:

fit-trackee
fit-trackee.dist

Result of nixpkgs-review pr 328942 run on x86_64-linux 1

2 packages failed to build:

fit-trackee
fit-trackee.dist

Result of nixpkgs-review pr 328942 run on x86_64-linux 1

2 packages built:

fit-trackee
fit-trackee.dist

Result of nixpkgs-review pr 328942 run on x86_64-linux 1

2 packages failed to build:

fit-trackee
fit-trackee.dist

Result of nixpkgs-review pr 328942 run on aarch64-linux 1

2 packages built:

fit-trackee
fit-trackee.dist


Add libqt-webkit to jenkins boxes

This adds libqt-webkit package to the jenkins boxes. This will enable jenkins to run smokey tests with javascript enabled.
The same is done in ci-puppet.
https://trello.com/c/WIunrfiT/322-investigate-turning-on-javascript-in-smokey

Do you need this to run on the CI agents, too? This change won't install the package on here, only on the deploy Jenkins and CI master machines.


Replace data-compose with maven dependency

PR: https://github.com/prperiscal/mytasks-user-service/pull/4


Failed to judge if the List is not empty in @Query

i'm using spring-data-r2dbc:3.0.4 and spirng-r2dbc:6.0.7;
here is my code
@Query("""
            SELECT SUM(age) FROM user
            where (:ids IS NULL OR id in (:ids))
                """)
    Mono<Long> sumAges(List<Long> ids);

and a io.r2dbc.spi.R2dbcBadGrammarException occur;
what can i do about this

Please provide a Minimimal Reproducable Example, preferable as a Github repository. Make sure to include the database, either as an in memory database or if that is not possible using Testcontainers.


Unable to deploy otomi without manual intervention

Describe the bug:
Unable to deploy otomi on brand new cluster.
To Reproduce
Steps to reproduce the behavior:

create a new cluster
call otomi deploy from you shell
Observe that deployments stucks
Observer that istiod k8s deployment is not able to schedule Pod k -n istio-system describe deploy istiod

Expected behavior:
It is enough to call once otomi deploy. No manual intervention needed.
Additional context:
The workaround is to remove the istio-injection label from istio-system, so the istiod can deploy without istio sidecar.
k label ns istio-system istio-injection-

I really do not understand why this label is there in the first place. What was the intention?

fixed

thanks


Android Basics: tip-calculator

URL of codelab
https://developer.android.com/codelabs/basic-android-kotlin-training-tip-calculator?continue=https%3A%2F%2Fdeveloper.android.com%2Fcourses%2Fpathways%2Fandroid-basics-kotlin-unit-2-pathway-1%23codelab-https%3A%2F%2Fdeveloper.android.com%2Fcodelabs%2Fbasic-android-kotlin-training-tip-calculator#4
In which task and step of the codelab can this issue be found?
Section 5
Describe the problem
At point 5.3 this line is quoted (among others):
        at com.example.tiptime.MainActivity.calculateTip(MainActivity.kt:22)
however a bit later (5.6) the line number from 22 becomes 20:
at com.example.tiptime.MainActivity.calculateTip(MainActivity.kt:20)

@dega03
Thank you so much for reporting this issue. This is fixed and will be published soon.


Increase the logging integration test logger application memory request

How to categorize this PR?

/area logging
/kind enhancement
What this PR does / why we need it:
Most of the logging integration tests are failing on nodes using flatcar OS. This is because the burstable memory is not enough to run a sandbox container for the logger applications. By increasing the memory limit to 30MiB the test does not struggle to create its workload.
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
Release note:

Make logging integration tests on flatcar OS  more stable by increasing the memory limit and request of the logger application

Can you share what was the exact container create error?

Can you share what was the exact container create error?

I see that with the old memory request and limit there are failures such as:
  Warning  Failed     12s (x2 over 23s)  kubelet            Error: failed to create containerd task: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: read init-p: connection reset by peer: unknown

With the new memory request I am not able to reproduce the error.


[bug report] the El tree component property 'default expanded keys' is not a default property, but a real-time value that will update the component view. It is recommended to change its name to' expanded keys'.

Element UI version
2.13.2
OS/Browsers version
windows/macOS
Vue version
2.6.11
Reproduction Link
https://codepen.io/chaoszsc/pen/RwrVmVo?editable=true%3Dhttps%3A%2F%2Felement.eleme.cn%2F
Steps to reproduce
1.给可选择el-tree组件‘default-expanded-keys’设置响应参数
2.选中节点时更新‘default-expanded-keys’的参数
3.该选中节点被展开
What is Expected?
‘default-expanded-keys’不应加default（默认）前缀，基于其实时更新节点属性，引发视图变更的功能，建议改为‘expanded-keys’
What is actually happening?
默认属性值变更时，实时更新了视图

I cannot observe the behavior you're writing about.
Or do the keys need to be numbers?

just look at this: https://codepen.io/chaoszsc/pen/GRozraj?editable=true%3Dhttps%3A%2F%2Felement.eleme.cn%2F

Ah, this may again be a problem with things working differently with asynchronous data load.

So I think it should not be prefixed with 'default'.


[bug]: Can't connect to wallet like SafePal, Trust Wallet...

Description
I have integrated the @web3modal/wagmi-react-native via the docs from WalletConnect into my app:
https://docs.walletconnect.com/web3modal/react-native/about
I use the W3mButton to connect with wallets, I can connect with only MetaMask wallet. When I try to connect with other wallets like SafePal and Trust Wallet, those wallets will stuck on the connecting step, and no connect confirmation alert appears. I'm not sure if it's an SDK bug or a wallet bug. Please help. Thanks for your time!
Web3Modal SDK version
"@web3modal/wagmi-react-native": "^1.2.1"
Output of npx react-native info
System:
  OS: macOS 14.4
  CPU: (16) x64 Intel(R) Core(TM) i7-10700K CPU @ 3.80GHz
  Memory: 32.40 MB / 32.00 GB
  Shell:
    version: "5.9"
    path: /bin/zsh
Binaries:
  Node:
    version: 20.5.1
    path: /usr/local/bin/node
  Yarn:
    version: 1.19.1
    path: ~/travala-mobile-app/node_modules/.bin/yarn
  npm:
    version: 9.8.0
    path: /usr/local/bin/npm
  Watchman:
    version: 2<IP_ADDRESS>
    path: /usr/local/bin/watchman
Managers:
  CocoaPods:
    version: 1.15.2
    path: /Users/alex/.rbenv/shims/pod
SDKs:
  iOS SDK:
    Platforms:
      - DriverKit 22.4
      - iOS 16.4
      - macOS 13.3
      - tvOS 16.4
      - watchOS 9.4
  Android SDK:
    API Levels:
      - "28"
      - "29"
      - "30"
      - "31"
      - "32"
      - "33"
      - "34"
    Build Tools:
      - 28.0.3
      - 29.0.2
      - 29.0.3
      - 30.0.2
      - 30.0.3
      - 31.0.0
      - 32.0.0
      - 33.0.0
      - 33.0.1
      - 34.0.0
    System Images:
      - android-29 | Google Play Intel x86 Atom
      - android-30 | Google APIs Intel x86 Atom
      - android-33 | Google APIs Intel x86 Atom_64
    Android NDK: Not Found
IDEs:
  Android Studio: 2022.1 AI-221.6008.13.2211.9477386
  Xcode:
    version: 14.3.1/14E300c
    path: /usr/bin/xcodebuild
Languages:
  Java:
    version: 17.0.10
    path: /usr/bin/javac
  Ruby:
    version: 3.1.2
    path: /Users/alex/.rbenv/shims/ruby
npmPackages:
  "@react-native-community/cli": Not Found
  react:
    installed: 18.2.0
    wanted: 18.2.0
  react-native:
    installed: 0.72.3
    wanted: 0.72.3
  react-native-macos: Not Found
npmGlobalPackages:
  "*react-native*": Not Found
Android:
  hermesEnabled: true
  newArchEnabled: false
iOS:
  hermesEnabled: true
  newArchEnabled: false

Expo Version (if applies)
No response
Steps to reproduce

Integrate the wallet connect flow via: https://docs.walletconnect.com/web3modal/react-native/about.
Use W3mButton component to render the connect button.
Use the button to connect to Trust Wallet or SafePal.
You will be redirected to those wallets.
You can't see the connect confirmation alert.

Snack, code example, screenshot, or link to a repository

hey @mtr1012 👋 Some questions:

can you check what happens if you close the dapp and open it again?
could you upload a video showing the complete flow?
Can you provide your implementation code? Which chain are you trying to connect?

hey @mtr1012 👋 Some questions:

can you check what happens if you close the dapp and open it again?
could you upload a video showing the complete flow?
Can you provide your implementation code? Which chain are you trying to connect?

I've tried but no luck
You can see the video here: https://youtu.be/vCOvlssJKnE
Here is my code:

//My Config
const chains = [mainnet, bsc]
const wagmiConfig = defaultWagmiConfig({ chains, projectId, metadata })
createWeb3Modal({
projectId,
chains,
wagmiConfig
})

// My app

   <Provider store={store}>
      <WagmiConfig config={wagmiConfig}>
            <RouterWithRedux scenes={AppRoutes} backAndroidHandler={this.onBackPress} />
            <Web3Modal />
      </WagmiConfig>
   </Provider>

// I use <W3mButton /> from '@web3modal/wagmi-react-native' to connect

@mtr1012 the issue is weird 🤔 and your configuration seems to be OK.
Some other questions:

Are you using the latest version of Trust? (v11.2 now)
Are you using a VPN? If so, can you check turning it off?
Does the error happen on both iOS and Android?
Can you check if this sample project works for you? https://github.com/ignaciosantise/walletconnect-test

@mtr1012 the issue is weird 🤔 and your configuration seems to be OK. Some other questions:

Are you using the latest version of Trust? (v11.2 now)
Are you using a VPN? If so, can you check turning it off?
Does the error happen on both iOS and Android?
Can you check if this sample project works for you? https://github.com/ignaciosantise/walletconnect-test

The missing icons parameter in metadata is the reason. Thanks for your support. I think you should update the docs and mark the icons parameter is required 😂

@mtr1012 thanks for reporting 🤝
do you use Typescript? It should throw an error if the metadata doesn't contain icons

@mtr1012 thanks for reporting 🤝
do you use Typescript? It should throw an error if the metadata doesn't contain icons

Unfortunately, I'm using js instead


Merge pull request #1 from tensorflow/master

Update

Can one of the admins verify this patch?

No, did it backwards, sorry.


correction in procedure

Added a correction while typing your tenant id during setup.
Problem: https://github.com/MicrosoftDocs/microsoft-365-docs/issues/1753

@JohanFreelancer9 - kindly do the copy editor review for this PR. Thanks.

@mypil Ready to go live.

@Sirkkuw - PR has been copy edited and is ready for final review. This is based on issue #1753.
Thank you.
cc: @kenwith


getTagMedia() no place for new api parameters

I have noticed that there is no way to enter the new api parameters
MIN_TAG_ID	Return media before this min_tag_id.
MAX_TAG_ID	Return media after this max_tag_id.
I really need to use this to keep track of media I have already liked in my app (without hitting the api), please consider adding it

hi I hacked it in, I haven't tested but from what I undertand of the code this should add the feature I asked for
iggetmedia.patch.txt


Activity automatically destroy when trying to merge audio and video

public void mergeAudioVideo(String audioFile, String videoFile, String mergedFile) {         Controller.getInstance().run(new String[]{                 "-i",                 videoFile,                 "-i",                 audioFile,                 "-c",                 "copy",                 "-map",                 "0:0",                 "-map",                 "1:0",                 mergedFile         });     }
here is the code i m using for audio video merger

Are you sure this is the right command?
https://superuser.com/questions/590201/add-audio-to-video-using-ffmpeg#590210

i have the smae problem bur with audio bitrate my  Controller.getInstance().run(new String[]{                         "-i",                         songPath,                         "-b:a 8k",                         getOutputPath(songPath)});
i have checked what you have said @madhavanmalolan
but it still occurs and there is no error just the activity destroy
the only suspect lines in the log 07-06 02:33:19.136 19897-19897/com.mohamedmorsi.audiocompressor I/Videokit: start run in main. 07-06 02:33:19.136 19897-19897/com.mohamedmorsi.audiocompressor I/Videokit: register_exit 07-06 02:33:19.137 19897-19897/com.mohamedmorsi.audiocompressor I/Videokit: term_init & parse options.  if any one could help


Markdown lexer improvements

This is a series of improvements to the Markdown Lexer:

Fenced code was not recognized when it contained empty lines
Inline code was not recognized after bulleted list entry (e.g. * `code` )
Bold and italics were not recognized when surrounded with non-space char (e.g. (**foo**))
There is no way to recognize code indented by 4 spaces using regex only - so remove the error-prone detection

I verified that the perf issue described in #1617 is fixed by this change with identical output from pygmentize on the test file. If this PR gets merged then my proposed fix in #1618 is unnecessary.


Ensure unit, integration, and e2e testing best practices

Overview
There are a lot of non-recommended practices in our unit & integration tests, such as (not limited to):

Non-accessible queries
Irrelevant assertions
Too large snapshot testings which make them hard to read and rendered them useless
etc.

All those practices potentially lead to regressions in the future by introducing false positives due to passing tests on breaking changes.
Testing Guidelines
In general, these are three guidelines that we need to pay attention to:

Query priorities
Testing appearance and disappearance
Opting in for interactions instead of events

Current Tasks
Unit Testing / Integration Testing

[ ] #598
[ ] #697
[ ] #717
[ ] #459
[ ] #468
[ ] #425

E2E Testing

[ ] #183
[ ] #186
[x] #195
[ ] #380
[ ] #381
[ ] #414
[ ] #731

@zainfathoni I updated the title and put all the lists about testing and e2e testing. Is that okay?
I will be focusing on that for this project. But, I little bit busy for 2 weeks until August 22. Can only visit the project on Sunday.
Maybe after that, I can have more time to help

Okay, no problem.
In the meantime, I'm working on #195 at #727 so that we can use it in our GitHub Actions CI.

I think we will need to wait for #697 to be done before we can continue.
For #459 and #468 is that e2e testing? or still on unit/integration testing
@zainfathoni

I think we will need to wait for #697 to be done before we can continue.

For the unit & integration tests, yes. I'll try to review it later tonight to unblock the rest of the issues.
However, I think nothing blocks the e2e issues at the moment.

For #459 and #468 is that e2e testing? or still on unit/integration testing

Based on my classifications at #374, they are parts of integration testing.

I think yes mas @zainfathoni, RTL, and Cypress are in different categories.
Even though they are closely related maybe testing for unit/integration test and e2e test for cypress related?.
I will update the description based on #374

I think yes mas @zainfathoni, RTL, and Cypress are in different categories.
Even though they are closely related maybe testing for unit/integration test and e2e test for cypress related?.

Let's rename the label for unit & integration into unit & integration test to make them more specific and obvious for the contributors.

I will update the description based on #374

Thanks! 🙏

Mas @rubiagatra, this epic becomes more important due to the upcoming #783.

Thank you for remind me mas @zainfathoni , will checkout this soon


Uncaught exception on comment-only doc

Hello,
I'm trying to migrate Arachni from Nokogiri to LibXML due to the considerable performance benefits, but I've run into an issue.
When searching a document that only includes a comment an exception is raised.
Reproduce with:
HTML = <<HTML
    <!-- stuff -->
HTML

doc = LibXML::XML::HTMLParser.string(
    HTML,
    options: LibXML::XML::HTMLParser::Options::NOIMPLIED |
                 LibXML::XML::HTMLParser::Options::NOERROR |
                 LibXML::XML::HTMLParser::Options::NOWARNING |
                 LibXML::XML::HTMLParser::Options::RECOVER |
                 LibXML::XML::HTMLParser::Options::NONET
)

doc.parse.find( '//*' ).size

Result:
ems/ruby-2.2.3/gems/libxml-ruby-2.8.0/lib/libxml/document.rb:87:in `node=': wrong argument type nil (expected Data) (TypeError)
        from /home/zapotek/.rvm/gems/ruby-2.2.3/gems/libxml-ruby-2.8.0/lib/libxml/document.rb:87:in `context'
        from /home/zapotek/.rvm/gems/ruby-2.2.3/gems/libxml-ruby-2.8.0/lib/libxml/document.rb:123:in `find'

If however a tag is introduced, like so:
HTML = <<HTML
    <!-- stuff -->
    <test/>
HTML

The issue goes away.
On a not so related note, I've been unable to completely silence the printed errors/warnings.
The above example (with the last HTML) will output:
Error: Tag test invalid at :2.

Can you add test case?

Is the test case I provided not enough?
Do you require more info?

Sorry, I meant a pull request adding it to the test suite that shows the failure.

To be honest I'm not familiar with the test framework you're using nor your structure.
I'd probably be better if you took a stab at it.

Ok, added a test case.  I can't duplicate on latest version of gem.  So hopefully it has been fixed along the way.


Unimplemented Operation: DaysToDate

Description
Ah. The facerakes, they are many.
https://archive.org/download/3-libjune-05/3LIBJUNE05.iso/library%2Fepocmisc%2Fprocyonr%2FProcre106Pro.sis
(http://software.psion.info/programs/0x1000070f/)
Still, it wasn't a database crash!
Details
Unimplemented operation DaysToDate
PROCYONR\LST:00003E8E: FF [NextOpcodeTable] 27 DaysToDate
PROCYONR\STARS:00000F81: 53 [RunProcedure] 0x0AEC (name="LST" nargs=0)
stack traceback:
	[C]: in function 'error'
	init.lua:478: in function 'unimplemented'
	ops.lua:2517: in function 'ops.DaysToDate'
	ops.lua:2302: in function 'ops.NextOpcodeTable'
	runtime.lua:864: in function <runtime.lua:849>
	[C]: in function 'xpcall'
	runtime.lua:983: in method 'pcallProc'
	runtime.lua:1253: in function 'runtime.runOpo'

Already fixed on the tomsci-more-things branch

fixed in https://github.com/inseven/opolua/pull/350 (https://github.com/inseven/opolua/commit/f8877319bdcb2fc818d908569fdeb698be2b01ec)


Tidy fails with: 'Error while processing X.Cpp'

Compiling X.Cpp succeeds.
Tidying fails with 'Error while processing X.Cpp'.
Workaround:

Rename 'X.Cpp' to 'X.cpp'; or

Fix:

Pass all options to clang-tidy that are passed to clang++ (here: '-x c++')

This issue was fixed.
It will be available in the next version to be published on the VS Marketplace.
Thanks for your feedback.

Thank you for sharing your tools and fixing the issue!


Using the vtk port

The vtk includes are no longer directly in installed//include but in a subdirectory thereof (vtk-9.0). This means that
#include <vtkSomething.h>
no longer finds the include when using VS integration. I would need to write
#include <vtk-9.0/vtkSomething.h>
but that would make the source code dependent on the vtk version, which is a bad thing.
The only solution I found is to add g:\dev\vcpkg\installed\x64-windows\include\vtk-9.0 to AdditionalIncludeDirectories for the projects using vtk, which defeats the purpose of vtk integration.
Is there a better solution?

I think @Neumann-A has answered your question. Please use CMake project @trebahl .
Closing this issue for now since there is no active thread.


task: set security headers

Description
Configure and explicitly set security headers on our site and dapp.
Share security header config via @klimadao/lib/config
Had issues with requiring modules so converted the next config filetype from .js to .mjs for import statement support, that seemed to work. Spent way too long getting it to work, was stubborn to share config across our apps 😓
Following security headers have been added:
[
  {
    key: "Referrer-Policy",
    value: "strict-origin-when-cross-origin",
  },
  {
    key: "Strict-Transport-Security",
    value: "max-age=31536000; includeSubDomains",
  },
  {
    key: "X-Content-Type-Options",
    value: "nosniff",
  },
  {
    key: "X-XSS-Protection",
    value: "0",
  },
  {
    key: "Referrer-Policy",
    value: "strict-origin-when-cross-origin",
  },
  {
    key: "Content-Security-Policy",
    value: "upgrade-insecure-requests",
  },
]

dapp
> curl --head http://localhost:3001

HTTP/1.1 200 OK
Referrer-Policy: strict-origin-when-cross-origin
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
X-XSS-Protection: 0
Content-Security-Policy: upgrade-insecure-requests
Cache-Control: no-store, must-revalidate
X-Powered-By: Next.js
ETag: "adf4-Wb38TORXxNRD8/eBTLnefZMUSPU"
Content-Type: text/html; charset=utf-8
Content-Length: 44532
Vary: Accept-Encoding
Date: Tue, 08 Mar 2022 15:01:08 GMT
Connection: keep-alive
Keep-Alive: timeout=5

site
> curl --head http://localhost:3000

HTTP/1.1 200 OK
Referrer-Policy: strict-origin-when-cross-origin
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
X-XSS-Protection: 0
Content-Security-Policy: upgrade-insecure-requests
Cache-Control: no-store, must-revalidate
X-Powered-By: Next.js
ETag: "17004-Y/QuZtb7zSTC5vlbVR99LEM0fgI"
Content-Type: text/html; charset=utf-8
Content-Length: 94212
Vary: Accept-Encoding
Date: Tue, 08 Mar 2022 15:02:50 GMT
Connection: keep-alive
Keep-Alive: timeout=5

Related Ticket

Resolves #107
Checklist

[x] Building the site with npm run build-site works without errors
[x] Building the app with npm run build-app works without errors
[x] I formatted JS and TS files with running npm run format-all

'Missing Header
key: 'X-Frame-Options',
value: 'SAMEORIGIN'
Note:
WWW already has a next.config.js in root, has these headers set on source: "/api/block-rate" already,: https://github.com/KlimaDAO/klimadao/blob/staging/site/next.config.js#L21

'Missing Header key: 'X-Frame-Options', value: 'SAMEORIGIN'
Note: WWW already has a next.config.js in root, has these headers set on source: "/api/block-rate" already,: https://github.com/KlimaDAO/klimadao/blob/staging/site/next.config.js#L21

Thanks, I misread what X-Frame-Options: SAMEORIGIN implies and added that back in.
Not sure what you mean in your note, could you clarify that please :)
Adding new headers to the headers declaration array won't override whats already defined there.
> curl --head http://localhost:3000/api/block-rate

Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET
Access-Control-Allow-Headers: X-CSRF-Token, X-Requested-With, Accept, Accept-Version, Content-Length, Content-MD5, Content-Type, Date, X-Api-Version
Referrer-Policy: strict-origin-when-cross-origin
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
X-XSS-Protection: 0
Content-Security-Policy: upgrade-insecure-requests
Cache-Control: no-store, must-revalidate
X-Powered-By: Next.js
ETag: "1e6f-oeOqo3KC4s1FbR33NQ1dQ0FwuYg"
Content-Type: text/html; charset=utf-8
Content-Length: 7791
Vary: Accept-Encoding
Date: Tue, 08 Mar 2022 23:30:01 GMT
Connection: keep-alive
Keep-Alive: timeout=5

Yesss thank you for taking this.
controversial opinion: I'd just copypasta the headers into both configs and ditch the shared config and .mjs Makes it easier if we want to configure the sites independently (like if we at some point want to allow embedding for dapp.klimadao.finance/#/offset)
I'm getting flashbacks to how crazy the nested/inherited configs got in my previous company's monorepo haha

Not controversial at all :)
There's valid points for and against on either approach, but I agree, probably a bit too early to be making the call for shared config. It's easy enough to change over at this stage! Definitely don't want no PTSD like situation either 😂


Remove the option for the command service rm

Remove the option for the command service rm which only has a --help option.
Signed-off-by: yuexiao-wang<EMAIL_ADDRESS>

LGTM
ping @dnephin

LGTM


Define id for content types

Right now, a machine name is created off of the name of a content type. An ID should be included in the actual config instead, so human-readable stuff can be changed and we can have a contract around that ID, just like attributes.

Closed by https://github.com/punchcard-cms/content-types/pull/6


Font family changed causes display bugs in legend

I have set the global defaultFontFamily to Geomanist for instance, and when trying to use a chart this happens.
    Chart.defaults.global.defaultFontFamily = 'Geomanist, Arial, sans-serif';
It seems Geomanist is not used at first render and when it suddenly is used, it causes the legend to get pushed offscreen, resulting in one missing legend entry.
I'm using Vue and Vue-chartjs FYI, but that shouldn't impact this
https://gfycat.com/leafyvictoriouslabradorretriever

I'm guessing the reason that it's not used at first is some kind of font loading issue where the first render happens before the font is loaded.
A couple of questions

Are you able to create a fiddle that reproduces this?
What version are you using?

Closing due to lack of test case


Build/Fix:

association set up, fixed model name from Plants to Plant, URL for plant image, spelling mistakes, auto increment.

I also changed the file name in models from 'Plants' to 'Plant' so it is more uniform with the rest of the file structure and project


calender is setting event for all years with same month and day.

When setting an event in the calendar, the event is scheduled not only for the current year but also for every year that has the same month and date.
Go to the calendar application.
Create a new event and set it for a specific date and time.
Save the event.
Observe that the event is scheduled not just for the selected year but also for future years on the same month and day.
Expected behavior
The event should be set only for the current year unless specified as a recurring event.

I have solved the issue plz label this as gssoc extend ,hactoberfest,level 3
and assign me this issue.

@SaurabhS55 @pulkitpathak99 I'm assigning the issue, kindly add the level label.

thx @arushi2610

you can assign this task to me...


Update README.md

The gradle-howto in readme.md is wrong. I needed to do
apply plugin: 'hugo'
instead of the 'com.jakewharton.hugo' line

@JakeWharton I am using com.jakewharton.hugo:hugo-plugin:1.1.0 and I had to use apply plugin: 'hugo' as when I used apply plugin: 'com.jakewharton.hugo', I got the following error:

Error:(13, 0) Plugin with id 'com.jakewharton.hugo' not found.

I am aware. For the version on master, however, that is not true.


update backstage app-config

What this PR does / why we need it:
Update the backstage config with the pollingInterval field
Change type

[ ] New feature
[X] Bug fix
[ ] Unit tests
[ ] Integration tests
[ ] CI
[ ] Documentation
[ ] Auto-generated SDK code

Impacted services

[X] Workflow Service
[ ] Notification Service

Checklist

[ ] Subject and description added to both, commit and PR.
[ ] Relevant issues have been referenced.

/lgtm

I already addressed this one here:
https://github.com/parodos-dev/parodos/pull/383


Improve FF and files path caching

Task/Issue URL:
Description
Steps to test this PR
Feature 1

[ ]
[ ]

UI changes

Before
After

!(Upload before screenshot)
(Upload after screenshot)

#5257  👈
develop

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @CrisBarreiro and the rest of your teammates on  Graphite


Flashbax in-place updating of buffer-state compatibility with end-to-end jitted RL training style of PureJAXRL or JaxMARL?

Hi all! First off, thank you for putting together this excellent repo, it's saving me quite a bit of time from having to implement my own buffers from scratch. I really like the end-to-end jitting that repos like PureJAXRL or JaxMARL use for implementing their algorithms (because you can vmap and pmap your entire training loop); however, the Flashbax documentation with regards to the usage of the donate_argnums parameter indicates I need to have my buffer initialized before my top-level function is jitted, which doesn't seem to be compatible with the end-to-end jitting approaches of the aforementioned repos.
I'm still new to JAX so I was wondering if there's any way around this or any way these two approaches can be reconciled so I can still get the performance benefits of in-place buffer updating while also being able to do end-to-end jitting of my RL training loop? Thanks in advance.

Understood, thank you very much @callumtilbury!

Hello @Chulabhaya Essentially, it depends on what you mean by initialise the buffer? Creating the buffer function needs to be done in a non-jitted function but the actual initialisation of buffer state can be done inside a jitted function. What this means is that all you have to do is pass the buffer_fn into a jitted function as a static argument and then the state can be created from within if you so wish. Here is an example:
First i would set up my learner in some way, below is an excerpt of how i would do this. This is a non-jitted function. Now in this example, i use buffer_fn.init(...) in a non-jitted function because i do not want to be resetting my buffer everytime i call my learn fn but thats a choice on my behalf and is not due to me being required to call init outside.
def learner_setup(
    env: Environment, keys: chex.Array, config: DictConfig
) -> Tuple[LearnerFn[DQNLearnerState], Actor, DQNLearnerState]:
    """Initialise learner_fn, network, optimiser, environment and states."""
    # Get available TPU cores.
    n_devices = len(jax.devices())

    # Get number of actions.
    action_dim = int(env.action_spec().num_values)
    config.system.action_dim = action_dim

    # PRNG keys.
    key, q_net_key = keys

    # Define networks and optimiser.
    q_network_torso = hydra.utils.instantiate(config.network.actor_network.pre_torso)
    q_network_action_head = hydra.utils.instantiate(
        config.network.actor_network.action_head,
        action_dim=action_dim,
        epsilon=config.system.training_epsilon,
    )

    q_network = Actor(torso=q_network_torso, action_head=q_network_action_head)

    eval_q_network_action_head = hydra.utils.instantiate(
        config.network.actor_network.action_head,
        action_dim=action_dim,
        epsilon=config.system.evaluation_epsilon,
    )
    eval_q_network = Actor(torso=q_network_torso, action_head=eval_q_network_action_head)

    q_lr = make_learning_rate(config.system.q_lr, config, config.system.epochs)
    q_optim = optax.chain(
        optax.clip_by_global_norm(config.system.max_grad_norm),
        optax.adam(q_lr, eps=1e-5),
    )

    # Initialise observation: Select only obs for a single agent.
    init_x = env.observation_spec().generate_value()
    init_x = jax.tree_util.tree_map(lambda x: x[None, ...], init_x)

    # Initialise q params and optimiser state.
    q_online_params = q_network.init(q_net_key, init_x)
    q_target_params = q_online_params
    q_opt_state = q_optim.init(q_online_params)

    params = QsAndTarget(q_online_params, q_target_params)
    opt_states = q_opt_state

    vmapped_q_network_apply_fn = q_network.apply

    # Pack apply and update functions.
    apply_fns = vmapped_q_network_apply_fn
    update_fns = q_optim.update

    # Create replay buffer
    dummy_transition = Transition(
        obs=jax.tree_util.tree_map(lambda x: x.squeeze(0), init_x),
        action=jnp.zeros((), dtype=int),
        reward=jnp.zeros((), dtype=float),
        done=jnp.zeros((), dtype=bool),
        next_obs=jax.tree_util.tree_map(lambda x: x.squeeze(0), init_x),
        info={"episode_return": 0.0, "episode_length": 0},
    )

    buffer_fn = fbx.make_item_buffer(
        max_length=config.system.buffer_size,
        min_length=config.system.batch_size,
        sample_batch_size=config.system.batch_size,
        add_batches=True,
        add_sequences=True,
    )
    buffer_fns = (buffer_fn.add, buffer_fn.sample)
    buffer_states = buffer_fn.init(dummy_transition)

    # Get batched iterated update and replicate it to pmap it over cores.
    learn = get_learner_fn(env, apply_fns, update_fns, buffer_fns, config)
    learn = jax.pmap(learn, axis_name="device")

Here is an example of creating the learn function that is pmapped and uses the buffer functions. In this example, the DQNLearnerState has the buffer_state object and adds to it. All functions created inside get_learner_fn are jittable. To answer your question, in theory if you wanted to, you call call buffer_fn.init to create a new clean buffer state everytime learn is called but that would be a waste of precious data.
def get_learner_fn(
    env: Environment,
    q_apply_fn: ActorApply,
    q_update_fn: optax.TransformUpdateFn,
    buffer_fns: Tuple[Callable, Callable],
    config: DictConfig,
) -> LearnerFn[DQNLearnerState]:
    """Get the learner function."""

    buffer_add_fn, buffer_sample_fn = buffer_fns

    def _update_step(learner_state: DQNLearnerState, _: Any) -> Tuple[DQNLearnerState, Tuple]:
        def _env_step(learner_state: DQNLearnerState, _: Any) -> Tuple[DQNLearnerState, Transition]:
            """Step the environment."""
            q_params, opt_states, buffer_state, key, env_state, last_timestep = learner_state

            # SELECT ACTION
            key, policy_key = jax.random.split(key)
            actor_policy = q_apply_fn(q_params.online, last_timestep.observation)
            action = actor_policy.sample(seed=policy_key)

            # STEP ENVIRONMENT
            env_state, timestep = jax.vmap(env.step, in_axes=(0, 0))(env_state, action)

            # LOG EPISODE METRICS
            done = timestep.last().reshape(-1)
            info = timestep.extras["episode_metrics"]

            transition = Transition(
                last_timestep.observation, action, timestep.reward, done, timestep.observation, info
            )

            learner_state = DQNLearnerState(
                q_params, opt_states, buffer_state, key, env_state, timestep
            )
            return learner_state, transition

        # STEP ENVIRONMENT FOR ROLLOUT LENGTH
        learner_state, traj_batch = jax.lax.scan(
            _env_step, learner_state, None, config.system.rollout_length
        )

        params, opt_states, buffer_state, key, env_state, last_timestep = learner_state

        # Add the trajectory to the buffer.
        buffer_state = buffer_add_fn(buffer_state, traj_batch)

        def _update_epoch(update_state: Tuple, _: Any) -> Tuple:
            """Update the network for a single epoch."""

            def _q_loss_fn(
                q_params: FrozenDict,
                target_q_params: FrozenDict,
                transitions: Transition,
            ) -> jnp.ndarray:

                q_tm1 = q_apply_fn(q_params, transitions.obs).preferences
                q_t = q_apply_fn(target_q_params, transitions.next_obs).preferences

                # Cast and clip rewards.
                discount = 1.0 - transitions.done.astype(jnp.float32)
                d_t = (discount * config.system.gamma).astype(jnp.float32)
                r_t = jnp.clip(
                    transitions.reward, -config.system.max_abs_reward, config.system.max_abs_reward
                ).astype(jnp.float32)
                a_tm1 = transitions.action

                # Compute Q-learning loss.
                batch_loss = q_learning(
                    q_tm1,
                    a_tm1,
                    r_t,
                    d_t,
                    q_t,
                    config.system.huber_loss_parameter,
                )

                loss_info = {
                    "q_loss": batch_loss,
                }

                return batch_loss, loss_info

            params, opt_states, buffer_state, key = update_state

            key, sample_key = jax.random.split(key)

            # SAMPLE TRANSITIONS
            transition_sample = buffer_sample_fn(buffer_state, sample_key)
            transitions: Transition = transition_sample.experience

            # CALCULATE Q LOSS
            q_grad_fn = jax.grad(_q_loss_fn, has_aux=True)
            q_grads, q_loss_info = q_grad_fn(
                params.online,
                params.target,
                transitions,
            )

            # Compute the parallel mean (pmean) over the batch.
            # This calculation is inspired by the Anakin architecture demo notebook.
            # available at https://tinyurl.com/26tdzs5x
            # This pmean could be a regular mean as the batch axis is on the same device.
            q_grads, q_loss_info = jax.lax.pmean((q_grads, q_loss_info), axis_name="batch")
            q_grads, q_loss_info = jax.lax.pmean((q_grads, q_loss_info), axis_name="device")

            # UPDATE Q PARAMS AND OPTIMISER STATE
            q_updates, q_new_opt_state = q_update_fn(q_grads, opt_states)
            q_new_online_params = optax.apply_updates(params.online, q_updates)
            # Target network polyak update.
            new_target_q_params = optax.incremental_update(
                q_new_online_params, params.target, config.system.tau
            )
            q_new_params = QsAndTarget(q_new_online_params, new_target_q_params)

            # PACK NEW PARAMS AND OPTIMISER STATE
            new_params = q_new_params
            new_opt_state = q_new_opt_state

            # PACK LOSS INFO
            loss_info = {
                "total_loss": q_loss_info["q_loss"],
            }
            return (new_params, new_opt_state, buffer_state, key), loss_info

        update_state = (params, opt_states, buffer_state, key)

        # UPDATE EPOCHS
        update_state, loss_info = jax.lax.scan(
            _update_epoch, update_state, None, config.system.epochs
        )

        params, opt_states, buffer_state, key = update_state
        learner_state = DQNLearnerState(
            params, opt_states, buffer_state, key, env_state, last_timestep
        )
        metric = traj_batch.info
        return learner_state, (metric, loss_info)

    def learner_fn(learner_state: DQNLearnerState) -> ExperimentOutput[DQNLearnerState]:
        """Learner function.

        This function represents the learner, it updates the network parameters
        by iteratively applying the `_update_step` function for a fixed number of
        updates. The `_update_step` function is vectorized over a batch of inputs.

        """

        batched_update_step = jax.vmap(_update_step, in_axes=(0, None), axis_name="batch")

        learner_state, (episode_info, loss_info) = jax.lax.scan(
            batched_update_step, learner_state, None, config.system.num_updates_per_eval
        )
        return ExperimentOutput(
            learner_state=learner_state,
            episode_metrics=episode_info,
            train_metrics=loss_info,
        )

    return learner_fn

There are variety of examples of off-policy single-agent algorithms using flashbax that can be found here if it helps your understanding: https://github.com/EdanToledo/Stoix/

Hey @Chulabhaya! I trust @EdanToledo's comments were helpful.
Re: the donate_argnums, I have had similar questions about in-place updates of buffer state. I've wondered: if I jit the outer function, will JAX respect / know to donate the memory inside the function, or should I be explicit?
I created the following MWE notebook which will hopefully help our discussion further.
https://colab.research.google.com/github/instadeepai/flashbax/blob/discussion/memory-donation/fbx_donate_argnums.ipynb
Findings:

vanilla loop = peak memory: 5052.51 MiB, increment: 4578.70 MiB
jitted loop = peak memory: 2765.12 MiB, increment: 2290.14 MiB
donate loop = peak memory: 5054.29 MiB, increment: 4577.61 MiB
jitted donate loop = peak memory: 2766.87 MiB, increment: 2290.04 MiB

These (empirical) results seem to suggest the following:

No problems in creating buffer state inside the jitted context, as highlighted by Edan 💯
JAX is smart enough to do the memory donation inside the training loop, even when we're not explicit. So we can just jit the outermost function, as per usual 👍🏻

If you take a look at that, can you let us know if we're understanding and answering your question? If not, let's keep discussing! :)

Hi @EdanToledo and @callumtilbury! Thank you guys very much for your extremely detailed replies, I really appreciate it! So I am currently setting up my buffer as follows, where everything is compiled but my buffer is also initialized within the compiled training function:
def make_train(args):
    # Initialize number of total training timesteps
    args.training_timesteps = args.total_timesteps - args.learning_starts

    # Set up environment
    env, env_params = make_env(args.env_id)

    # Training function
    def train(key):
        # Initialize replay buffer and buffer state
        replay_buffer = fbx.make_flat_buffer(
            args.buffer_size, args.learning_starts, args.batch_size, False, None
        )
        # TODO: Generalize this
        buffer_state = replay_buffer.init(
            {
                "observations": jnp.array([5.0, 4.0, 2.0, 1.0]),
                "actions": jnp.array(0),
                "rewards": jnp.array(1.0),
                "next_observations": jnp.array([5.0, 4.0, 2.0, 1.0]),
                "dones": jnp.array(True),
            }
        )
        ...
if __name__ == "__main__":
    # Seeding
    key = jax.random.PRNGKey(1)

    # Compile training function
    train_vjit = jax.jit(make_train(args))

    # Run training
    train_output = jax.block_until_ready(train_vjit(key))

Based on what you guys are saying it seems like my initialization of the flat_buffer within the train function is incorrect? What might the consequences of this approach be?

I thought it possible that there might be some side-effects but looking at the fbx.make_flat_buffer function it actually might be fine. Although what I will say is we haven't explicitly tested this in our unit tests, only the functions of the buffer itself has been tested. @callumtilbury what do you think? Can you see a problem presenting itself here?

A follow up question I have is that I'm doing some offline RL work where in my PyTorch-based code I usually load my dataset from a pickle file into a replay buffer, and then sample batches. If I'm understanding correctly, when I jit my entire training loop, including the initialization of the buffer, then that means my replay buffer is contained entirely on the GPU, right? This would therefore be untenable in the offline setting when your datasets can be large and might not necessarily fit on the GPU?

Hey @Chulabhaya! :)

So I am currently setting up my buffer as follows, where everything is compiled but my buffer is also initialized within the compiled training function:

Are there any errors? Is training as fast as you expect? As Edan says, the code seems fine on the surface, but JAX is a mysterious beast 😂 I am wondering if there are any sanity checks we can use in your example?

What might the consequences of this approach be?

I can't think of anything right now, but not 100% convinced either way.

I'm understanding correctly, when I jit my entire training loop, including the initialization of the buffer, then that means my replay buffer is contained entirely on the GPU, right?

I believe so, yes. And indeed, this is a challenge of processing large amounts (> VRAM) of data—as in Offline RL—in a jitted context. On our side, we've been brainstorming ways to efficiently do this: train with buffer batches taken from a Vault (inside of jit), and periodically fetch more data from the Vault (outside of jit).

Hi @callumtilbury, thanks for your reply! So to answer your questions, I haven't gotten any errors and performance is definitely as fast as I expect so that's credit to you guys and the great package you put together!
What style of jitting would you most recommend currently if I'd like to use Flashbax for offline learning? Obviously end-to-end jitting won't work because you can't fit a huge buffer containing an offline dataset onto a GPU. Do you think my best option for now is to just jit the individual loss function components and keep the overall training loop where I sample batches un-jitted?

I'm no expert in offline RL specifically so @callumtilbury might have better insights but I imagine what would be probably the best approach is to create a jitted train fn that takes parameters and buffer state in as arguments and returns the trained parameters. With this you can then cycle through vault data with different sized slices (which are their own buffer state) and setup the train function to train on each data slice for however long you want. I imagine this would be the fastest however it would probably require pretty decent tuning for things like size of data slice, number of gradient steps you do per execution of the train function as well as how you would cycle through the vault data outside of the train function. This would give you flexibility to trade off end-to-end training speed with larger dataset storage on a cpu. If you only performed one gradient step inside the train function this would intuitively just boil down to separately jitting everything. @callumtilbury what do you think of something like this as a compromise between speed and memory capability?

Hey @Chulabhaya & @EdanToledo :)

I imagine what would be probably the best approach is to create a jitted train fn that takes parameters and buffer state in as arguments and returns the trained parameters.

That is my thinking too, and the approach we're taking 🙏🏻
(But honestly, we're still figuring out a lot of this stuff—e.g. the hyperparameters Edan mentioned are not well known. Hopefully we'll continue to develop an understanding of it all, and will share with the community as we do.)

Hey @Chulabhaya! Do you have any further points for discussion here? If not, I'll close the issue, but we're always available for further chats ☺

Closing for now, but happy to revive discussions at any point :)

Hi @callumtilbury and @EdanToledo! Apologies for the delay in getting back to you guys, I've been caught up in work. No further points from me for now, I really appreciate your guys' insight and look forward to more awesome updates to flashbax in the future. Thanks again for following up with all my questions!


Update handling of the bucket name

for https://jirab.statcan.ca/browse/BTIS-408

Here is what the container name results into with these latest commits

I think i can clean it up a bit more. Like in one of the comments i added in the code, it might look better to only add a number to the name when a duplicate value is expected

Changed to take the last part of the mount path for the container name. This will help us a bit more to differenciate the different containers. If you see the above picture, they both just say "s3bucket"
But if i take the last part,

it's a bit easier to differenciate the two containers based on the name

I wasn't too creative with my test values for shares, but at least we see the distinct container names because a number was added.


Uploading Blob with empty trying value metadata resolves as a null

@azure/storage-blob:
"12.11.0":
MAC OS:
nodejs v18.3.0
chrome
typescript 4.7.4

Describe the bug
Uploading a blob with a value in the metadata as a empty string "" resolves as null metadata value header in the blob put request.
To Reproduce

Upload a a file with a metadata value as an empty string like below:

 await blockBlobClient.upload(
      selectedFile,
      selectedFile?.arrayBuffer.length,
      {
        metadata: {
          information: "",
          file_name: "the_test"
        },
      }
    );

Check the request and the blob storage and see that information is null and also the blob storage container file as seen in pictures below:

Expected behavior
I except the value to be a empty string not null

I think this happens somewhere in the  sdk/core/core-client/src/urlHelpers.ts, but got to dig deeper in the code to see where this mapping happens.

Oh it just occurs to me that blob storages do not allow empty strings for metadata values, so maybe that's why its mapped to null?

@EmmaZhu can answer definitively, but I suspect there isn't much difference between null and empty string given the way these are encoded as HTTP headers: https://learn.microsoft.com/en-us/rest/api/storageservices/set-blob-metadata?tabs=azure-ad
Short of treating null as some special string value, the header value would simply be empty, which I am assuming is parsed as 'null' by the service.

Hey @xirzec yeah I think you are correct. So I don't think there is anything this package can do. So ill close this issue.
I thought I would also post for anyone else who runs into this in the future, but If you have a mount storage connection between an app in Azure app service and a storage container and you upload a blob file to that storage container that has metadata value which is null/empty string, then this will silently crash the mount connection between Azure App service and that storage container (I.e the mount will list no files in the storage container, even though there are files). Sometimes it will also crash Kudu running the app service.  Only way to fix it is make that metadata value not null or remove the file.
TL;DR: Be careful when uploading empty string metadata to a storage container which is being used as mount.


Missing Functions in train_eval.py script

from utils import MultiModalReader, MultiModalDataset, VitalDiscretizer, VitalNormalizer, PhysiDiscretizer, PhysiNormalizer
The following functions are missing, could not find them anywhere in the code.
Could you please add them to the codebase.
Thanks in advance,
AA

Thank you for pointing out this issue. VitalDiscretizer and  PhysiDiscretizer were at vital_preprocess.py , physi_preprocess.py. We have fixed issues with new version.


Hierarchial Configuration

Hi Tom,
Congrats and thanks for your good initiative.
IMHO, As the intention is to provide a gateway for MicroServices we shall split the reroute configuration.
New  MicroService level configuration or Downstream host level configuration, here we shall have the Authentication part.  we use identity server to protect our client API,  Instead of specifying this for each route we shall specify at top level.
A wild card mapping will be a lot interesting as well.
Waiting to hear from you, will send a pull request soon.  Cheers

@abdulkaderjeelani Sounds good, when you do the pull request please do not break existing ways of configuring Ocelot. It need to support many configuration methods!


Test samples from their installed location

The original build system tested the samples in their installed location, that is Open Enclave was installed itself under the build output directory and the samples were run there. When cmake was introduced last year, the sample build scripts were reworked to reference includes, libraries, and commands under the source tree. This methodology is fragile and does not test whether the samples would actually build for the end-user.
Unfortunately, to get all this goodness, the old build system did a complete build and install. A compromise might be to use the DESTDIR variable so the samples could be installed under the output build directory as follows.
make DESTDIR=<output build directory>

This proved to be quite difficult. The DESTDIR option only kind of works. It correctly pointed a CMake installation to another directory, but it failed to update the CMAKE_INSTALL_PREFIX which meant that the installed CMake packages could not be used. I had to instead implement logic to update the cache variable before doing the test install for the samples, which is unfortunate, but it only happens if the tests are run.

The linked PR should close this issue, though it's only the first step as it fixes one sample, with @aserdean to follow up with conversions of the rest of the samples.

Actually, I believe I've just managed to make the CMake package relocatable, so my hack went away 😄

Closed via #1472.