how to restore an a big size arangodump

My Environment

ArangoDB Version:        3.4.2
Storage Engine:             RocksDB
Deployment Mode:
Deployment Strategy:   Manual Start
Infrastructure:               physical server/ PC server
Operating System:       centos 7
Total RAM in your machine:       e.g. 256G
Disks in use:      SSD
Used Package:

Problem:
I have a very huge collection, almost 300 million documents. when i used arangodump and arangorestore command to restore other server (two server's setting is the same),  restore performance is too slow and always failed I don't know something is wrong. and then I try to upgrade to 3.4 version . and add batch size and threads parameter. but still unable to restore.
below is my backup and restore command.
nohup arangodump --server.endpoint tcp://<IP_ADDRESS>:8529 --server.username root --server.password Password01! --server.database kgms  --output-directory "/opt/dump_prod_2019_1_30" 2>&1 &

nohup arangorestore --server.endpoint tcp://<IP_ADDRESS>:8529 --server.username root --server.password Password01! --server.database kgms --create-database true --collection kg_entity_12236201 --collection kg_edage_12236201 --input-directory "dump_prod_2019_1_30"  --threads 4  --batch-size 100663296 --force true 2>&1 &

anyone can give me some suggestions, thanks a lot.

Can you add the details about your environment please?

ArangoDB Version: 3.4.2
Storage Engine: rocksdb
Deployment Mode: manual set up
Infrastructure: physical server/ PC server
Operating System: centos 7
Total RAM in your machine: 256 G
Disks in use: SSD

Hi @zeroluojin,
The value passed for the option --batch-size seems to be to high (100MB), can you lower that value and retry ? We suggest the default value of  8388608
Best,

Hi @zeroluojin,
The value passed for the option --batch-size seems to be too high (100MB), can you lower that value and retry ? We suggest the default value of  8388608
Best,

thank you for you reply
Actually, for this option value, i have many times experience, things like default value of  8388608,
50 MB, 80 MB, 100 MB. but if i set 8388608, the performance is too slow,  the progress is 20% need to take 2 days and the last information is the response time out.
and I have another question, can i just direct to copy data file directory to another server. to implement 'cold backup' ?

Closing this as hot backup is available.
In other news, ArangoDB 3.7 is available.

Warum wurde Issue geschlossen? Das Problem besteht weiterhin.  Arangodump funktioniert mittlerweile super schnell, Arangorestore ist weiter super langsam.  Es sollte auch eine vernünftige Alternative ohne Enterprise geben. Andere DB bekommen das auch hin. Auch ArangoDb 3.7 löst das Problem nicht. Ich habe eine 2,4 GB große Datenbank und sie braucht für den Dump 70 Sekunden, für den Import aber über 22 Minuten!! Hardware: 12 Kerne, 32 GB RAM, SSD mit 3000MB/sec. Alle --thread und --batch-size Einstellungen wurden bereits ausprobiert (8,80,800 MB). Der Import variiert nur in 1 bis zwei Minuten, bleibt aber immer über 20 Minuten. Der Issue sollte wieder geöffnet werden, da Arangorestore im Moment als Backup bzw. Restore nicht zufriedenstellend funktioniert und kein funktionierendes Backup-Konzept dafür ist. Danke

Long story short: restoring requires much more computing power than dumping, hence this can not become be as fast.
Detail reasons:

indices have to be maintained; hence hashes have to be calculated, and checked for clashes
documents have to be compressed
RocksDB will do internal compaction runs to ensure disk usage is acceptable
replication has to send documents to followers

At least the RocksDB step could be optimized, by configuring your environment along the lines to become faster:
Reducing the Memory Footprint of ArangoDB servers
(while the title the article will make you expect the opposite, the article explains which screws to turn to reduce the footprint and what the resulting effects are. For sure these effects can also be moved into the opposite direction using the same paramters)
HotBackup on the other hand uses orchestrated approach to get RocksDB snapshots, and to be able to get in the shortest possible time frame a persistent on disk state, which it can use to copy the data afterward this snapshotting has been finished, while the actual operations may continue.
But, as you already said, its an enterprise feature, and it will remain that way. So from our perspective this issue is solved.
The next best approach you could choose is to do a short stop of your database, and use LLVM to capture a snapshot of your on disk data.

But, as you already said, its an enterprise feature, and it will remain that way. So from our perspective this issue is solved.

Really, that's your answer to a problem. We have a premium feature, so that's the end of the matter for us? Is this a typical German answer? I find this very clumsy, especially for developers who want to switch to ArangoDB.
A reasonable import/export should offer every database without enterprise version. Of course I understand that an import always takes longer, I have been working with databases for over 20 years. But with you the time ratio of export/import is just not right: 70 sec vs 1200 sec ?? I have already considered the article "Reducing the Memory Footprint ...". We run a website with ArangoDB, so we can't just stop the database.
Maybe you should think about it again and get a developer to do it. I think I am not the only developer who would like to use ArangoDB, but finds the import time clearly too long.

We should not close this issue unless we understand why the restore is slow.

Thanks

@admtech : I think I am a bit late to the game here, so just trying to understand the issue.
Is my following understanding correct?

the current ArangoDB version in use is 3.7.x, but the problem was always present (i.e. no new issue)
the performance of arangodump is fine, but the performance of arangorestore is not (restore is about factor 20 slower than dump)
the restore is done into a single server (i.e. no cluster)
when restoring, the target database/collection do not exist
there is one large collection that is restored and that is creating the problem. if not, how many collections are restored and what are their sizes?

Especially the last point would be important to understand. If there is more than one collection to restore, arangorestore can use multiple threads (controlled via --threads option) to restore the data into multiple collections concurrently.
However, if there is one collection which is substantially larger than the other collections in the dump, the restore may work effectively single-threaded for a long time while this collection is processed.
If the restore is really working mostly on a single collection, I am wondering if splitting the input file into multiple chunks would help here, and then parallelizing the restore of the chunks. That is nothing that arangorestore can do right now, but it may be a way to improve it for large collections.
What may also have an impact on restore performance is the collection's waitForSync property, whether or not the collection has a schema attached and the secondary indexes that are present for the collection. Can you share these details with us?
From the arangosh, that would be the output of db.<collection-name>.properties(); and of db.<collection-name>.indexes().
Finally, starting in ArangoDB 3.8 there is a new option for arangodump (--envelope) that controls the dump format. The default value for it is true, meaning the format will remain unchanged. However, when setting this option to false, the generated dumps will be smaller and slightly quicker to produce. arangorestore in version 3.8 can handle such dumps automatically, and I guess the restore time will also go down by using this format.
I don't expect anything near a factor of 20 here, rather a small percentage.
To make restore massively faster we may need to parallelize restoring of a single collection. But it would be good if you could let me know if my above assumptions are appropriate, and if you could share the collection properties and indexes info.
Thanks!

Hello @jsteemann,
I'm glad you're following up. I'll try to answer all the questions:

the current ArangoDB version in use is 3.7.x, but the problem was always present (i.e. no new issue)

yes, ArangoDB version 3.7.10 and the problem has existed since we started using Arangodb (we started with 3.4)

the performance of arangodump is fine, but the performance of arangorestore is not (restore is about factor 20 slower than dump)

Exactly, the speed of arangodump is very good.  Our database is 2.4 GB and it takes about 70 seconds. For the restore of the 2.4 GB database it currently needs 20 to 22 minutes (AMD 12 cores, 32 GB RAM, SSD with 3000MB/sec.) So it can't be the performance of the hardware or disks.

the restore is done into a single server (i.e. no cluster)

yes, currently it is still a single server

when restoring, the target database/collection do not exist

Good question: we use: "arangorestore --create-database" and for the test we imported all this into a new DB. It didn't exist the first time, but it did for all subsequent tests. I'll test if there is a difference in the import. In any case, it is not a DB that anyone else is accessing in parallel.

there is one large collection that is restored and that is creating the problem. if not, how many collections are restored and what are their sizes?

Our database contains several large table:

Articles = 500.000 entries
Comments = 1.5 million entries
Jobs =. 60.000 entries
Users = 130.000 entries
Messaging system =. 200.000 entries
Geo data = 300.000 entries
and then many small collections

Articles, Users and Comments are the biggest items, also in terms of volume. It takes the longest time for these 3 collections.

Especially the last point would be important to understand. If there is more than one collection to restore, arangorestore can use multiple threads (controlled via --threads option) to restore the data into multiple collections concurrently.

We have tested the "-threads x" option extensively (4, 6, 8, 12). With "dump" it really brings a lot, with "restore" it has almost no effect with us.

However, if there is one collection which is substantially larger than the other collections in the dump, the restore may work effectively single-threaded for a long time while this collection is processed.

These are then the above described three collection "User", "Comments" and "Articles". A certain parallelism by "threads" I can not recognize on the basis of the output, however.

If the restore is really working mostly on a single collection, I am wondering if splitting the input file into multiple chunks would help here, and then parallelizing the restore of the chunks. That is nothing that arangorestore can do right now, but it may be a way to improve it for large collections.

But then arangodump would have to support that as well.

What may also have an impact on restore performance is the collection's waitForSync property, whether or not the collection has a schema attached and the secondary indexes that are present for the collection. Can you share these details with us?
From the arangosh, that would be the output of db..properties(); and of db..indexes().

Schemas are not attached. The "waitForSync" property is set to "false" for all collections.
Here are the outputs of the big 3 collections:
COMMENTS:
db.comments.properties()
{
"isSystem" : false,
"waitForSync" : false,
"keyOptions" : {
"allowUserKeys" : true,
"type" : "traditional",
"lastValue" : 67350216
},
"writeConcern" : 1,
"cacheEnabled" : false,
"schema" : null
}
db.comments.indexes()
[
{
"fields" : [
"_key"
],
"id" : "comments/0",
"name" : "primary",
"selectivityEstimate" : 1,
"sparse" : false,
"type" : "primary",
"unique" : true
},
{
"deduplicate" : true,
"fields" : [
"status",
"languages",
"stamp_create"
],
"id" : "comments/82766853",
"name" : "stamp_create",
"selectivityEstimate" : 0.9974430780469986,
"sparse" : false,
"type" : "persistent",
"unique" : false
},
{
"deduplicate" : true,
"fields" : [
"_key",
"status"
],
"id" : "comments/82766904",
"name" : "comment_key",
"selectivityEstimate" : 1,
"sparse" : false,
"type" : "persistent",
"unique" : true
},
{
"deduplicate" : true,
"fields" : [
"rating.count",
"rating.user",
"status"
],
"id" : "comments/82766924",
"name" : "rating",
"selectivityEstimate" : 0.00001980741533037354,
"sparse" : false,
"type" : "persistent",
"unique" : false
},
{
"deduplicate" : true,
"fields" : [
"reply_to_comment",
"status"
],
"id" : "comments/82766934",
"name" : "reply",
"selectivityEstimate" : 0.21391268017547524,
"sparse" : false,
"type" : "persistent",
"unique" : false
},
{
"deduplicate" : true,
"fields" : [
"content_key",
"status"
],
"id" : "comments/82766944",
"name" : "content_key",
"selectivityEstimate" : 0.13676811860360286,
"sparse" : false,
"type" : "persistent",
"unique" : false
},
{
"deduplicate" : true,
"fields" : [
"content_key",
"status",
"reply_to_comment"
],
"id" : "comments/82766958",
"name" : "detail_comments",
"selectivityEstimate" : 0.3000952450728991,
"sparse" : false,
"type" : "persistent",
"unique" : false
}
]
USER:
db.users.properties()
{
"isSystem" : false,
"waitForSync" : false,
"keyOptions" : {
"allowUserKeys" : true,
"type" : "traditional",
"lastValue" : 70904606
},
"writeConcern" : 1,
"cacheEnabled" : false,
"schema" : null
}
db.users.indexes()
[
{
"fields" : [
"_key"
],
"id" : "users/0",
"name" : "primary",
"selectivityEstimate" : 1,
"sparse" : false,
"type" : "primary",
"unique" : true
},
{
"deduplicate" : true,
"fields" : [
"status",
"stamp_create",
"language"
],
"id" : "users/83718800",
"name" : "create",
"selectivityEstimate" : 0.9915274751876059,
"sparse" : false,
"type" : "persistent",
"unique" : false
},
{
"deduplicate" : true,
"fields" : [
"user_hash",
"status",
"personal.privacy"
],
"id" : "users/83718804",
"name" : "user_profile",
"selectivityEstimate" : 1,
"sparse" : false,
"type" : "persistent",
"unique" : false
}
]
CONTENT:
db.content.properties()
{
"isSystem" : false,
"waitForSync" : false,
"keyOptions" : {
"allowUserKeys" : true,
"type" : "traditional",
"lastValue" : 78444052
},
"writeConcern" : 1,
"cacheEnabled" : false,
"schema" : null
}
db.content.indexes()
[
{
"fields" : [
"_key"
],
"id" : "content/0",
"name" : "primary",
"selectivityEstimate" : 1,
"sparse" : false,
"type" : "primary",
"unique" : true
},
{
"deduplicate" : true,
"fields" : [
"user_key",
"languages",
"status"
],
"id" : "content/87160379",
"name" : "user_key",
"selectivityEstimate" : 0.28707597420801795,
"sparse" : false,
"type" : "persistent",
"unique" : false
},
{
"deduplicate" : true,
"fields" : [
"status",
"primary_language"
],
"id" : "content/87160383",
"name" : "primary_language",
"selectivityEstimate" : 0.000007567273057481006,
"sparse" : false,
"type" : "persistent",
"unique" : false
},
{
"deduplicate" : true,
"fields" : [
"tag_key",
"languages",
"status"
],
"id" : "content/87160387",
"name" : "tag_keys",
"selectivityEstimate" : 0.000609165481127221,
"sparse" : false,
"type" : "persistent",
"unique" : false
},
{
"deduplicate" : true,
"fields" : [
"art",
"status",
"tag_key",
"tags"
],
"id" : "content/87160391",
"name" : "content_counter",
"selectivityEstimate" : 0.002746920119865605,
"sparse" : false,
"type" : "persistent",
"unique" : false
},
{
"deduplicate" : true,
"fields" : [
"tags",
"status",
"art",
"languages",
"stamp_create"
],
"id" : "content/87160401",
"name" : "content_tags_create",
"selectivityEstimate" : 1,
"sparse" : false,
"type" : "persistent",
"unique" : false
},
{
"deduplicate" : true,
"fields" : [
"status",
"languages",
"stamp_create"
],
"id" : "content/87160409",
"name" : "create_date",
"selectivityEstimate" : 0.9997559189650964,
"sparse" : false,
"type" : "persistent",
"unique" : false
},
{
"deduplicate" : true,
"fields" : [
"statistic.comments_count",
"status"
],
"id" : "content/87160423",
"name" : "comment_counter",
"selectivityEstimate" : 0.000003783636528740503,
"sparse" : false,
"type" : "persistent",
"unique" : false
},
{
"deduplicate" : true,
"fields" : [
"stamp_last_activity",
"status"
],
"id" : "content/87160427",
"name" : "last_activities",
"selectivityEstimate" : 0.999633923123856,
"sparse" : false,
"type" : "persistent",
"unique" : false
},
{
"deduplicate" : true,
"fields" : [
"status",
"statistic.read_count",
"stamp_create",
"languages"
],
"id" : "content/87160437",
"name" : "create_status_read_count",
"selectivityEstimate" : 1,
"sparse" : false,
"type" : "persistent",
"unique" : false
},
{
"deduplicate" : true,
"fields" : [
"status",
"art",
"stamp_create",
"languages"
],
"id" : "content/87160449",
"name" : "side_content",
"selectivityEstimate" : 0.9995119570522206,
"sparse" : false,
"type" : "persistent",
"unique" : false
}
]

Finally, starting in ArangoDB 3.8 there is a new option for arangodump (--envelope) that controls the dump format. The default value for it is true, meaning the format will remain unchanged. However, when setting this option to false, the generated dumps will be smaller and slightly quicker to produce. arangorestore in version 3.8 can handle such dumps automatically, and I guess the restore time will also go down by using this format.
I don't expect anything near a factor of 20 here, rather a small percentage.

Sounds good and we will give it a try.

To make restore massively faster we may need to parallelize restoring of a single collection. But it would be good if you could let me know if my above assumptions are appropriate, and if you could share the collection properties and indexes info.
Thanks!

I am happy to do so at any time.
Greetings
Frank

@admtech : I spent several hours looking into the reasons for the slow restore performance.
To simplify things for me, I focused on a single use case: restoring one large collection from a dump, with no secondary indexes.
Although this is not exactly your use case, improving this particular case should improve a lot of other cases too, e.g. restoring multiple large collections or restoring a mix of large and small collections.
One (known) problem with arangorestore is that when restoring only a single collection, it can't use any multi-threading. The restore thread that reads the input file blocks when sending a batch of data to the server until a response is received. Thus arangorestore is idle while a request is in flight (of course, this is only true when restoring a single collection - when restoring multiple collections the other threads can still do other work).
I have hammered out a change to arangorestore that can overcome this limitation, so that the thread that is reading the input file can dispatch the request-response handling to another thread. This means we will have the other thread working and the original thread can go on and read more of the input file: more parallel work and less idle time on the arangorestore side!
The restore thread that reads the input file is supposed to read the input data faster than what arangorestore can get to the server via request-response. In order to maximize parallel work, the reading thread is allowed to dispatch as many request-response jobs as there are idle arangorestore threads.
For example, when restoring a single collection with --threads 8, the thread reading the input file can dispatch to up to 7 other threads, so that up to 8 concurrent requests can be in flight for one collection.
While this sounds nice, there are also a few caveats:

for now, the change is only available in a PR: https://github.com/arangodb/arangodb/pull/14010. If the change works fine, it will be merged into devel, which will become 3.9 eventually. It may be backported to 3.8 as well, but very likely not to 3.7.
increased parallelism can only be used when restoring dumps that were created with the non-enveloped format (i.e. an arangodump run with --envelope false option). This option is available from 3.8 onwards, and its defaults to true. In 3.8 this option is opt-in. In devel/3.9 the PR will change the default value to false, so dumps will be slick and restore can use the improved parallelism
increasing the number of arangorestore threads currently doesn't lead to a linear speedup of the restore process. Rather, the speedup is sub-linear. This is because the work can be bottlenecked on reading and decompressing the input file. In addition, there is a critical section without multi-threading when a chunk of data is handed over to another thread and needs to be copied for that. Finally, restoring data will only be as fast as the server can ingest the data. Restoring data is a lot of work and a lot more expensive than dumping it with arangodump. While for dumping the data it is sufficient to iterate over all the documents in a collection and stream them, restoring requires checking for conflicts in primary and secondary indexes, and then requires writing the documents plus all their index entries.

For my particular test case (restoring a single collection, 9.5 million documents with 7 attributes each, no seconary indexes), the PR cuts the restore speed in half.
For other use cases, the mileage may vary, but I would expect at least some improvement from it.
I will update this ticket once the above PR gets merged to devel, so it can be tried out in a nightly should there be demand. I will also update the ticket in case the PR will be backported to 3.8, which may be even more useful as the expected GA date for 3.8 is in Q2.

An excellent development. I look forward to trying it out in 3.9 devel. Thanks for the effort and support. This is exactly how it should be, you guys did a great job here. Greetings to Cologne from Limburg a.d. Lahn :-)

Great, thanks for the info. As soon as it is available, we will test it.

@admtech : the latest change I made has been merged into 3.8 as well now.
It can be tried by building 3.8 from source or by using a nightly 3.8 build that includes the change (i.e. that has a build date >= now).
Nightly 3.8 builds can be found here: https://download.arangodb.com/nightly/3.8/index.html
Please note that nightlies are built only every few days, so it may take some time for the change to appear in a nightly. Note that this is an unreleased 3.8 version which should only be used for testing and nothing else yet.
With a bit of luck the changes will help speeding up the restore a bit, but to manage expectations, I don't expect a 10x speedup or such, rather something like cutting the restore time by 30 to 50%.
If the majority of the time is spent on writing data into the indexes, then my changes will not help so much. They will help mostly if the majority of time is spent waiting on server responses and arangorestore sitting idle meanwhile.

Hi,
so I tested the nightly "arangodb3-3.8.0-0.2.x86_64.rpm" today on our test server (Fedora 33, AMD 12 cores, 34 GB Ram, SSD).
The result is more than convincing.
{restore} Processed 14 collection(s) in 313.206299 s, read<PHONE_NUMBER> byte(s) from datafiles, sent 275 data batch(es) of<PHONE_NUMBER> byte(s) total size
We are now at very good times on the import. Currently the system needs 5 to 6 minutes for the restore.
Here is a small list of our current results:
1 run:
Dump: 2.81 GB DB, 12 Thread, Batchsize 10 MB = 1,23 minutes
Restore: 2.81 GB DB, 6 Thread, Batchsize 10 MB = 6,83 minutes
Restore: 2.81 GB DB, 12 Thread, Batchsize 10 MB = 5,18 minutes
2 run:
Dump: 2.81 GB DB, 12 Thread, Batchsize 10 MB = 1,26 minutes
Restore: 2.81 GB DB, 12 Thread, Batchsize 10 MB = 5,12 minutes
Restore: 2.81 GB DB, 12 Thread, ohne batchsize = 5,20 minutes
Restore: 2.81 GB DB, 12 Thread, Batchsize 100 MB = 5,94 minutes
Cross-check version 3.8:
Dump: 2.81 GB DB, 12 Thread, Batchsize 10 MB, envelope false = 1,23 minutes
Restore: 2.81 GB DB, 12 Thread, Batchsize 10 MB, envelope false  = 16,11 minutes
In general, 3.8 seems to be a bit faster than version 3.7 when restoring.
Cross-check version 3.7:
Restore: 2.72 GB DB, 12 Thread, Batchsize 10 MB,  envelope false  = 21,41 minutes
This is an increase of 68% (version 3.8 envelope false) and 76.5% (version 3.7) :-)
Thanks @jsteemann for the good work and fast integration. The journey from "case closed" to import improvement of 50-70% was worth it from our point of view. That's where we like to put time into testing. Thanks to everyone at ArangoDB for making this possible.
P.S.: maybe you should consider converting the seconds values in the output of arangodump and arangorestore into a directly readable format (h:m:s). Converting the seconds is a hassle every time. Similar is true for the byte values. ;-)
Keep up the good work

Hi @admtech, thanks for the feedback!
The performance improvements look indeed encouraging. I was expecting less from it :sweat_smile:
Regarding your suggestion for better readable outputs (times, bytes) of the client tools, I have addressed this partly via https://github.com/arangodb/arangodb/pull/14081.
This should improve legibility of the size values (bytes, MB, GB, TB) but not yet durations due to lack of time. We may fix this eventually, but I can't guarantee it.
Until then I can recommend prefixing the dump/restoring commands with the time builtin, which will automatically print the elapsed time the command has taken.


Clarify that "n sats per min" in jobs board is NOT the wage / compensation being offered for the job

There's WAY too many people who are confusing the "n sats per min" in the jobs board as being the wage being offered for the job.

Exhibit A: https://stacker.news/items/24097
Exhibit B: https://t.me/TheSimplyBitcoinChannel/52231

Related discussion:

Perhaps there needs to be a prefix (e.g., listing fee: n sats per min) or that being shown under column of its own -- with a title,  or perhaps some other way to explain to what that references.

https://stacker.news/items/24144

Another example of this occurring:
sats/min: Salary of the Future
https://stacker.news/items/31831

I've seen it on twitter too. Yikes.

I think that the sats per minute statistic can be removed entirely. From the end user perspective it doesn't add much to the information of the job and I can be misleading

Hey. Are y'all accepting PRs from outside of your organization? If so, I'm looking for a quick, easy issue that'll get me clicking throughout the codebase and this is a good candidate.
If you've decided on removing it, can I take this ticket?

If you've decided on removing it, can I take this ticket?

PRs are always welcome. Note, this specific PR might not be merged until a larger set of job board enhancements are introduced.


AxonDeepSeg Refactoring

PS: this will be discussed in the meeting on 07 Nov 18.
AxonDeepSeg nowadays has a very complex codebase, specially for the model definition where the TensorFlow low-level API is being used. What we want to do is to simplify some things to reduce the amount of complexity and code from the codebase.
One ideia is to migrate the code to use either Keras or PyTorch. This would be a change that will of course reuse the data loading mechanisms (image loading, patching, etc), but it might need some rework to make the dataset suitabe for each framework (Keras/PyTorch for example).
With that done, it would easier and faster to exchange the model for another model for the sake of experimentation, improvement, etc.
Some things to consider:

Should we go for PyTorch or Keras ?
Are there any other issues we want to address in this refactoring ?
What should be the use case for the users ? Should they be able to easily fine-tune the model with their data ? Should we fine-tune for them ? Due to the variability in images, I don't see much that can be done if not by fine-tuning on at least some patches of new data (which can be selected by active learning, the field that @MelanieLu is working);
Should we have a developer documentation (Sphinx) for newly developed code ?

If you have more points to consider or any other comments, please edit this issue or comment below.

Are there any other issues we want to address in this refactoring ?

Right now, some of the code is hardcoded very specifically for SEM/TEM (e.g. see below some examples). I think identifying where these occur, and how these can be generalised/extended for any new model we want to introduce, might also worth exploring during this refactoring.
https://github.com/neuropoly/axondeepseg/blob/476bb3fcf1a5092514c4c73d124a8e9f46d6015b/AxonDeepSeg/data_management/dataset_building.py#L126-L131
https://github.com/neuropoly/axondeepseg/blob/9e59f536da6c709d0341999ae2b11945de588cb2/AxonDeepSeg/segment.py#L221-L237

Closed, stale.


February 8th 2022, Community Meeting Agenda

Dapr Community Meetings
Every two weeks we host a community meeting to showcase new features, review upcoming milestones, and engage in Q&A with the Dapr community - all are welcome, and we encourage participation.
The purpose of this thread is to form a discussion amongst the Dapr community on potential topics to highlight during the meeting. If you have a topic you wish to present or learn more about, please comment and be sure to include your name and a short description of the topic.
The conversations in this thread will directly influence the meeting agenda for the community call on February 8th , 2022, at 10:00am PST. The meeting agenda will be finalized by a Community Call Host 3 days prior to the meeting.
Thank you and we look forward to hearing your input.

If there is space in the agenda at the end, I would like to get some feedback to walk through a set of proposals to improve Dapr middleware.

Adding support for gRPC middleware: https://github.com/dapr/dapr/issues/4153
Using CEL for middleware selectors: middleware https://github.com/dapr/dapr/issues/4167
Using Dapr components in middleware (to support use cases like caching): https://github.com/dapr/dapr/issues/2950

Absolutely, @jjcollinge. Adding this topic to the agenda. How much time should we allocate, 15-20 mins?

@msfussell @greenie-msft isn't Feb 8th the asian-friendly timezone call?

Absolutely, @jjcollinge. Adding this topic to the agenda. How much time should we allocate, 15-20 mins?

It'll only take me 5-10 mins to walk through the proposals. Then it depends on how much engagement/feedback we get.

@greenie-msft I just had a little run through the content I plan to talk about and actually I'm probably closer to 15/20 mins if I don't rush through it.

I may be too late for this meeting, but I made progress on integrating plugins into daprd https://github.com/dapr/dapr/issues/3787#issuecomment-1030957998

I believe we can squeeze that in. @greenie-msft does this close the agenda?

@patrickhuber, yes there is room! How much time would you like for this topic?

@patrickhuber you're on for tomorrow! Thanks

@patrickhuber, @jjcollinge do either of you have slides that you plan on presenting? If so, please send me a message on discord (nickgreenfield#4810) and I can send you a link to the deck.  Also, please try to connect to the community meeting 10 minutes early for a quick tech check.
See you then! Thanks!

@greenie-msft I don't have any slides, I will just be sharing my screen if that's ok? Joining 10 mins early is no problem👌

@jjcollinge Yup, that's perfect.

I don't have any slides. I will be doing a demo.
On Mon, Feb 7, 2022, 4:05 PM greenie-msft @.***> wrote:

@patrickhuber https://github.com/patrickhuber, @jjcollinge
https://github.com/jjcollinge do either of you have slides that you
plan on presenting? If so, please send me a message on discord
(nickgreenfield#4810) and I can send you a link to the deck. Also, please
try to connect to the community meeting 10 minutes early for a quick tech
check.
See you then! Thanks!
—
Reply to this email directly, view it on GitHub
https://github.com/dapr/community/issues/123#issuecomment-1031922969,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AABMQ7RIHZWEMA2WW75FFUDU2AXZ7ANCNFSM5JVGPJCA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you were mentioned.Message ID:
@.***>

If there is space in the agenda at the end, I would like to get some feedback to walk through a set of proposals to improve Dapr middleware.

Adding support for gRPC middleware
Using CEL for middleware selectors
Using Dapr components in middleware (to support use cases like caching)

If @daixiang0 has any further thoughts on WASM integration then this could follow on.

Sorry for slow response, I would like to share it in next community.

@greenie-msft is there time for @mstrYoda to demo his work on a HTTP cache middleware after me?

is there a chance to squeeze the "resiliency block" in?

is there a chance to squeeze the "resiliency block" in?

Next call 🤙

Passed


TX1 tensorNet.cpp.o Error 1

Hello,
I have tried to went through the tutorial. I have tried to build a source on Jetson. My jetson runs Ubuntu 16.04. I have followed the instructions; however when I run "make" on terminal, I have received following error:
Scanning dependencies of target jetson-inference
[ 53%] Building CXX object CMakeFiles/jetson-inference.dir/tensorNet.cpp.o
In file included from /home/nvidia/jetson-inference/tensorNet.cpp:23:0:
/home/nvidia/jetson-inference/tensorNet.h:135:18: error: ‘nvinfer1::IInt8Calibrator’ has not been declared
nvinfer1::IInt8Calibrator* calibrator=NULL, cudaStream_t stream=NULL );
^
/home/nvidia/jetson-inference/tensorNet.h:150:18: error: ‘nvinfer1::IInt8Calibrator’ has not been declared
nvinfer1::IInt8Calibrator* calibrator=NULL, cudaStream_t stream=NULL );
^
/home/nvidia/jetson-inference/tensorNet.h:247:19: error: ‘nvinfer1::IInt8Calibrator’ has not been declared
nvinfer1::IInt8Calibrator* calibrator, std::ostream& modelStream);
^
/home/nvidia/jetson-inference/tensorNet.cpp: In function ‘nvinfer1::DataType precisionTypeToTRT(precisionType)’:
/home/nvidia/jetson-inference/tensorNet.cpp:75:26: error: ‘kINT8’ is not a member of ‘nvinfer1::DataType’
case TYPE_INT8: return nvinfer1::DataType::kINT8;
^
/home/nvidia/jetson-inference/tensorNet.cpp: At global scope:
/home/nvidia/jetson-inference/tensorNet.cpp:297:20: error: ‘nvinfer1::IInt8Calibrator’ has not been declared
nvinfer1::IInt8Calibrator* calibrator,
^
/home/nvidia/jetson-inference/tensorNet.cpp: In member function ‘bool tensorNet::ProfileModel(const string&, const string&, const std::vector<std::__cxx11::basic_string >&, unsigned int, precisionType, deviceType, bool, int*, std::ostream&)’:
/home/nvidia/jetson-inference/tensorNet.cpp:421:86: error: ‘class nvinfer1::IBuilder’ has no member named ‘getInt8Mode’
device %s, building INT8:  %s\n", deviceTypeToStr(device), builder->getInt8Mode
^
/home/nvidia/jetson-inference/tensorNet.cpp: At global scope:
/home/nvidia/jetson-inference/tensorNet.cpp:463:20: error: ‘nvinfer1::IInt8Calibrator’ has not been declared
nvinfer1::IInt8Calibrator* calibrator, cudaStream_t stream )
^
/home/nvidia/jetson-inference/tensorNet.cpp:477:20: error: ‘nvinfer1::IInt8Calibrator’ has not been declared
nvinfer1::IInt8Calibrator* calibrator, cudaStream_t stream )
^
/home/nvidia/jetson-inference/tensorNet.cpp: In member function ‘bool tensorNet::LoadNetwork(const char*, const char*, const char*, const char*, const std::vector<std::__cxx11::basic_string >&, uint32_t, precisionType, deviceType, bool, int*, cudaStream_t)’:
/home/nvidia/jetson-inference/tensorNet.cpp:482:86: error: ‘NV_TENSORRT_PATCH’ was not declared in this scope
TensorRT version %u.%u.%u\n", NV_TENSORRT_MAJOR, NV_TENSORRT_MINOR, NV_TENSORRT
^
CMakeFiles/jetson-inference.dir/build.make:76: recipe for target 'CMakeFiles/jetson-inference.dir/tensorNet.cpp.o' failed
make[2]: *** [CMakeFiles/jetson-inference.dir/tensorNet.cpp.o] Error 1
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/jetson-inference.dir/all' failed
make[1]: *** [CMakeFiles/jetson-inference.dir/all] Error 2
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2
Could you please help me to solve this problem?
Do I have to build a host before building the source on Jetson?
Looking forward to hearing from you

Hi, which version of JetPack is your Jetson running?

From: ismailbaayer<EMAIL_ADDRESS>Sent: Friday, February 22, 2019 7:41:48 AM
To: dusty-nv/jetson-inference
Cc: Subscribed
Subject: [dusty-nv/jetson-inference] TX1 tensorNet.cpp.o Error 1 (#293)
Hello,
I have tried to went through the tutorial. I have tried to build a source on Jetson. My jetson runs Ubuntu 16.04. I have followed the instructions; however when I run "make" on terminal, I have received following error:
Scanning dependencies of target jetson-inference [ 53%] Building CXX object CMakeFiles/jetson-inference.dir/tensorNet.cpp.o In file included from /home/nvidia/jetson-inference/tensorNet.cpp:23:0: /home/nvidia/jetson-inference/tensorNet.h:135:18: error: ‘nvinfer1::IInt8Calibrator’ has not been declared nvinfer1::IInt8Calibrator* calibrator=NULL, cudaStream_t stream=NULL ); ^ /home/nvidia/jetson-inference/tensorNet.h:150:18: error: ‘nvinfer1::IInt8Calibrator’ has not been declared nvinfer1::IInt8Calibrator* calibrator=NULL, cudaStream_t stream=NULL ); ^ /home/nvidia/jetson-inference/tensorNet.h:247:19: error: ‘nvinfer1::IInt8Calibrator’ has not been declared nvinfer1::IInt8Calibrator* calibrator, std::ostream& modelStream); ^ /home/nvidia/jetson-inference/tensorNet.cpp: In function ‘nvinfer1::DataType precisionTypeToTRT(precisionType)’: /home/nvidia/jetson-inference/tensorNet.cpp:75:26: error: ‘kINT8’ is not a member of ‘nvinfer1::DataType’ case TYPE_INT8: return nvinfer1::DataType::kINT8; ^ /home/nvidia/jetson-inference/tensorNet.cpp: At global scope: /home/nvidia/jetson-inference/tensorNet.cpp:297:20: error: ‘nvinfer1::IInt8Calibrator’ has not been declared nvinfer1::IInt8Calibrator* calibrator, ^ /home/nvidia/jetson-inference/tensorNet.cpp: In member function ‘bool tensorNet::ProfileModel(const string&, const string&, const std::vector<std::__cxx11::basic_string >&, unsigned int, precisionType, deviceType, bool, int*, std::ostream&)’: /home/nvidia/jetson-inference/tensorNet.cpp:421:86: error: ‘class nvinfer1::IBuilder’ has no member named ‘getInt8Mode’ device %s, building INT8: %s\n", deviceTypeToStr(device), builder->getInt8Mode ^ /home/nvidia/jetson-inference/tensorNet.cpp: At global scope: /home/nvidia/jetson-inference/tensorNet.cpp:463:20: error: ‘nvinfer1::IInt8Calibrator’ has not been declared nvinfer1::IInt8Calibrator* calibrator, cudaStream_t stream ) ^ /home/nvidia/jetson-inference/tensorNet.cpp:477:20: error: ‘nvinfer1::IInt8Calibrator’ has not been declared nvinfer1::IInt8Calibrator* calibrator, cudaStream_t stream ) ^ /home/nvidia/jetson-inference/tensorNet.cpp: In member function ‘bool tensorNet::LoadNetwork(const char*, const char*, const char*, const char*, const std::vector<std::__cxx11::basic_string >&, uint32_t, precisionType, deviceType, bool, int*, cudaStream_t)’: /home/nvidia/jetson-inference/tensorNet.cpp:482:86: error: ‘NV_TENSORRT_PATCH’ was not declared in this scope TensorRT version %u.%u.%u\n", NV_TENSORRT_MAJOR, NV_TENSORRT_MINOR, NV_TENSORRT ^ CMakeFiles/jetson-inference.dir/build.make:76: recipe for target 'CMakeFiles/jetson-inference.dir/tensorNet.cpp.o' failed make[2]: *** [CMakeFiles/jetson-inference.dir/tensorNet.cpp.o] Error 1 CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/jetson-inference.dir/all' failed make[1]: *** [CMakeFiles/jetson-inference.dir/all] Error 2 Makefile:127: recipe for target 'all' failed make: *** [all] Error 2
Could you please help me to solve this problem?
Do I have to build a host before building the source on Jetson?
Looking forward to hearing from you
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://github.com/dusty-nv/jetson-inference/issues/293, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AOpDK6rBfM2axuZ1rJl_gACiGeVEiib_ks5vP-WLgaJpZM4bJhbH.

This email message is for the sole use of the intended recipient(s) and may contain
confidential information.  Any unauthorized review, use, disclosure or distribution
is prohibited.  If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.

Hello dusty. I have the same problem. Please help me, I need this for my thesis. Thank you

Hi @Joaquin1910 , @ismailbaayer , can you please pull from master branch and try compiling again?
I applied some fixes in commit 835447.

Hi @dusty-nv and @ismailbaayer .it was fixed, when installing jetpack, tensorrt was not installed, so, install it and it worked.
Mr @dusty-nv . I am following the github jetson-inference, and I would like to know how I can adapt the code so that the acquisition is through an IP camera, that is to say that the image comes via Ethernet, if you have an example I would thank you. On the other hand, I would like to know how I can perform a profiling on the performance of the inference to see metrics between the CPU and the GPU. It's for a colombia school project. I thank you for your help. Thank you

Hi, see this fork for IP camera:   https://github.com/Abaco-Systems/jetson-inference-gv

From: Joaquin1910<EMAIL_ADDRESS>Sent: Tuesday, March 12, 2019 8:31:41 PM
To: dusty-nv/jetson-inference
Cc: Dustin Franklin; Mention
Subject: Re: [dusty-nv/jetson-inference] TX1 tensorNet.cpp.o Error 1 (#293)
Hi @dusty-nvhttps://github.com/dusty-nv and @ismailbaayerhttps://github.com/ismailbaayer .it was fixed, when installing jetpack, tensorrt was not installed, so, install it and it worked.
Mr @dusty-nvhttps://github.com/dusty-nv . I am following the github jetson-inference, and I would like to know how I can adapt the code so that the acquisition is through an IP camera, that is to say that the image comes via Ethernet, if you have an example I would thank you. On the other hand, I would like to know how I can perform a profiling on the performance of the inference to see metrics between the CPU and the GPU. It's for a colombia school project. I thank you for your help. Thank you
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/dusty-nv/jetson-inference/issues/293#issuecomment-472233368, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AOpDKyWTJooFi75uFIADr65l4_AWAOFBks5vWEbtgaJpZM4bJhbH.

This email message is for the sole use of the intended recipient(s) and may contain
confidential information.  Any unauthorized review, use, disclosure or distribution
is prohibited.  If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.

Hi thanks it works now :)

Hi @dusty-nv
this only applies only to abacus IP cameras or also to another type of camera


Implement parsers for sprite headers

Finished chumsky versions of set, vars, broadcasts, lists, costumes & sounds parsers.


Error in models/guild.py

   File "/home/okame/bots/flex/cogs/http.py", line 93, in dboard
     for guild in self.discord.fetch_guilds():
   File "/home/okame/.local/lib/python3.8/site-packages/flask_discord/client.py", line 107, in fetch_guilds
     return [models.Guild(payload) for payload in guilds_payload]
   File "/home/okame/.local/lib/python3.8/site-packages/flask_discord/client.py", line 107, in <listcomp>
     return [models.Guild(payload) for payload in guilds_payload]
   File "/home/okame/.local/lib/python3.8/site-packages/flask_discord/models/guild.py", line 26, in __init__
     self.id = int(self._payload["id"])
TypeError: string indices must be integers

I tried myself and cannot reproduce the same error. Can you make sure you're using the latest version 0.1.10?
Also, can you check what is returned in payload object in your case. Seems like it isn't returning the list of guilds.

Seems like it isn't returning the list of guilds.
It does return full list of my servers even with their ids

...
{'id': '703204504828641340', 'name': 'Дистанционка', 'icon': None, 'owner': True, 'permissions':<PHONE_NUMBER>, 'features': []}
global
[2020-04-28 07:19:03,721] ERROR in app: Exception on /dashboard/ [GET]
Traceback (most recent call last):
  File "/home/okame/.local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app

It appears that doing this:
for guild in self.discord.fetch_guilds():
                    print(guild)

Gives only guild names, not dict

.fetch_guilds() returns models.Guild object. Using print(guild) will call the __str__ method of this object which is name of this guild. To access any of the attributes of guilds, you can use guild.name, guild.id, guild.icon_url, ...
However, if you still want the actual payload returned from discord, you can use guild.to_json().

for guilb in self.client.guilds:
                for guild in self.discord.fetch_guilds():
                    if guilb.id == guild.id:
                       print("ok")

Tried this, still gives this error

May I know what your self.client.guilds property returns? Also the error your'e getting.

self.client.guilds returns everything that it should, I just checked. Error I'm getting is:
[2020-04-28 11:47:42,663] ERROR in app: Exception on /dashboard/ [GET]
Traceback (most recent call last):
  File "/home/okame/.local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/okame/.local/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/okame/.local/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/okame/.local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/home/okame/.local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/okame/.local/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/okame/bots/flex/cogs/http.py", line 93, in dboard
    for guild in self.discord.fetch_guilds():
  File "/home/okame/.local/lib/python3.8/site-packages/flask_discord/client.py", line 107, in fetch_guilds
    return [models.Guild(payload) for payload in guilds_payload]
  File "/home/okame/.local/lib/python3.8/site-packages/flask_discord/client.py", line 107, in <listcomp>
    return [models.Guild(payload) for payload in guilds_payload]
  File "/home/okame/.local/lib/python3.8/site-packages/flask_discord/models/guild.py", line 26, in __init__
    self.id = int(self._payload["id"])
TypeError: string indices must be integers
<IP_ADDRESS> - - [28/Apr/2020 11:47:42] "GET /dashboard/ HTTP/1.0" 500 -

Can replicate this issue locally.
[2020-04-28 13:49:43,842] ERROR in app: Exception on /dashboard/ [GET]
Traceback (most recent call last):
  File "/home/okame/.local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/okame/.local/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/okame/.local/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/okame/.local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/home/okame/.local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/okame/.local/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/okame/TeamFlex/Flex/cogs/http.py", line 93, in dboard
    for guild in self.discord.fetch_guilds():
  File "/home/okame/.local/lib/python3.8/site-packages/flask_discord/client.py", line 107, in fetch_guilds
    return [models.Guild(payload) for payload in guilds_payload]
  File "/home/okame/.local/lib/python3.8/site-packages/flask_discord/client.py", line 107, in <listcomp>
    return [models.Guild(payload) for payload in guilds_payload]
  File "/home/okame/.local/lib/python3.8/site-packages/flask_discord/models/guild.py", line 26, in __init__
    self.id = int(self._payload["id"])
TypeError: string indices must be integers
<IP_ADDRESS> - - [28/Apr/2020 13:49:43] "GET /dashboard/ HTTP/1.1" 500 -

UPDATE
After updating the dashboard page code, the issue is non-replicable in our current workflow.

Okay, thanks for help @thec0sm0s! Closing that issue

I am curious on what caused that to be raised. As if payload is just a string.

The same thing happened to me before, but a reload fixed it. I think some of the code @KittyLe wrote caused flask-discord to chew up and break. He was comparing data from d.py to data from flask-discord after all.


scExamples

Hi @osumray,
Fantastic work; I would like to know if there are plans to include the code for the results shown in your paper. Or, a code showing how the single-cell data can be handled by your approach.
IMHO, this could help you to have more people trying out your method.
Best,

Hi @jsnagai,
Thanks! Yes we should be able to give some example code, is there one of the three methods (eigenscores, MLS, PRQ) that you are particularly interested in? There is already a notebook for the eigenscores, available here.

Hi @osumray
Thank you for the quick reply.
I'd love to have the code related to the PRQ model. It would be nice to test it in my single cell dataset.
Best


Suppress Standby Failover Warning Messages in DN logs during Webhdfs

The below Logs in DN only arise during Webhdfs calls to DN. WebHDFS call is a state less engine where it does not maintain / cache in NN information. Every time a request is made, Webhdfs Client on DN looks up NN information. These are benign warning messages. This Pull request is to change the log level to "ERROR".
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby multiple times in datanode log

Saumil, will other consumers of this log4j config potentially see negative impacts? (Are there any less than ERROR level org.apache.hadoop.ipc.Client messages of interest?)

There is another one which might be of interest but not sure, if we see this often.
LOG.warn("Address change detected. Old: " + server.toString() +
                                 " New: " + currentAddr.toString());

This means that something in your DNS infra changed .
src code

The problem is that there are lots of WARN errors we may want to see in the ipc.Client log4j queue.  Even some StandbyExceptions are interesting.  Just not specifically the combination of a WARN level and a RemoteException(StandbyException).
There exists a log4j filter in the log4j-extras project that can handle that level of specific filtering:
http://blog.trifork.com/2011/08/23/filtering-specific-exceptions-when-using-log4j/
The log4j-extras jar is included in HDP, but only in the flume and hive libs.  Is there an easy way for us to use this filter more broadly?

Closing for cleanup. Feel free to reopen if this work gets picked up later on.


fix(huff-codegen) use error selector instead of name

Fixing https://github.com/huff-language/huff-rs/issues/212

Done, I've ran the commands in Contributing but two semi colons have been added on formatting. Is that OK ?


Java extractor: change failure to warning on detection of non-java sources [phab:D497]

Created by schroederc at 2015-09-03 21:13:29:

shahms accepted this revision at 2015-09-03 21:32:12

schroederc closed via commit 589ffb54c543dbcb88e6501fd2992aa83eebc2d4 at 2015-09-03 21:33:18


Correct guard against non SSL data in ReferenceCountedOpenSslEngine

Motivation:
When non SSL data is passed into SSLEngine.unwrap(...) we need to throw an SSLException. This was not done at the moment. Even worse we threw an IllegalArgumentException as we tried to allocate a direct buffer with capacity of -1.
Modifications:

Guard against non SSL data and added an unit test.
Make code more consistent

Result:
Correct behaving SSLEngine implementation.

@nmittler you also want this for conscrypt

@Scottmitch PTAL

@Scottmitch PTAL again

Cherry-picked into 4.1 (0ca2c3016b1e8aa1fa8d789ce3eac68ffff3d912) and 4.0 (4d327463c7ab14aafdacb1c6cc3902a0271bfb28)


Configure Gradle parallelism and Kotlin compiler in sdk-sync

Motivation and Context
Each invocation of gradlew aws:sdk:assemble from sdk-sync was creating some form of unbounded parallelism that was causing issues with the release process. These changes restrict that a bit more, although something still seems to be doing things in parallel besides sdk-sync even after these changes.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

was causing issues with the release process

What issues?


Upgrade notification command does not copy properly due to box borders

Have you read the Contributing Guidelines on issues?

[X] I have read the Contributing Guidelines on issues.

Prerequisites

[X] I'm using the latest version of Docusaurus.
[X] I have read the console error message carefully (if applicable).

Description
Summary
It appears that the text does not copy properly from this docusaurus warning message. The file which generates this message is packages/docusaurus/bin/beforeCli.mjs. This could be a non-docusaurus specific problem, but it does not appear to be a shell problem (I tried with zsh and bash).

I recently got this upgrade note, and wanted to upgrade my docusaurus version. I copy and pasted the command into my terminal, but the following happened:
(base) sanderschulhoff@Sanders-MacBook-Pro Learn_Prompting % npm i @docusaurus/core@latest @docusaurus/plugin-client-redirects@latest @docusaurus/plugin-google-analytics@latest     │
   │          @docusaurus/plugin-ideal-image@latest @docusaurus/preset-classic@latest @docusaurus/theme-mermaid@latest           │
   │                                          @docusaurus/module-type-aliases@latest
npm ERR! code EINVALIDTAGNAME
npm ERR! Invalid tag name "│" of package "│": Tags may not have any characters that encodeURIComponent encodes.

npm ERR! A complete log of this run can be found in: /Users/sanderschulhoff/.npm/_logs/2023-08-17T23_20_04_692Z-debug-0.log
zsh: command not found: │
zsh: command not found: │

The | were quite weird as they didn't appear in the image I uploaded. I pasted the text elsewhere, and found that it pasted as:
npm i @docusaurus/core@latest @docusaurus/plugin-client-redirects@latest @docusaurus/plugin-google-analytics@latest     │
   │          @docusaurus/plugin-ideal-image@latest @docusaurus/preset-classic@latest @docusaurus/theme-mermaid@latest           │
   │                                          @docusaurus/module-type-aliases@latest

The terminal (a zsh shell) could not properly handle it. I asked GPT-4 to reformat it, and it gave me the following, which ran properly.
npm i @docusaurus/core@latest \
      @docusaurus/plugin-client-redirects@latest \
      @docusaurus/plugin-google-analytics@latest \
      @docusaurus/plugin-ideal-image@latest \
      @docusaurus/preset-classic@latest \
      @docusaurus/theme-mermaid@latest \
      @docusaurus/module-type-aliases@latest

Reproducible demo
No response
Steps to reproduce

Run these individually:

npm install<EMAIL_ADDRESS><EMAIL_ADDRESS>npx docusaurus --version

Copy the command output (between the backticks):

Paste the command into the terminal and attempt to run it.

Your environment

npx version: 9.7.2
node version: v20.4.0
VSCode, a bash or zsh terminal
Self-service

I'm okay to get rid of the box, if it makes the command easier to copy. However it should be immediately obvious for most people that the box borders are also copiable characters so you need to copy them one line at a time.

it should be immediately obvious for most people that the box borders are also copiable characters so you need to copy them one line at a time.

Could you explain this further? I am only copying the text between the backticks. There is only one npm i command, which should be directly copiable.

This box you see is entirely comprised of text. When you copy text across multiple lines, you always copy the entire line, so the borders, which are pipe characters, are also copied.

Ah, that makes sense. I did not think of this :)
Removing the box (or making it uncopiable?) would be good for usability. Thanks for walking me through this!

Ok to remove the box 👍 hope we can find a terminal display that's not too ugly and stand out a bit.
Accepting external contributor PRs. Don't claim: first draft PR gets assigned the issue.


voyager.service.ts#L42 url is rendering as a 404

https://issues.jboss.org/browse/AGMS-286

@finp - can you merge, I cannot.


[CSL-1497] Fix security check in rollback and disable it in LRC

Issue's description suggests to resolve this issue in another way, tbh I like your way more than in the issue's description. but gromak might diss you.


CI: health test failed with "Connection timed out"

NOTE: this occurred in a PR, but my branch only made changes to bugtool; thus the changes in the branch were using cilium-agent code that was equivalent to master.
Build link: https://jenkins.cilium.io/job/Cilium-PR-Ginkgo-Tests-Validated/7057/testReport/junit/k8s-1/12/K8sHealthTest_checks_cilium_health_status_between_nodes/
Output from CI run:
Stacktrace

/home/jenkins/workspace/Cilium-PR-Ginkgo-Tests-Validated/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:383
Expected
    <string>: Connection timed out
to be empty
/home/jenkins/workspace/Cilium-PR-Ginkgo-Tests-Validated/src/github.com/cilium/cilium/test/k8sT/Health.go:112
Standard Output

Number of 'context deadline exceeded' in logs: 0
Number of 'level=error' in logs: 0
⚠  Number of 'level=warning' in logs: 28
Number of 'Cilium API handler panicked' in logs: 0
⚠  Number of 'Goroutine took lock for more than' in logs: 14
Cilium pods: [cilium-6m7qp cilium-89mp5]
Netpols loaded:
CiliumNetworkPolicies loaded:
Endpoint Policy Enforcement:
     cilium-etcd-vlpwctn594             =>   none
     coredns-65b788cf-l5rgp             =>
     prometheus-core-5888d94764-52cnz   =>   none
     app1-9d8945dc9-tskd5               =>   ingress
     app2-85b74b9c79-28275              =>   none
     app3-66c6cdf976-bj75z              =>   none
     cilium-etcd-kjsd4bbwg5             =>   none
     cilium-etcd-tvszmxx9zp             =>   none
Cilium agent 'cilium-6m7qp': Status: Failure  Health: Ok Nodes "k8s2 k8s1" ContinerRuntime: Ok Kubernetes: Ok KVstore: Failure Controllers: Total 30 Failed 0
Cilium agent 'cilium-89mp5': Status: Failure  Health: Ok Nodes "k8s1 k8s2" ContinerRuntime: Ok Kubernetes: Ok KVstore: Failure Controllers: Total 18 Failed 0

Standard Error

STEP: checking that cilium API exposes health instances
STEP: checking that `cilium-health --probe` succeeds
STEP: checking API response for "{.nodes[0].endpoint.icmp.status}"
STEP: checking API response for "{.nodes[0].endpoint.http.status}"
STEP: checking API response for "{.nodes[0].host.primary-address.icmp.status}"
STEP: checking API response for "{.nodes[0].host.primary-address.http.status}"
STEP: checking API response for "{.nodes[1].endpoint.icmp.status}"
=== Test Finished at 2018-10-08T19:41:14Z====
===================== TEST FAILED =====================
cmd: kubectl get pods -o wide --all-namespaces
Exitcode: 0
Stdout:
 	 NAMESPACE     NAME                               READY   STATUS    RESTARTS   AGE   IP              NODE   NOMINATED NODE
	 kube-system   cilium-6m7qp                       1/1     Running   0          21s   <IP_ADDRESS>   k8s2   <none>
	 kube-system   cilium-89mp5                       1/1     Running   0          21s   <IP_ADDRESS>   k8s1   <none>
	 kube-system   coredns-65b788cf-l5rgp             1/1     Running   0          26s   <IP_ADDRESS>     k8s2   <none>
	 kube-system   etcd-k8s1                          1/1     Running   0          35m   <IP_ADDRESS>   k8s1   <none>
	 kube-system   etcd-operator-7cb79cdf99-jjm9c     1/1     Running   0          23s   <IP_ADDRESS>      k8s2   <none>
	 kube-system   kube-apiserver-k8s1                1/1     Running   0          35m   <IP_ADDRESS>   k8s1   <none>
	 kube-system   kube-controller-manager-k8s1       1/1     Running   0          35m   <IP_ADDRESS>   k8s1   <none>
	 kube-system   kube-proxy-4nbml                   1/1     Running   0          29m   <IP_ADDRESS>   k8s2   <none>
	 kube-system   kube-proxy-kx5hk                   1/1     Running   0          36m   <IP_ADDRESS>   k8s1   <none>
	 kube-system   kube-scheduler-k8s1                1/1     Running   0          35m   <IP_ADDRESS>   k8s1   <none>
	 monitoring    prometheus-core-5888d94764-52cnz   1/1     Running   0          29m   <IP_ADDRESS>      k8s2   <none>

Stderr:

cmd: kubectl exec -n kube-system cilium-6m7qp -- cilium endpoint list
Exitcode: 0
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                              IPv6                   IPv4          STATUS
	            ENFORCEMENT        ENFORCEMENT
	 42082      Disabled           Disabled          129        k8s:io.cilium.fixed-identity=kube-dns                    f00d::a0a:100:0:a462   <IP_ADDRESS>   ready
	                                                            k8s:io.cilium.k8s.policy.cluster=default
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=coredns
	                                                            k8s:io.kubernetes.pod.namespace=kube-system
	                                                            k8s:k8s-app=kube-dns
	 64054      Disabled           Disabled          4          reserved:health                                          f00d::a0a:100:0:fa36   <IP_ADDRESS>    ready
	 65416      Disabled           Disabled          128        k8s:io.cilium.fixed-identity=kv-store                    f00d::a0a:100:0:ff88   <IP_ADDRESS>    ready
	                                                            k8s:io.cilium.k8s.policy.cluster=default
	                                                            k8s:io.cilium.k8s.policy.serviceaccount=cilium-etcd-sa
	                                                            k8s:io.kubernetes.pod.namespace=kube-system
	                                                            k8s:name=etcd-operator

Stderr:

cmd: kubectl exec -n kube-system cilium-89mp5 -- cilium endpoint list
Exitcode: 0
Stdout:
 	 ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])   IPv6                 IPv4          STATUS
	            ENFORCEMENT        ENFORCEMENT
	 59624      Disabled           Disabled          4          reserved:health               f00d::a0a:0:0:e8e8   <IP_ADDRESS>   ready

Stderr:

===================== Existing AfterFailed =====================

Logs:
4de6fcf6_K8sHealthTest_checks_cilium-health_status_between_nodes.zip

Another case:
https://jenkins.cilium.io/job/Ginkgo-CI-Tests-Pipeline/4353/testReport/junit/k8s-1/12/K8sHealthTest_checks_cilium_health_status_between_nodes/

Another case:
https://jenkins.cilium.io/job/Ginkgo-CI-Tests-Pipeline/4356/testReport/junit/k8s-1/12/K8sHealthTest_checks_cilium_health_status_between_nodes/

On cilium-89mp5, short cilium-health status response in bugtool:
Probe time:   2018-10-08T19:42:03Z
Nodes:
  k8s1 (localhost):
    Host connectivity to <IP_ADDRESS>:
      ICMP:   OK, RTT=83.68µs


Configured views not set correctly after updating

I've had this issue two times already where the configured views don't show as I've previously configured. The two times it has happened has been right after updating to a new StartPage version.
This is how it looks after updating:

The view that is on the upper left, was in the panel below like this before the bug. This is after moving it to the previous place.

Notice that even after moving, there is another Game Shelve in that same panel. So it seems that for some reason, the plugin placed two different ``Game Shelves `views in the same Panel:

This is the the configured panels before the bug happened:

After the bug, the configuration I had for the Game Shelves panel below got reverted to a configuration I had before and not the latest one so some shelves went missing, filters got reverted, etc.. This happened the previous time too:

There's nothing in the logs about the StartPage extension.

Does the plugin have some sort of config backup system? Because both times it has reverted back to the same old configuration

Not really. I just stored a default layout that is loaded if there is no layout present. Otherwise users would be greeted with an empty screen when they installed the extension.

Could you provide your config.json files for StartPage? Either broken ones or the ones that broke after an update. I never had an issue like that when updating.

That's weird. Do you have any idea why it could be reverting back to that specific configuration then?
I recreated how I remember I had it before the bug happened the two times:
config.zip

Also, for game shelves, the config is not stored as part of the layout. It is stored separately in a dictionary with an Guid as the key. Maybe that leads to the effect of it reverting when the layout breaks.

I cannot replicate this behavior. I loaded your config with v2.3.1, then updated to v2.3.2. Everything looks the same. But I may have an idea what happens when your layout is broken. What it sounds like is, that multiple nodes of the layout tree have a shelve view with the same instance id. Also, only leaf nodes are supposed to have views, but apparantly in your case, also non-leaf nodes have views, leading to the stacking of views. Although I was sure that even if a non-leaf node had a view, it should not be visible. Maybe I will put more validation in place when loading and saving the layout tree, but I don't really know why this happened when updating.

Was there maybe something else causing Playnite to crash when exiting/updating which could have corrupted the config file?

No, it has not crashed. But even if that was the case, I've closed/restarted Playnite several times after initially configuring the views so the saved config file should be there, which makes me wonder why it is reverting to this specific, old views configuration. There's also the issue that when this happens, there are two Game Shelves place in the same panel. I'll see if I can find something the next time it happens.

It happened once again. Nothing special happened, I just have closed and started Playnite normally.

The config file without moving anything:
config - 1.zip
https://user-images.githubusercontent.com/1389286/166190355-60cca741-082d-41a9-89fb-c9decdd7886d.mp4

I just noticed that in the video above, it seems to think that the Clock is the active view in the panel?

I just noticed that in the video above, it seems to think that the Clock is the active view in the panel?

Items that cannot be added are disabled. In this case, there is already a clock in the panel to the right and clocks cannot have multiple instances, so it is disabled. Everything else can be added because it allows multiple instances (Game Shelves) or does not exist yet in any panel. So this is the expected behavior.

I at least know why it looks like that. For some reason the dictionary that stores the shelve config for each is empty. If a shelve with no config is requested, the first one that is always stored as the default one. So the one in the top right apparantly requests its view first, giving it the view that would usually be below it. Every request after that just gets an empty shelve.

Also I suspect that it happens whenever the settings view is canceled instead of saved.

Also there is no issue of stacked views. You simply swapped the empty shelve view with a non-empty one. It just looks like the one on the bottom is an empty panel because the buttons are too far down. You can tell that there must be a shelve view attached to that panel by the settings icon in the top right corner.

felixkmh_StartPage_Plugin_2_3_2.zip
You can try this and see if it happens again. Like I said, I think it was happening whenever the settings view was closed without saving.

Does the new update include that fix? Now that I remember I have opened the settings window and canceled so it could very well be the cause of the issue. I'll let you know if it happens again

It happened again :(
I was checking something and was starting and closing Playnite in a few seconds, other than that I can't think of anything since I didn't even open the StartPage view

I didn't even open the StartPage view

Which is probably the reason that happened again xD
To prevent unused instances to be stored, I only save those that have actually been loaded when Playnite exits. But if StartPage is never opened, nothing will have loaded and all instances are discarded. Should be an easy fix.

Yeah I can reproduce it now by Start Playnite -> Exit Playnite (Without opening Start Page) -> Start Playnite again and go to Start Page.
I'll see if it happens with the new version and report.

It seems to be fixed with 2.3.4 so I think the issue can be closed. If it happens again I'll let you know.


side-nav overlay not disappearing

Great work guys, i only just recently discovered the entire materialize framework and its wonderful.
I'm using the data-closeonclick="true" option for side-nav. It works, but the overlay doesn't disappear sometimes and at other times it disappears as it should.

This sounds like a bug with materialize.
Check to see if Materialize also has this bug, and in that case report it with them.
Angular-materialize is just simple directives that start use Materialize code, so the bug might very well be at their end.

Thanks for the response. I'm actually using the angular-materialize on a phonegap android app and it seems to work fine on the mobile device. I will test this on materialize alone as well and send a message to the authors. Thank you again

Good, reopen this issue if the problem turns our to be with angular-materialize


Flutter Workflow Error : Method not found 'BlocConsumer'

Build and Release apk / Build APK
failed 5 minutes ago in 2m 58s
Search logs
  Set up job3s
  Run actions/checkout@v12s
  Run actions/setup-java@v17s
  Run subosito/flutter-action@v11m 9s
  Run flutter pub get16s
  Run flutter build apk --target-platform android-arm,android-arm64,android-x64 --split-per-abi1m 21s
##[error]Process completed with exit code 1.
Run flutter build apk --target-platform android-arm,android-arm64,android-x64 --split-per-abi
Initializing gradle...                                              6.4s
Resolving dependencies...                                          53.2s
Running Gradle task 'assembleRelease'...

Compiler message:

lib/ui/coronaHome.dart:41:16: Error: Method not found: 'BlocConsumer'.

        child: BlocConsumer < CoronaBloc, CoronaState > (

               ^^^^^^^^^^^^

lib/ui/coronaHome.dart:41:16: Error: The method 'BlocConsumer' isn't defined for the class '_CoronaHomeState'.

 - '_CoronaHomeState' is from 'package:covid19/ui/coronaHome.dart' ('lib/ui/coronaHome.dart').

Try correcting the name to the name of an existing method, or defining a method named 'BlocConsumer'.

        child: BlocConsumer < CoronaBloc, CoronaState > (

               ^^^^^^^^^^^^
Compiler terminated unexpectedly.

Running Gradle task 'assembleRelease'...                           18.0s
Gradle task assembleRelease failed with exit code 1
##[error]Process completed with exit code 1.
 Create a Release APK0s
  Complete job

Hi @r0hnx
flutter_bloc is a third party plugin
you may want to open an issue in the dedicated github
Closing, as this isn't an issue with Flutter itself,
if you disagree please write in the comments and I will reopen it
Thank you

@iapicca I don't think the problem is flutter_bloc. The CI goes like this.
on: push
name: Build and Release apk
jobs:
  build:
    name: Build APK
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v1
    - uses: actions/setup-java@v1
      with:
        java-version: '12.x'
    - uses: subosito/flutter-action@v1
      with:
        flutter-version: '1.9.1+hotfix.6'
    - run: flutter pub get
    - run: flutter build apk --target-platform android-arm,android-arm64,android-x64 --split-per-abi
    - name: Create a Release APK
      uses: ncipollo/release-action@v1
      with:
        artifacts: "build/app/outputs/apk/debug/*.apk"
        token: ${{ secrets.TOKEN }}

This change in my main.yml file seemed to fix the error
steps:

uses: actions/checkout@v1
uses: actions/setup-java@v1
with:
java-version: '12.x'
uses: subosito/flutter-action@v1
with:
channel: 'stable' # or: 'dev' or 'beta'
run: flutter pub get
run: flutter test
run: flutter build apk

Hi @r0hnx
I'm glad that you fix the error
thank you for letting me know


The solution is to create ~/.config/buildkit/buildkitd.toml (/etc/buildkit/buildkitd.toml for root) like this

# registry configures a new Docker register used for cache import or output.
[registry."docker.io"]
  mirrors = ["yourmirror.local:5000"]
  http = true
  insecure = true
  ca=["/etc/config/myca.pem"]
  [[registry."docker.io".keypair]]
    key="/etc/config/key.pem"
    cert="/etc/config/cert.pem"

# optionally mirror configuration can be done by defining it as a registry.
[registry."yourmirror.local:5000"]
  http = true

https://github.com/moby/buildkit/blob/master/docs/buildkitd.toml.md

I am using Rancher Desktop (0.7.1) on Windows 10 Enterprise (under a big company).
My command (test example) is:
nerdctl build -t nginx:nerd .

with this Docker file
# Dockerfile
FROM nginx:alpine
RUN echo 'Hello Nerdctl from containerd'

Despite, I created the ~/.config/buildkit/buildkitd.toml file with the proposed content, I still get the same error:
[+] Building 0.7s (3/3) FINISHED
 => [internal] load build definition from Dockerfile                                                                                0.1s
 => => transferring dockerfile: 31B                                                                                0.0s
 => [internal] load .dockerignore                                                                                0.1s
 => => transferring context: 2B                                                                                0.0s
 => ERROR [internal] load metadata for docker.io/library/nginx:alpine                                                                                0.6s
------
 > [internal] load metadata for docker.io/library/nginx:alpine:
------
error: failed to solve: failed to solve with frontend dockerfile.v0: failed to create LLB definition: failed to do request: Head https://registry-1.docker.io/v2/library/nginx/manifests/alpine: x509: certificate signed by unknown authority
FATA[0000] unrecognized image format

I want to notice also that when I pull or run the image, it works fine.

I have the same issue.

Signing out from my company VPN fixed the issue. VPN was messing up the cert.

This issue is solved in the version v1.0.0

--insecure-registry does apply to  nerdctl pull but not apply to nerdctl build operations
try to add buildkitd.toml  for buildkitd as following
/etc/buildkit/buildkitd.toml
[registry."yourmirror.local:5000"]
  http = true
  insecure = true

and don't forget to run systemctl restart buildkit


strip trailing slash from package_dir before counting slashes

Hi! Apologies, wasn't sure of your process so figured i'd just throw this over the wall and see what response, if any it provoked.
I was getting Can't get a consistent path to setup script from installation directory when trying to pip install -e a package the other day, and eventually tracked it down to a trailing slash in package_dir. The docs i've seen don't use trailing slashes in the examples, but also don't mention that they aren't allowed.
It looks like the edited code is using slashes as a heuristic for how far up the tree to walk, and i guessed that it might be getting confused by the trailing slash
unfortunately i'm new to this codebase, so was hoping for some guidance from yourself or some other core dev as to other (potentially bad) implications of this change, as well as how to best test it
Thanks!

I figured it was something like that. Thanks for providing the background. It sure would be nice to have a test for this - I'm not sure where it would go, if it should be a test for the develop command, or for the package_dir values. The fix also should consider non-Unix paths. Perhaps the whole routine should be written to use os.path.relpath or similar in order to avoid re-inventing the wheel.


Issue indexing by xarray's own time values + offset

I'm struggling to work out how to index by a xarray time value + an offset (either created using np.timedelta64 or datetime.timedelta). I read through https://github.com/pydata/xarray/issues/1240 and https://github.com/pydata/xarray/issues/1240 because they appear related, but I'm not sure how to correctly achieve this.
MCVE Code Sample

import xarray as xr
import numpy as np
import datetime as dt

now = dt.datetime.now()
dt_array = xr.DataArray(
   range(10), dims=('time', ),
   coords=dict(time=[now + dt.timedelta(seconds=i) for i in range(10)])
)

# this works
dt_array.loc[dt_array.time.min():dt_array.time.max()].count() == 10

# this fails, only the first value is returned (adding
# the time delta appears to have no effect)
dt_array.loc[dt_array.time.min():dt_array.time.min() + np.timedelta64(seconds=4)].count() == 4

# this fails, an exception is raised when trying to add
# a datetime.timedelta to the xarray value
dt_array.loc[dt_array.time.min():dt_array.time.max() + dt.timedelta(seconds=4)].count() == 4

# also fails, I got the impression from issue #1240
# that `.loc[...]` should work for indexing too, but just to double-check
dt_array.sel(time=slice(dt_array.time.min(), dt_array.time.min() + np.timedelta64(seconds=4))).count() == 4

# fails, showing that adding a time increment has no effect
dt_array.time.min() + np.timedelta64(seconds=10) != dt_array.time.min()

Expected Output
Where I am indexing by the minimum time plus a np.timedelta64 offset of 4 seconds I would expect a DataArray of length 4 to be return. It would be nice if it was possible to add an increment with a native datetime.timedelta object.
Problem Description
I can't work out how to correctly add an increment to a time value in an xarray DataArray. It would be nice if one of the above approaches worked. Or maybe if I'm missing something obvious I could add an example to the documentation on datetime-indexing?
Versions
Output of xr.show_versions()
INSTALLED VERSIONS
------------------
commit: None
python: 3.6.7 |Anaconda, Inc.| (default, Oct 23 2018, 19:16:44)
[GCC 7.3.0]
python-bits: 64
OS: Linux
OS-release: 3.10.0-957.27.2.el7.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_GB.UTF-8
LOCALE: en_GB.UTF-8
libhdf5: 1.10.1
libnetcdf: 4.5.0
xarray: 0.15.1
pandas: 0.25.3
numpy: 1.15.4
scipy: 1.1.0
netCDF4: 1.4.0
pydap: None
h5netcdf: 0.7.4
h5py: 2.10.0
Nio: None
zarr: None
cftime: <IP_ADDRESS>
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: 2.2.0
bottleneck: None
dask: 0.20.0
distributed: 1.24.0
matplotlib: 2.2.3
cartopy: 0.16.0
seaborn: 0.9.0
numbagg: None
setuptools: 46.1.3
pip: 10.0.1
conda: None
pytest: 5.3.2
IPython: 7.1.1
sphinx: None

take a look at the result of np.timedelta64(seconds=4):
In [2]: np.timedelta64(seconds=4)
Out[2]: numpy.timedelta64(0)

In [3]: np.timedelta64(dt.timedelta(seconds=4))
Out[3]: numpy.timedelta64(4000000,'us')

In [4]: np.timedelta64(4, 's').astype("timedelta64[us]")
Out[4]: numpy.timedelta64(4000000,'us')

so you need to also specify the desired unit or cast a dt.timedelta object. I'm not sure why timedelta64 accepts the seconds kwarg, though.

Ah! It's an issue with how I am using np.timedelta64 issue then. I (stupidly) assumed that np.timedelta64 has the same call signature as datetime.timedelta. It appears that np.timedelta64 silently ignores any kwargs 😕
Anyway, I won't be making that mistake again. Thank you @keewis !


Improve build process to build for CommonJS, ES6 and legacy

I need some help setting up the build process to output files for CommonJS, ES6 imports and legacy browser (regular <script> tags)

Done in version 2.x with the transition to Vite.


v-slot error when noPropertyAccessFromIndexSignature is enabled with v1.0.14+

Similar to #561, after updating from 1.0.13 to 1.0.14 (same with .15 and .16), if using <template #actions> or <template v-slot:actions> then vue-tsc --noEmit fails with error TS4111: Property 'actions' comes from an index signature, so it must be accessed with ['actions'].

This has started cropping up again.


superscript citations (eg Nature style) appearing after punctuation instead of before

Hi, I'm using the standard Nature csl (https://www.zotero.org/styles/nature) but citations are rendering abnormally, appearing after preceding punctuation instead of in the location specified in R Markdown.
For example, the text experiences [@Paper8; @Paper9; @Paper10]. renders as in the following image:

The rendered text "8-10" should appear before the period, as is Nature house style, but appears after. This doesn't appear to be affected by adding lang: en-UK to the yaml header.

Put
notes-after-punctuation: false

in your metadata.

ah brilliant! thanks :D


Replace placeholders in licenses

Replace #22117. Implement it in a more maintainable way.
Some licenses have placeholders e.g. the BSD licenses start with this line:
Copyright (c) <year> <owner>.

This PR replaces the placeholders with the correct value when initialize a new repo.
FAQ

Why not use a regex?
It will be a pretty complicated regex which could be hard to maintain.

There're still missing placeholders.
There are over 500 licenses, it's impossible for anyone to inspect all of them alone. Please help to add them if you find any, and it is also OK to leave them for the future.

Some tests may be nice, e.g. extract to a function that does the string replacement and test it separately.


Measure basic information about the consul namer.

This should help diagnose consul behavior.
Furthermore, fix a malformed consul client naming (was doing Path.toString
instead of Path.show).
Finally, LinkerConfig should scope namer stats explicitly rather than rely on
the namer to do the right scoping.

:star: 📐


not print character $

Hello, how are you? First of all congratulations for the excellent work.
I would like your help.
I am trying to print $ character only that it prints ¥.
I tried to change codepage and encoding but to no avail.
Do you know what it can be?
In test test store applications for printer normally prints the $ character.
Thanks in advance.

Duplicate of #74
Hi, same issue , i want to print pound sign but  i can not , it prints question sign insted , could you solved  this ???

Hello, how are you? First of all congratulations for the excellent work.
I would like your help.
I am trying to print $ character only that it prints ¥.
I tried to change codepage and encoding but to no avail.
Do you know what it can be?
In test test store applications for printer normally prints the $ character.
Thanks in advance.

Hi, same issue , i want to print pound sign but i can not , it prints question sign insted , could you solved this ???

Hello,
For the euro sign the configuration is this:
{
encoding:' Cp858',
codepage: 13
}

for the French language


Gateway UI - DAO screen, proposal listing

Suggest to change to the usual multipage view - there could be 100 proposals, and so will be very tedious to have to scroll up and down to find the right one.

added the pagination on proposal with 3 proposal per page so we can have better view.
changes are available into the PR
https://github.com/omgnetwork/optimism/pull/354


[WIP] Add Basic Support For Parallel Tempering

Description
This PR aims to add basic support for performing parallel tempering simulations, making use of the fantastic utilities in openmmtools. This will in principle enable more comprehensive sampling of molecules with configurations separated by large energy barriers, such as when estimating the vacuum energies of carboxylic acids and esters.
Status

[ ] Ready to go

GHMC is likely a poor choice for condensed phase systems since it will lead to very long correlation times. I'd highly suggest using BAOAB for condensed phase systems.
Ideally, you would use @maxentile's thresholds tool to identify the max stable timestep at high temperature, and use that throughout.

GHMC is likely a poor choice for condensed phase systems since it will lead to very long correlation times. I'd highly suggest using BAOAB for condensed phase systems.

100% agree for the condensed phase, and BAOAB will absolutely remain the default there!
I've seeing some strange behaviour in the gas phase however whereby the distributions sampled at each replica temperature using parallel tempering didn't match the distribution produced by running just a regular simulation, so I've been trying to look into the cause of this by varying things such as the integrator / timestep. I'll update more on this if I manage to get some illuminating results.

Ideally, you would use @maxentile's thresholds tool to identify the max stable timestep at high temperature, and use that throughout.

Agree also that this would be a great addition, unfortunately I just haven't had the time to integrate this yet.

Please do file an issue in openmmtools if you notice anything odd! Ideally you could I close XML files of the system and a few lines of code to reproduce the parallel tempering simulation.

Closing this as stale as the PR is now very far behind master, and the implementation flawed.


Fix for issue 1236

This PR creates a regression as explained here: https://github.com/swagger-api/swagger-parser/issues/1329


Editing title causes 404

Reproduction steps:

Create a report
enter a title
Save the report
Publish report
Edit published report
change title
change focus out of title field (by clicking, tab, etc)
notice error in console, and notice that save button isn't active until you change focus, even though unsaved changes warning appears
click Save
get error

I go this error when attempting an a draft report, even though the change goes through just fine:

When I edit a published document, I get the same error as above (highlighted below) except now when I click save I get a 404 page. I noticed the new title is in the URL, maybe that has something to do with it?

I can't reproduce this following the steps above. Can we discuss this during our office hours.

After attempting to recreate this again, it looks like this only happens if the alias field was empty when the document was first created/published. There are two fixes we would like to put in place to avoid this:

Remove default text from Title field of "Untitled Document"
Make the alias field a required field

Okay, great! I think that edge case is way further out on the edge than how it performs today. I say we go with it and call it good unless we see a bunch of user issues come up.


Dev addition of individual triggerbit information

Addition of trigger bit information for DZ filter efficiency in 2016postVFPUL and 2016preVFPUL for tag and probe measurement.

Done a split of muon correctionlib into muon and embeddingselection.


chore(deps): update module github.com/cyberark/conjur-api-go to v0.12.9

This PR contains the following updates:

Package
Type
Update
Change

github.com/cyberark/conjur-api-go
require
patch
v0.12.4 -> v0.12.9

Release Notes

cyberark/conjur-api-go (github.com/cyberark/conjur-api-go)
v0.12.9
Compare Source
[0.12.9] - 2024-12-13
Added

HTTP timeout is now configurable. cyberark/conjur-api-go#150

v0.12.7
Compare Source
[0.12.7] - 2024-11-18
Fixed

Updated shellescape package to fix Go error on installation
cyberark/conjur-api-go#182

v0.12.6
Compare Source
[0.12.6] - 2024-10-30
Security

Remove debug logging of "Authorization" header when processing requests (CNJR-7025)

v0.12.5
Compare Source
[0.12.5] - 2024-10-23
Added

Added RoleMembershipsAll to fetch inherited memberships for a role (CNJR-5213)
Added support for creating a client using an OIDC token (CNJR-6081)
Added support for rotating the API key of an authenticated host (CNJR-5699)
Added support for checking authenticator status (CNJR-6371)
Return error when trying to call unsupported methods in Conjur Cloud (CNJR-5700)

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

ℹ Artifact update notice
File name: go.mod
In order to perform the update(s) described in the table above, Renovate ran the go get command, which resulted in the following additional change(s):

2 additional dependencies were updated
The go directive was updated for compatibility reasons

Details:

Package
Change

go
1.22.6 -> 1.23.4

github.com/zalando/go-keyring
v0.2.5 -> v0.2.6

golang.org/x/sys
v0.25.0 -> v0.26.0


service-center有没有监控的UI页面？ 查看微服务列表等。

service-center有没有监控的UI页面？ 查看微服务列表等。

Service-center Have no  Web UI，shouldn't rely on HUAWEI Cloud.
Service-Center是否一定要依赖于华为云？ 作为开源软件，应该降低微服务管理的门槛。

@zhuhoudong
Thanks for conern.
Service-center does not depend on Huawei cloud, it is supported several deployment modes, https: //github.com/ServiceComb/ServiceComb-Company-WorkShop.git example provides several deployment templates, you can refer to it to easily get it.
To enable ServiceComb with a monitoring UI is a good Idea, the current service-center community version has not yet provide monitored UI , if you are interested, we can participate in the contribution together to make servicecomb better.

@zhuhoudong , we have released a new version of Service-Center with frontend where you can see all the basic details of microservices and their schemas, you can download the latest release from here, There is also a complete proposal for Service-Center UI here, if you are interested then feel free to contribute to the frontend.


Plugin on 14.1.5 seems to have issues with @Value annotation

Use of constructor of below class compiles from cmd line, not in IDE.
@Value
public class Greeting {
    private String id;
    private String name;
    private long count;
}

Error:(25, 28) java: constructor Greeting in class demo.Greeting cannot be applied to given types;
required: no arguments
found: java.lang.String,java.lang.String,long
reason: actual and formal argument lists differ in length

Looks like for some reason the annotation processing had been switched off again in the IntelliJ preferences - would be really useful if the plugin could report/recommend this if they are not switched on!


More Calendar tweaks; New log types

What's New?

Add a 5 minute grace period to calendar scheduling so stuff doesn't disappear so fast.
Add an ephemeral debug log type for "noisy" messages.

Coverage remained the same at 31.202% when pulling b358a9ef157a9f84c72bcaa167a6c1d85401d90d on RickCarlino:master into 96df189677b29e56cbe59a72adf5d25f7b1f550a on FarmBot:master.

Coverage remained the same at 31.202% when pulling b358a9ef157a9f84c72bcaa167a6c1d85401d90d on RickCarlino:master into 96df189677b29e56cbe59a72adf5d25f7b1f550a on FarmBot:master.


Extend DMS to cover Kafka Premium extensions

recheck

recheck

recheck

recheck

recheck

recheck

recheck

merge

merge

merge

/merge


lnconfig: Support utilizing Environment Variables in lnd.conf for rpcuser and rpcpass fields.

Change Description

Added supplyEnvValue function in config.go that returns The value of the specified environment variable, or the default value if provided, or the original input string if no matching variable is found or set.
Called supplyEnvValue function in parseRPCParams function in config.go file where rpcuser and rpcpass fields are being parsed.

Additional Context
supplyEnvValue extracts the value of an environment variable from a string. It supports the following formats:

$ENVIRONMENT_VARIABLE
${ENVIRONMENT_VARIABLE}
${ENVIRONMENT_VARIABLE:-DEFAULT_VALUE}

It follows the following naming conventions for the Standard environment variable:

ENVIRONMENT_VARIABLE contains letters, digits, and underscores, and does not start with a digit.
DEFAULT_VALUE follows the rule that it can contain any characters except whitespace.

It takes the following parameters:

value: The input string containing references to environment variables.

Closes #8295
Steps to Test
Try the following tests in lnd.conf on the bitcoind for example:

$ENVIRONMENT_VARIABLE -> It extracts the value of ENVIRONMENT_VARIABLE if it is not set it raises error.
$ENVIRONMENT_VARIABLE} or ${ENVIRONMENT_VARIABLE -> As this is not a valid expression for accessing it deals with them as a typical string.
${ENVIRONMENT_VARIABLE} -> It extracts the value of ENVIRONMENT_VARIABLE if it is not set it raises error.
${ENVIRONMENT_VARIABLE:-DEFAULT_VALUE} -> It extracts the value of ENVIRONMENT_VARIABLE if it is not set it gets the value of DEFAULT_VALUE.
${ENVIRONMENT_VARIABLE:-} -> As this is not a valid expression for accessing ${ENVIRONMENT_VARIABLE:-} it deals with it as a typical string.
${:-DEFAULT_VALUE} -> If the ENVIRONMENT_VARIABLE is empty here it get the value of DEFAULT_VALUE.

You could also check the logic workflow of the used regular expression in the diagram tool:

Pull Request Checklist
Testing

[ ] Your PR passes all CI checks.
[ ] Tests covering the positive and negative (error paths) are included.
[ ] Bug fixes contain tests triggering the bug to prevent regressions.

Code Style and Documentation

[ ] The change obeys the Code Documentation and Commenting guidelines, and lines wrap at 80.
[ ] Commits follow the Ideal Git Commit Structure.
[ ] Any new logging statements use an appropriate subsystem and logging level.
[ ] Any new lncli commands have appropriate tags in the comments for the rpc in the proto file.
[x]  There is a change description in the release notes, or [skip ci] in the commit message for small changes.

📝 Please see our  Contribution Guidelines for further guidance.

Thanks for the great feedback @guggero @ProofOfKeags I've addressed your suggestions!

Before I'm going to review in detail again, just one comment: It feels weird to me that we would want to support different formats and even fall back values within the parsing code in lnd itself.
It makes the regular expression quite complicated and hard to maintain. What's the use case you see for allowing to specify a fall back value? This would need to be documented really well, otherwise I can already see all the support issues being created because users unfamiliar with the bash like syntax remove the - character after the colon because they think it's part of the fall back value instead of the actual bash syntax.

@mohamedawnallah, remember to re-request review from reviewers when ready

@mohamedawnallah, remember to re-request review from reviewers when ready

@mohamedawnallah, remember to re-request review from reviewers when ready

@mohamedawnallah, remember to re-request review from reviewers when ready

@mohamedawnallah, remember to re-request review from reviewers when ready

@mohamedawnallah, remember to re-request review from reviewers when ready

@mohamedawnallah, remember to re-request review from reviewers when ready

I'm going to unsubscribe from this since I have approved. Please ping me if my input is needed again.

@guggero I'd like to discuss this issue again anytime you're available. I agree with your thoughts That the changes that this PR proposes add much complexity compared to the value it added. I made the fallback value since the main issue mentioned that in the demo rpcuser=${BTC_RPCUSER:-btc} also that bitcoind does support this fall-back value and To be honest I don't have any use case in mind in the meantime for allowing to specify a fall-back value. That said I'm very open to suggestions e.g. having one format something like ${ENVIRONMENT_VARIABLE} would solve the issue?
What's the use case you see for allowing to specify a fallback value?

cc: @ProofOfKeags

@guggero I'd like to discuss this issue again anytime you're available. I agree with your thoughts That the changes that this PR proposes add much complexity compared to the value it added. I made the fallback value since the main issue #8295 mentioned that in the demo rpcuser=${BTC_RPCUSER:-btc} also that bitcoind does support this fall-back value and To be honest I don't have any use case in mind in the meantime for allowing to specify a fall-back value. That said I'm very open to suggestions e.g. having one format something like ${ENVIRONMENT_VARIABLE} would solve the issue?

Okay, I guess it could be useful to have a fall back value. Although I think that can also lead to hard-to-find errors if you're not expecting that.
So I guess my main ask here is to simplify the regular expression, so it's easier to read and understand.
My suggestion would be to have an expression per supported format, then have a switch statement that checks which one matches. Then the extraction will be easier to understand as well.

So I guess my main ask here is to simplify the regular expression, so it's easier to read and understand. My suggestion would be to have an expression per supported format, then have a switch statement that checks which one matches. Then the extraction will be easier to understand as well.

This looks great! I'm gonna submit a patch and will let you know. Thanks for sharing!

Thanks a lot, @guggero for your suggestion. It is now much simpler. I've addressed all of your feedback!