New Twitter Icon (X)

Icon name
twitter-x
Use cases
it's is the new brand logo of Twitter (X)
Design ideas
Should looks like the official brand logo
Checklist

[X] I have searched if someone has submitted a similar issue before and there weren't any. (Please make sure to also search closed issues, as this issue might already have been resolved.)
[X] I have searched existing icons to make sure it does not already exist and I didn't find any.
[X] I am not requesting a brand logo and the art is not protected by copyright.
[X] I am not requesting an icon that includes religious, political imagery or hate symbols.
[X] I have provided appropriate use cases for the icon(s) requested.

I am not requesting a brand logo and the art is not protected by copyright.

Actually, it is.


Fix NPE in Maven2Gradle conversion

These checks were missed when converting from Groovy to Java in #14164.
Fixes #15827

@bot-gradle test QF

@bot-gradle test QF

OK, I've already triggered QuickFeedback build for you.

OK, I've already triggered QuickFeedback build for you.

@bot-gradle test QF

@bot-gradle test QF

OK, I've already cancelled the old build and triggered a new QuickFeedback build for you.

OK, I've already cancelled the old build and triggered a new QuickFeedback build for you.

@bot-gradle test QF

@bot-gradle test QF

OK, I've already triggered QuickFeedback build for you.

OK, I've already triggered QuickFeedback build for you.


paper

Hello, is there a paper I can refer to?

Person Re-Identification in Human Following Scenarios: An Experience with RGB-D Cameras


Support for layout definitions in ZMK/Zephyr modules?

As I'm guessing you're aware, ZMK is shying away from maintaining boards and shields in its main repository, and to that end has provided a canonical way to externalize these definitions into modules. This allows experienced stakeholders to maintain "source of truth" modules which end users can simply source—instead of end users having to fork, duplicating code, and assuming a maintenance burden that is likely not understood until the build breaks.
Much like board and shield definitions, it would be ideal if Keymap Editor could refer back to one of these canonical modules or some other repository with a dedicated maintainer rather than end users having to wrangle their own copies. I understand that you maintain a repo with layout definitions; while I laud that effort, it also strikes me as you assuming the exact type of data management burden that ZMK is taking great pains to decentralize.
It's probably not reasonable to parse the west manifest, but...

Maybe some way for the user to say "my layout definition is over at this URL" when it comes up as "not found".
Maybe these dedicated layout definition repos could be pulled in as submodules (...this is probably just as complex as manifest-wrangling)

I understand that this would be in no way a small undertaking, but wanted to at least gauge the temperature of the room post-Zephyr module addition.
Your time and consideration are very much appreciated.

Hi!
Thanks so much for starting this discussion, I appreciate you putting so much consideration into the potential addition. I'm aware of ZMK's support for Zephyr modules but I haven't looked into it in much detail yet. My high level understanding is that in addition to the boards and/or shields a module provides they would also provide the layout JSON data?
I definitely see the benefit, but one thing that immediately jumps to my mind is the additional overhead of blindly searching additional repositories that may or may not include this metadata. That sounds minor, but the trick is that if I'm not looking for something specific (e.g., for corne I already know to look for config/boards/shields/corne/... and not config/boards/<arch?>/<keyboard name>/...) so it has to be a more generic. Even with GitHub's GraphQL API I can avoid recursively traversing the repo but it still takes time, and I don't know that actually cloning repositories will scale (I don't know how long this will be the case but right now I'm getting by on my cloud provider's free tier).
Caching helps a little but it's conceivable that someone creating such a module will be making changes quickly and testing so I have to consider that possibility. Even just watching my logs I've seen users load up the app and basically refresh triggering these external calls over and over again. All that to say I'm trying to be mindful of resource usage by preparing layout data upfront when possible and still allowing users to "sideload" custom versions.
I do actually like the idea of looking at west.yml to potentially understand some things a little better (right now I'm waffling on whether or not to try and replace layer label properties with display-name, and one way to be safer about that in the short term might be using that config to see that they're building against ZMK main already) even if I'm not necessarily willing to fetch external resources from it.
Even just allowing the user to paste in the URL to metadata is pretty reasonable, CORS issues aside. I've been toying with the idea of having a "setup wizard" component to the app to help users who get as far as creating an empty GitHub repo and then lose steam, and that would be useful for extending to the out-of-tree keyboards.
I hope that gives you some idea of where my head is at. Sorry if it's a little hard to follow, I'm quickly typing this up during my lunch break, but I'm certainly interested in continuing the discussion and exploring more ideas.

Hey, just want to give you a heads up that I'm not (deliberately) ignoring this. I did have a response drafted yesterday that I generally like the idea but would have to explore options to not get killed by the extra API requests, but this comes after receiving some unfortunate news at work so I'm afraid there won't be any movement on this anytime soon.
If you'd like to keep the discussion going here or share updates with your experiments or conventions for working with Zephyr modules I'm happy to keep reading and offer my insight where I can.


Revert "Depend on model-inference"

Reverts vespa-engine/vespa#6531
I think you need something more... in vespa-yahoo/factory....

Thanks, I had, but that one was not merged.


fix: remove svg files from rollup-plugin-image scope

This PR prevents @rollup/plugin-image to inline svg files instead of @svgr/rollup

:tada: This PR is included in version 1.8.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Feat(plugins): Add password type to metaschema + validator (bgp only)

Change Summary
Add the capability in the schema to validate BGP passwords
Requires #2385
Related Issue(s)
Fixes #1770
Component(s) name
arista.avd.eos_cli_config_gen
arista.avd.plugins
Proposed changes

Add a password type to the metaschema
use the type in eos_cli_config_gen for BGP password for neighbors and peer-groups

How to test
molecule
Checklist
Repository Checklist

[x] My code has been rebased from devel before I start
[x] I have read the CONTRIBUTING document.
[x] My change requires a change to the documentation and documentation have been updated accordingly.
[x] I have updated molecule CI testing accordingly. (check the box if not applicable)

Closing stale PR. Awaiting support for "formats". Still tracked in open issue.


fix(keras): load_model should pass custom_objects when loading models in tf format

# this fails if the loaded model is a SavedModel (saved with format="tf")
from tensorflow.keras import models
model = models.load_model("/path/to/tf_model", custom_objects={"custom_loss": custom_loss})

The custom_object should be passed to the compile correctly (as it is with when loading a h5 file).
Should fix: #33646

Anybody going to take a look at this? Is there something left I have to take care of?

@tanzhenyu gentle ping for review?

same issue here, when you save the model in tf format, you can't re-load the model with custom_objects, this should be fixed.

Issue still presists. Would be great if someone could take a look for a review.

Can a reviewer approve this important PR ? because this bug prevent to load any model with custom optimizer/loss/metric.

Thanks for the PR! A similar fix was submitted and this issue should be resolved in nightly

Is there any other method to reload this function without modifing the original tensorflow code?


persistent advanced options diff with aws.elasticsearch.Domain resource

Hello!

Vote on this issue by adding a 👍 reaction
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already)

Issue details
I am seeing a persistent diff in the aws.elasticsearch.Domain resource like the following:
    ~ aws:elasticsearch/domain:Domain: (update)
...
      ~ advancedOptions: {
          - override_main_response_version: "false"
        }

Steps to reproduce

The resource is configured as follows:
        this.domain = new aws.elasticsearch.Domain(domainName, {
                elasticsearchVersion: "7.7",
                clusterConfig: {
                    instanceType: "t3.small.elasticsearch"
                },
                ebsOptions: {
                    ebsEnabled: true,
                    volumeSize: 10,
                    volumeType: "gp2"
                },
                advancedOptions: {
                    "rest.action.multi.allow_explicit_index": "true",
                    override_main_response_version: "false"
                },
                snapshotOptions: {
                    automatedSnapshotStartHour: 23
                }
            });

Note that the resource existed prior to the opensearch change, which might be relevant given the option in question.
Expected:
No diff.
Actual:
A persistent diff.

I tried the repro today but I am not getting any diffs after the original deployment. Maybe it was a point-in-time issue indeed. I'll go ahead and close the issue as fixed/not actionable.


Recording interactions with TCF notices

Description Of Changes
Adding served_notice_history_id to the TCF fields of the PATCH /privacy-preference request.
Code Changes

[ ] Created a distinction between banner and tcf_banner for FidesUIShown events
[ ] PATCH /notices-served is called from the tcf_banner and the consent overlay
[ ] Updated Cypress tests for the TCF banner worfklows

Steps to Confirm

[ ] Set FIDES__CONSENT__TCF_ENABLED to true
[ ] Start fidesplus
[ ] Navigate to http://localhost:3000/fides-js-demo.html?geolocation=fi-18 to view the TCF banner
[ ] Look for the Dispatching event type FidesUIShown from tcf_banner in the browser console
[ ] Opt-in or opt-out and verify a call to PATCH /privacy-preferences is made, the TCF fields should have the served_notice_history_id

Pre-Merge Checklist

[ ] All CI Pipelines Succeeded
[ ] Issue Requirements are Met
[ ] Update CHANGELOG.md

I was finally able to get back to this. I updated the conditions to trigger the notices served to include the TCF_OVERLAY. The only time we don't trigger the notice is for the consent banner, since that's the only component where notices aren't shown directly.

whew, way to persist through this PR! I know it's probably been haunting you for a while now 😄 great job! ⭐

I'm free!


Homing Projectile: Make a projectile object move towards a target object

Describe the extension
This extension requires two actions to function correctly:

"Lock projectile to target" should be used once per projectile.
"Move projectile towards target" should be run every frame.  This will move all projectiles that are locked to targets.

Parameters:

Rotation Speed: How fast the projectile rotates. This is critical in determining how fast the projectile can turn corners.
Initial Speed: Starting speed of the projectile
Acceleration: How much speed is added over time
Max Speed:  The max speed a projectile can travel (does not apply to physics object)
Max Lifetime:  The number of seconds a projectile can exist after locking onto a target.  Once this time is reached the object is deleted.

Tips:

Make sure to set the angle of the projectile before you use the "Move" action.  The projectile will turn towards the target, but you might want the projectile to be facing a certain direction.
Since Max Speed is not enforced on the physics object, you can accomplish this by changing Linear Decay on the Physics behavior
Use the lifetime parameter to prevent missiles from chasing objects forever

Checklist

[x] Extension has a proper name and description.
[x] Extension has tags (for example: "platform, brick, breakable").
[x] All behaviors have a description.
[x] All functions (actions, conditions, expressions) have descriptions and proper sentences.
[x] I've followed the best practices listed on http://wiki.compilgames.net/doku.php/gdevelop5/extensions/best-practices
[x] I confirm that this extension can be intergrated to this GitHub repository, distributed and MIT licensed.

Extension file
HomingProjectileExtension_0.1.1.json.zip
Game example
https://victrisgames.itch.io/extension-homing-projectile
Video
https://user-images.githubusercontent.com/8879811/120903848-d542e800-c605-11eb-907c-7b217016abe2.mp4

Looks great! Great work, as usual :)
I've seen that you used the mass to have an impulse of ~constant length across objects. How did you come up with this result? We could add a Mass expression maybe too, to avoid doing the manual work you did of computing densitywidthheight.
Anyway, for now I've added the extension! 👍

I agree that the Physics Behavior needs a Mass expression.  I used the AABB area to calculate mass, but that is only accurate if the user has not changed the physics collision mask.  I will create an issue to track this idea.


The scanning number is higher than the migrated data

Hi everyone,
When I run the following command, note the amount of scanned items is 3069 (witout the --scan-match is 56000).
[ec2-user@host ~]$ riot-redis --stacktrace -c -h "$ORIGIN_REDIS_ENDPOINT" -p "$REDIS_PORT" -a "$REDIS_PASSWORD" --tls replicate-ds -c -h "$DESTINATION_REDIS_ENDPOINT" -p "$REDIS_PORT" -a "$REDIS_PASSWORD" --tls --scan-match='l_*' --reader-batch=50 --batch 25 --mode snapshot --no-verify
Scanning 100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3069/3069 (0:00:03 / 0:00:00) 1023.0/s

But when i log to both origin and target redis using the redis-cli and run a keys * command total item count is 600 in both sides.
What does the scanning information means ?

It looks like the scan size estimate was not accurate. It will be fixed in the upcoming release of RIOT

🎉 This issue has been resolved in v3.1.1 (Release Notes)


Dont check storage compatibility is current implementation is address(0)

Might be useful to deploy the first implementation for beacons that are so far unset.

Needs issue to discuss. The beacons the plugin provides will never have implementation address 0 so this PR wouldn't affect them.


[CELEBORN-1003][0.3][FOLLOWUP] Add missing protobuf license file into spark shaded client jars

What changes were proposed in this pull request?
Add missing protobuf license file into spark shaded client jars
Why are the changes needed?
license/LICENSE-protobuf.txt is missing in celeborn-client-spark-2-shaded_2.11-0.3.1-incubating.jar and celeborn-client-spark-3-shaded_2.12-0.3.1-incubating.jar
Note: this is an backporting issue, it only affects branch-0.3, main is fine
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Review

merged to 0.3


Unique Session Per Webview?

Is it possible to create a unique session per webview? I've got a situation where I need to open up tabs to multiple sites on the same server, but I want each tab to be its own session. It would be quite convenient if I could control this on a per-tab basis, so I could make two tabs share the same session where two other tabs would have their own session entirely.

This is a duplicate of #959.

@zcbenz I don't understand why this task, along with https://github.com/atom/electron/issues/2032, have been closed as a duplicate of #959. The latter issue is completely different (although with a misleadingly similar title). It talks about being able to persist session and restore it. Not about being able to have multiple active sessions in concurrence in different tabs.

Any update on this? Or is there any solution to this? @gajus @zcbenz

I think this issue should be open since there is no solution to this.

@geeeeeeeeek https://github.com/atom/electron/pull/3514 ?

@gajus
I tried partition:A and partition:B for two BrowserWindows. However when I delete cookies in one of the two sessions, cookies in the other session are also deleted. Behaves like they still share the same session.
Code:
  let ses = windows[name].webContents.session;
  ses.cookies.remove("https://wx.qq.com", "wxuin", ()=>{})

I have passed different partition strings to new BrowserWindow object.
Anything wrong with my code? Could you share yours?

I am also interested in this. Any updates? I can't seem to get it to work either using different partitions

So this was just closed and two years later, nobody of the core team cared enough to reopen it?
I'm looking for this functionality as well. I have one URL where I want to sign in with multiple accounts at the same time. This issue prevents me from doing that because the session is shared between all the webviews. I looked into the docs of webview and tried to set a different partition and guestinstance, but no dice, it just doesn't work.

I'm looking for this functionality as well, how solve?

Would love to know how to do this as well.


User feed filter background color

Looking that the user feed filter, it seems to get a little lost and doesn't have a very compliant design, I thought doing something like this:

As against the original:

It could do with some text color changes or even a font change but I believe something needs to be done.

Something like this? cc @JoshFabian
After:

Before:

yea, looks good


Remove qv_policy_t.

Remove qv_policy_t since it doesn't appear to be used.

@samuelkgutierrez : I left it in there intentionally. I had a discussion with Edgar today about what I wanted to do with it and I plan to discuss this on Tuesday's meeting.

We can discuss where this should go and its naming next meeting.


pgm-call: result-xml structure corrupt due to randomly added characters (urgent)

Node.js version: 12.14.0
itoolkit version: 1.0.0
IBM i version: 7.3
XMLSERVICE version: XML Toolkit 2.0.2-dev

Describe the bug
The node-script requests IBM i DB-Data via a call to rpg-pgm.
As working with daily-data for one month the call to the program is added to a toolkit conn-object for 30/31 times.
Running the con results in a successful execution but gives back a corrupted xml-structure (e. g. ... type=a'4p2' ..., ... <d ...).
We are using different person-ids in the request. With some PER# the error is reproducible others work fine.
Maybe there is a stringbuffer involved?!?
! Running the script multiple times with identic PERS# results always in the same error at the same position in xml-string.
To Reproduce
see above
Expected behavior
Please locate the bug and provide a corrected version of itoolkit or xmlservice
Files
If needed, we can provide our code snippet and some xml-data.

@dmabupt
Any thoughts on what may be causing this issue?

@dmabupt
Any thoughts on what may be causing this issue?

Seems to be an encoding issue. Need to reproduce it for debugging.

This is a good point. The error occurs only for employees with names including "ä ö ü Ä Ö Ü" german special-characters.
Data for other persons seem to be fine.

Interesting I wonder if this issue is related to https://github.com/IBM/nodejs-idb-connector/issues/70.

Or try this #20

Read through your references, but didn't got a clue how to proceed.
Maybe it's worth mentioning that our scripts run on the same IBM i system, where the database is located.
Is there any character conversion involved? Where?

Read through your references, but didn't got a clue how to proceed.
Maybe it's worth mentioning that our scripts run on the same IBM i system, where the database is located.
Is there any character conversion involved? Where?

In the beginning of your Node.js code, adding process.env.DB2CCSID = '0'; to make your program running with its job CCSID and check if it resolve the problem.

At the beginning of your Node.js code, add process.env.DB2CCSID = '0'; to make your program running with its job CCSID and check if it resolve the problem.

Well, the error with malformed xml-tags disappeared ...
... but now we have n issue with special characters (öäü) in the data.
see https://github.com/markdirish/node-odbc/issues/79

From my point of view this issue is not fixed yet. If we are forced to use DB2CCSID='0', we can not use this connector anymore as the special characters are not processed correctly.
How can we proceed with this?

Not sure why the bot closed it - reopening.

@kadler please reopen as it is not fixed. What could be the next step from your / our side?

Since this is a conversion issue, can you please provide

value of QCCSID
job CCSID
job default CCSID
job langid
job cntryid
PASE locale used (locale -a)

FYI, I asked for locale -a, but that was wrong. I meant just to run locale. I've updated my comment above.

value of QCCSID
1141
    job CCSID
1141
    job default CCSID
1141
    job langid
DEU
    job cntryid
DE
PASE locale used (locale)
QSH:
LANG=/QSYS.LIB/DE_DE.LOCALE
LC_COLLATE=
LC_CTYPE=
LC_MESSAGES=
LC_MONETARY=
LC_NUMERIC=
LC_TIME=
LC_ALL=
Putty/bash:
LANG=
LC_COLLATE="C"
LC_CTYPE="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_MESSAGES="C"
LC_ALL=

I have a hunch that the problem is occurring here: https://github.com/IBM/nodejs-idb-connector/blob/master/src/db2ia/dbstmt.cc#L2477
If the SQLFetch returns an indicator with a valid length (ie. not SQL_NTS or SQL_NULL_DATA), we will copy the data but not null terminate it here: https://github.com/IBM/nodejs-idb-connector/blob/master/src/db2ia/dbstmt.cc#L2418-L2421
Since it's not null-termiated and we're not passing in the length, the Napi::String::New constructor will just keep going until it finds a null-terminator.

I have a hunch that the problem is occurring here: https://github.com/IBM/nodejs-idb-connector/blob/master/src/db2ia/dbstmt.cc#L2477
If the SQLFetch returns an indicator with a valid length (ie. not SQL_NTS or SQL_NULL_DATA), we will copy the data but not null terminate it here: https://github.com/IBM/nodejs-idb-connector/blob/master/src/db2ia/dbstmt.cc#L2418-L2421
Since it's not null-termiated and we're not passing in the length, the Napi::String::New constructor will just keep going until it finds a null-terminator.

@kadler Maybe we should detect the rlength parameter first? Like --
        default:
          if(resultSetInC[row][col].rlength > 0)
            value = Napi::String::New(env, resultSetInC[row][col].data, resultSetInC[row][col].rlength);
          else
            value = Napi::String::New(env, resultSetInC[row][col].data)
          break;

Alternatively, just always set rlength to the proper value here: https://github.com/IBM/nodejs-idb-connector/blob/master/src/db2ia/dbstmt.cc#L2404-L2414
When it's SQL_NTS we already get the length in to colLen (why we don't use strdup here, I'm not sure). For SQL_NULL_DATA, we could set it to 0, and for other negative values it could be set to 1 (in the future, this code should probably handle this case better than returning "-", though).

Alternatively, just always set rlength to the proper value here: https://github.com/IBM/nodejs-idb-connector/blob/master/src/db2ia/dbstmt.cc#L2404-L2414
When it's SQL_NTS we already get the length in to colLen (why we don't use strdup here, I'm not sure). For SQL_NULL_DATA, we could set it to 0, and for other negative values it could be set to 1 (in the future, this code should probably handle this case better than returning "-", though).

I am not sure about the behavior of Napi::String::New() when the size is 0. I may detect the rlength for now.

I'm not sure either. AFAICT it doesn't even mention the ability to pass a NULL pointer.

I guess I would set the length correctly always, then pass it in if the pointer is not NULL, otherwise just pass in the NULL pointer.

Hello @nedi-dev , idb-connector 1.2.10 has been released. You may upgrade to this version and check if it fix the issue.
If not, we still need to recreate the issue on local system first.

Thank you for the new release of idb-connector. Unfortunately it took some days to get back to the project.
Even more worse: The error still shows up!
After reinstalling the package<EMAIL_ADDRESS>the npm list looks like this:
bash-4.4$ npm ls idb-connector
--<EMAIL_ADDRESS> +--<EMAIL_ADDRESS>  --<EMAIL_ADDRESS>`--<EMAIL_ADDRESS>deduped
In the xml you can find:
...

<datpa type='4p2' name='PKommt'>0,00
0,00
...
Please let me know if I can support you recreating the issue on your system.

Hello @nedi-dev ,
I did a simple test to check if idb-connector can handle the data with CCSID 1141:

Create a table with a column with CCSID 1141

CREATE TABLE UNITAB(C1 CHAR(20) CCSID 1141);
INSERT INTO UNITAB VALUES('öäüABCÅÄÖ');
SELECT * FROM UNITAB;

Query this table using idb-connector

const {dbconn, dbstmt} = require('idb-connector');

const sSql = 'SELECT * FROM UNITAB';
const connection = new dbconn();
connection.conn('*LOCAL');
const statement = new dbstmt(connection);

statement.exec(sSql, (x) => {
  console.log(JSON.stringify(x));
  statement.close();
  connection.disconn();
  connection.close();
});

The output is correct:

$ node index.js
[{"C1":"öäüABCÅÄÖ           "}]

Your program is a little different. There is a program LGIZEIT/ZEI109 between idb-connector and the backend database.
I think we may need to find out what the backend database is and then try to query the table using idb-connector directly.
We need to check if the problem exist in the program LGIZEIT/ZEI109 or the backend database.

This has just cropped up for me today. Just another one of those IBM i things.
I have a program call using XMLSERVICE/itoolkit and the program returns a long character string (65k). It is using the idb transport. The long character parameter is a JSON string generated in the program.
See below; the O character should not be there.

I can also see that it is coming back in the XML in the xmlOutput from itoolkit.
  connection.add(programCall);
  connection.run((error, xmlOutput) => {
    console.log(xmlOutput); //I can see that invaid 'O' here.

Also, it can move around and it's not always O. Here's another example of it happening somewhere else:

Another important note is that this only happens when using the idb transport. I've tried to recreate it with ssh and it seems to work perfectly there.

To reaffirm this issue, I switched the app over to ODBC (node-odbc) and it has resolved the issue.


Encoding NuGet source names

Encoding the source name so it is a valid XML element.

Hi @emanuelquintero, I'm your friendly neighborhood Microsoft Pull Request Bot (You can call me MSBOT). Thanks for your contribution!
You've already signed the contribution license agreement. Thanks!
The agreement was validated by Microsoft and real humans are currently evaluating your PR.
TTYL, MSBOT;

Closing and creating https://github.com/Microsoft/vso-agent-tasks/pull/781


Placeholder Images get streched/shrinked if actual Source Image has a different aspect ratio

We're using FFImageLoading with placeholder images which are shown during loading (as you likely will know).
When those placeholder images fade to the actual source image after its loaded via http, the placeholder images are stretched to the source image's aspect ratio during that fade.

Yes, I think we have the same issue as https://github.com/square/picasso/issues/427
I have no idea yet how to fix it.

+1 for Xamarin.Forms. Solution may be not cross-fade (placeholder disappear and image appear the same time) but do it step-by-step - placeholder disappear with animation, image view resize itself, image appear with animation. Currently cross-fade don't look good, sequence may looks better :)

Here's the code for fading on Android: https://github.com/luberda-molinet/FFImageLoading/blob/28ca8b323215a5fbbdcb5fd0b7f3274f0dda7af4/source/FFImageLoading.Droid/Drawables/FFBitmapDrawable.cs#L29-L66

Should be fixed in 2.1.9-pre-168-core-r. Please reopen it if doesn't work for you.


Training error: ModuleNotFoundError: No module named 'pointnet2_ops'

Hi
I am getting error during training. bash ./scripts/train.sh 0     --config ./cfgs/PCN_models/PoinTr.yaml     --exp_name example
from pointnet2_ops import pointnet2_utils
ModuleNotFoundError: No module named 'pointnet2_ops'
I tried ti install:
pip install "git+git://github.com/erikwijmans/Pointnet2_PyTorch.git#egg=pointnet2_ops&subdirectory=pointnet2_ops_lib"
Collecting pointnet2_ops

Hi,
I have solved by adding manually this repo in extensions folder then Run
python setup.py install --user manually under each dependency directory
pointnet2_ops_lib.zip

Thanks

Hi, I have solved by adding manually this repo in extensions folder then Run python setup.py install --user manually under each dependency directory
pointnet2_ops_lib.zip

I exactly did as said,
I'm facing this error,
Help please.
nvcc fatal   : Unsupported gpu architecture 'compute_37'
error: command '/usr/local/cuda/bin/nvcc' failed with exit code 1

您好，您的信息我已收到


KAFKA-13270: Set JUTE_MAXBUFFER to 4 MB by default

ZooKeeper 3.6.0 changed the default configuration for JUTE_MAXBUFFER from 4 MB to 1 MB.
This causes a regression if Kafka tries to retrieve a large amount of data across many
znodes – in such a case the ZooKeeper client will repeatedly emit a message of the form
"java.io.IOException: Packet len <####> is out of range".
We restore the 3.4.x/3.5.x behavior unless the caller has set the property (note that ZKConfig
auto configures itself if certain system properties have been set).
I added a unit test that fails without the change and passes with it.
I also refactored the code to streamline the way we handle parameters passed to
KafkaZkClient and ZooKeeperClient.
See https://github.com/apache/zookeeper/pull/1129 for the details on why the behavior
changed in 3.6.0.
Credit to @rondagostino for finding and reporting this issue.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@omkreddy if you have cycles, this is a 3.0 blocker.

@dajac Thanks for the review, I pushed a commit that addresses your feedback.


HashStore Utility/Converter

After further discussion with the team, we have confirmed that HashStore will assist the client, Metacat, in the migration process for existing data and metadata objects by providing a new utility class. Metacat will coordinate the iterative process so that the respective default checksum list for each data object is stored. This class, HashStoreConverter, has a single convert method:

convert(Path pathToDoc, String pid, Stream systema)

Creates a stream from the path to the object, generates the 5 default checksums, creates a hard link between an existing data or metadata object, tags the pid and data object, and stores the system metadata.

Note: If a given path is null, but system metadata is present, we will store the system metadata only with the given pid.
Note 2: The process may fail, in which Metacat may attempt to restore hard links for the entire set of data and documents. We need to be sure to check for the existence of a data object/pid relationship to save time before attempting the hard linking process.

This utility class will need to call a new class FileHashStoreLinks which extends FileHashStore. We will have a new public method storeHardLink(...) which will follow the same existing flow as storeObject. Except this method will eventually arrive at a new override method for move, which calls Files.createLink instead of Files.move.
// Files.move(sourceFilePath, targetFilePath, StandardCopyOption.ATOMIC_MOVE);
Files.createLink(sourceFilePath, targetFilePath);

Process Diagram (via @taojing2002 and @artntek)

To Do:

Create two new classes in a package under hashstore called 'hashstoreconverter'

HashStoreConverter

convert(Path pathToDoc, String pid, Stream systema)

FileHashStoreLinks

storeHardLink(InputStream object, String pid)
move(Path sourcePath, Path targetPath)

Write new junit tests to confirm functionality

Unfortunately, simply overriding move will not be enough to address this issue. To create a proper hard link, the original supplied path pathToDoc must be used as the source.
Using the existing storeObject flow, and overriding the final move operation, will actually only create a link between the tmpFile generated and the target permanent address. As a result, this link points to the new inodes/data blocks allocated for tmpFile, rather than referencing the original inodes of pathToDoc.

zenuml Syntax
zenuml
    title HashStoreConverter Process
    Client->HashStoreConverter.convert(Path existDoc,pid, Stream sysmeta) {
      "new InputStream existDocStream"
      FileHashStoreLinks.storeHardLink(existDocStream, pid) {

        FileHashStore.storeObject(existDocStream, pid) {
          syncPutObject
          writeToTmpFileAndGenerateChecksums
          move
          // -
          // Override 'move'
          // Creates a hard link
          FileHashStoreLinks->FileHashStoreLinks
          return ObjectMetadata
        }

        FileHashStore.storeMetadata(sysmeta, pid) {
        // -
        // If sysmeta fails to store,
        // an exception will be thrown.
        // -
        // The hard link/tags created
        // for the data obj will remain.
        return pathToSysmeta
        }
        return ObjectMetadata
      }
      return ObjectMetadata
    }

Instead of overriding move, we will not follow the storeObject flow directly but indirectly (like in the scenario where a client receives a data stream first before metadata). However, instead of calling storeObject, we will directly call writeToTmpFileAndGenerateChecksums.
Since we are really only after the map of checksums/hex digests, there is no need to follow the storeObject synchronization process as the tmpFile being written into is discarded after. Once that is completed, we can call tagObject which is thread-safe and synchronized, then create the hard link, and lastly store the sysmeta.

zenuml Syntax New Flow
zenuml
    title HashStoreConverter Process
    Client->HashStoreConverter.convert(Path existDoc,pid, Stream sysmeta) {
      "new InputStream existDocStream"
      FileHashStoreLinks.storeHardLink(existDocStream, pid) {
        FileHashStore.generateTmpFile {
            return tmpFile
        }
        FileHashStore.writeToTmpFileAndGenerateChecksums {
          return hexDigests/Checksums
        }
        delete(tmpFile)
        // -
        // 'tagObject' is synchronized/thread safe
        FileHashStore.tagObject(pid, cid)

        FileHashStore.getHashStoreDataObjectPath(cid) {
            return cidExpectedObjectPath
        }

        createHardLink(existDoc, cidExpectedObjectPath)
        FileHashStore.storeMetadata(sysmeta, pid) {
        // -
        // If sysmeta fails to store,
        // an exception will be thrown.
        // -
        // The hard link/tags created
        // for the data obj will remain.
        return pathToSysmeta
        }
        return ObjectMetadata
      }
      return ObjectMetadata
    }

That makes sense. So maybe in the new move method, it should do nothing or
just delete the temp file. The Converter.convert method will do the job to
create the hard link - it has the original path and it has the
ObjectMetadata instance which has the sha256 checksum ( and can break it
down to the file path), so link them together.
On Thu, Jul 25, 2024 at 1:45 PM Dou Mok @.***> wrote:

Unfortunately, simply overriding move will not be enough to address this
issue. To create a proper hard link, the original supplied path pathToDoc
must be used as the source.
Using the existing storeObject flow, and overriding the final move
operation, will actually only create a link between the tmpFile generated
and the target permanent address. As a result, this link points to the new
inodes/data blocks allocated for tmpFile, rather than referencing the
original inodes of pathToDoc.
image.png (view on web)
https://github.com/user-attachments/assets/36a15e8b-45dc-45e5-ae02-ff2af81606a9 zenuml
Syntax
zenuml
title HashStoreConverter Process
Client->HashStoreConverter.convert(Path existDoc,pid, Stream sysmeta) {
"new InputStream existDocStream"
FileHashStoreLinks.storeHardLink(existDocStream, pid) {
    FileHashStore.storeObject(existDocStream, pid) {
      syncPutObject
      writeToTmpFileAndGenerateChecksums
      move
      // -
      // Override 'move'
      // Creates a hard link
      FileHashStoreLinks->FileHashStoreLinks
      return ObjectMetadata
    }

    FileHashStore.storeMetadata(sysmeta, pid) {
    // -
    // If sysmeta fails to store,
    // an exception will be thrown.
    // -
    // The hard link/tags created
    // for the data obj will remain.
    return pathToSysmeta
    }
    return ObjectMetadata
  }
  return ObjectMetadata
}

Loading
Instead of overriding move, we will not follow the storeObject flow
directly but indirectly (like in the scenario where a client receives a
data stream first before metadata). However, instead of calling
storeObject, we will directly call writeToTmpFileAndGenerateChecksums.
Since we are really only after the map of checksums/hex digests, there is
no need to follow the storeObject synchronization process as the tmpFile
being written into is discarded after. Once that is completed, we can call
tagObject which is thread-safe and synchronized, then create the hard
link, and lastly store the sysmeta.
image.png (view on web)
https://github.com/user-attachments/assets/18d68535-bf4c-4260-8e55-ee7b47bc2e97 zenuml
Syntax New Flow
zenuml
title HashStoreConverter Process
Client->HashStoreConverter.convert(Path existDoc,pid, Stream sysmeta) {
"new InputStream existDocStream"
FileHashStoreLinks.storeHardLink(existDocStream, pid) {
FileHashStore.generateTmpFile {
return tmpFile
}
FileHashStore.writeToTmpFileAndGenerateChecksums {
return hexDigests/Checksums
}
delete(tmpFile)
// -
// 'tagObject' is synchronized/thread safe
FileHashStore.tagObject(pid, cid)
    FileHashStore.getHashStoreDataObjectPath(cid) {
        return cidExpectedObjectPath
    }

    createHardLink(existDoc, cidExpectedObjectPath)
    FileHashStore.storeMetadata(sysmeta, pid) {
    // -
    // If sysmeta fails to store,
    // an exception will be thrown.
    // -
    // The hard link/tags created
    // for the data obj will remain.
    return pathToSysmeta
    }
    return ObjectMetadata
  }
  return ObjectMetadata
}

Loading
—
Reply to this email directly, view it on GitHub
https://github.com/DataONEorg/hashstore-java/issues/95#issuecomment-2251365168,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB5QQDFJNFJCP6QOMHBMUW3ZOFPXVAVCNFSM6AAAAABLNKH2YKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJRGM3DKMJWHA
.
You are receiving this because you were mentioned.Message ID:
@.***>

Thank you for the quick feedback @taojing2002! I believe we actually don't need to override the move method at all. The HashStoreConverter.convert method will make two calls to FileHashStoreLinks:

storeHardLink(Path existDoc, InputStream existDocStream, String pid)

This will get the 5 default checksums by writing into and then deleting a tmpFile
Then it will call tagObject, and finally create the actual hard link afterwards

I am contemplating creating the hard link first before calling tagObject

storeMetadata(InputStream sysmeta, String pid)

Updated zenuml diagram

zenuml Syntax Converter Process Updated
zenuml
    title HashStoreConverter Process
    Client->HashStoreConverter.convert(Path existDoc,pid, Stream sysmeta) {
      "new InputStream existDocStream"
      FileHashStoreLinks.storeHardLink(existDoc, existDocStream, pid) {
        FileHashStore.generateTmpFile {
            return tmpFile
        }
        FileHashStore.writeToTmpFileAndGenerateChecksums {
          return hexDigests/Checksums
        }
        delete(tmpFile)
        // -
        // 'tagObject' is synchronized/thread safe
        FileHashStore.tagObject(pid, cid)

        FileHashStore.getHashStoreDataObjectPath(cid) {
            return cidExpectedObjectPath
        }

        createHardLink(existDoc, cidExpectedObjectPath)
        return ObjectMetadata
      }
        // -
        // Close object stream
        "existDocStream.close()"
        FileHashStoreLinks.storeMetadata(sysmeta, pid) {
        // -
        // If sysmeta fails to store,
        // an exception will be thrown.
        // -
        // The hard link/tags created
        // for the data obj will remain.
        FileHashStore.storeMetadata(sysmeta, pid) {
            return pathToSysmeta
        }
            return pathToSysmeta
        }
        // -
        // - Close sysmeta stream
        "sysmeta.close()"
        return ObjectMetadata
    }

Sounds good. To me, it sounds reasonable to create the hard link first.
Creating the link means the object is ready, then you can tag it.
On Thu, Jul 25, 2024 at 3:06 PM Dou Mok @.***> wrote:

Thank you for the quick feedback @taojing2002
https://github.com/taojing2002! I believe we actually don't need to
override the move method at all. The HashStoreConverter.convert method
will make two calls to FileHashStoreLinks:

storeHardLink(Path existDoc, InputStream existDocStream, String pid)

This will get the 5 default checksums by writing into and then
deleting a tmpFile
Then it will call tagObject, and finally create the actual hard
link afterwards

I am contemplating creating the hard link first before calling
tagObject

storeMetadata(InputStream sysmeta, String pid)

Updated zenuml diagram
image.png (view on web)
https://github.com/user-attachments/assets/15992a57-c52f-44bb-987a-bb317d3a8379 zenuml
Syntax Converter Process Updated
zenuml
title HashStoreConverter Process
Client->HashStoreConverter.convert(Path existDoc,pid, Stream sysmeta) {
"new InputStream existDocStream"
FileHashStoreLinks.storeHardLink(existDoc, existDocStream, pid) {
FileHashStore.generateTmpFile {
return tmpFile
}
FileHashStore.writeToTmpFileAndGenerateChecksums {
return hexDigests/Checksums
}
delete(tmpFile)
// -
// 'tagObject' is synchronized/thread safe
FileHashStore.tagObject(pid, cid)
    FileHashStore.getHashStoreDataObjectPath(cid) {
        return cidExpectedObjectPath
    }

    createHardLink(existDoc, cidExpectedObjectPath)
    return ObjectMetadata
  }
    // -
    // Close object stream
    "existDocStream.close()"
    FileHashStoreLinks.storeMetadata(sysmeta, pid) {
    // -
    // If sysmeta fails to store,
    // an exception will be thrown.
    // -
    // The hard link/tags created
    // for the data obj will remain.
    FileHashStore.storeMetadata(sysmeta, pid) {
        return pathToSysmeta
    }
        return pathToSysmeta
    }
    // -
    // - Close sysmeta stream
    "sysmeta.close()"
    return ObjectMetadata
}

Loading
—
Reply to this email directly, view it on GitHub
https://github.com/DataONEorg/hashstore-java/issues/95#issuecomment-2251481046,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB5QQDCBTZMF6T4JCLQYOYTZOFZGDAVCNFSM6AAAAABLNKH2YKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJRGQ4DCMBUGY
.
You are receiving this because you were mentioned.Message ID:
@.***>

@doulikecookiedough Some idle noodling on all this, and how long it will take...
Process is still writing, not linking
The purpose of the use of a hard link is to completely avoid having to rewrite the data to disk. If you call writeToTmpFileAndGenerateChecksums, doesn't that rewrite the data  to a tempfile in the process of calculating checksums? If so, then I think there will be no performance gain at all compared to just calling store_object() on all of the objects. If you can avoid the write altogether it has promise. Reading the data from disk to calculate the checksums should be faster I think. For a random 3GiB file on our cephfs filesystem, I just measured that it takes about 6.7s to cp the file to a new location, and about 20s to calculate a sha256 hash. When that gets done for the 140+TB of data on the ADC, its going to take a long time.
Batching with CephFS API
Also, one of the things that it would be good to avoid is running an iterative loop across 2 million+ objects and calling a method on each, which will take a long time on its own. When I first mentioned this optimization, I mentioned that it would be much faster to call the CephFS API to modify the metadata server directly (hopefully in batch ops) than it would be to use the posix link ln command on each file. Even these metadata operations on this many files is going to be very time consuming. Maybe the link API would be faster than using ln or cp -l, maybe not, needs some testing.
Possible alternative to consider
An alternative approach could be below, that I have no idea whether it will work or is supported by the Ceph API, jst food for thought:
1) QUERY: query metacat to get a table listing the PID, SID, docid, rev, and sysmeta for every object in the system
2) LINK OBJECTS: use that info to batch update cephfs creating hard links for all of the objects (see https://docs.ceph.com/en/reef/cephfs/api/libcephfs-py/#cephfs.LibCephFS.link) and create other hashstore info needed for each file
3) CALC CHECKSUMS: in a multithreaded process, iterate across all obects and calculate and store the checksums of the objects
4) UPDATE/UPGRADE METACAT: using the checksums from 3) do a batch update of all of those checksums in the Metacat checksums table (e.g., use `SQL COPY` not `SQL INSERT`)

Some quick metrics on our system write speeds
Of course, this is entirely out-of-band from both hashstore and metacat, but it might be orders of magnitude faster than trying to insert millions of objects one call at a time. In particular, once (2) is done, you can then massively parallelize (3) on our k8s cluster (although we will be quickly limited by read I/O rate).  Because this process is out-of-band (and specific to our systems), it probably would not be the best automatic upgrade path for small metacat deployments. Rather, I see this as a pre-processing step to prepare a hashstore dir and set of checksums for our large installations that have a lot of objects (e.g., ADC/KNB/cn.dataone.org). Smaller metacat installs might just use the Hashstore API you are designing to convert directly.
A few stats follow for reference. I created a directory with 100 3 GiB files on our cephfs filesystem (named testfile-100 to testfile-199).
Hardlinking all of the files was fast, and took 2.0 seconds, which is an average of 50 files per second, or 1.5 GiB/s
jones@datateam:/mnt/ceph/home/jones/test$ time seq 100 199 | parallel "cp -l testfile-{} linkfile-{}"

real	0m2.006s
user	0m1.048s
sys	0m2.214s

Copying all of the data to a new file was I/O bound, took 4m58s, which is an average of 0.01 GiB/s
jones@datateam:/mnt/ceph/home/jones/test$ time seq 100 199 |  parallel "cp testfile-{} cpfile-{}"

real	4m58.120s
user	0m2.092s
sys	26m31.029s

We have about 140TB of data on the ADC. So, at that rate, it would take about 3982 hours (166 days) to make a single write for the data, whereas hardlinking the same files would be much faster (a few days). All of that depends on sustained rates of transfer, which would likely be lower than this small test demonstrates (both because there would be more activity on the systems, and because these are all identical files so probably benefiting from OS caching/optimization). So, if we can cut back on our writes substantially, we will save a lot of time in the conversion. Last, calculating checksums for these files will also take some time:
jones@datateam:/mnt/ceph/home/jones/test$ time seq 100 199 | parallel "shasum -a 256 testfile-{} > sha-{}"

real	2m5.624s
user	39m51.798s
sys	5m20.154s

So, that's about half the time of the file writes, and hopefully something that we can parallelize much more extensively on the k8s cluster because sha calculations are generally compute-bound.
All just food for thought and discussion. Not sure what you should do, but thought it was worth pointing out your "link" op is really a "write" op.

Thank you for sharing your insightful thoughts with us @mbjones :pray:!
If I call writeToTmpFileAndGenerateChecksums, I'm indeed writing to a tmpFile while calculating the checksums - so there is no performance gain. However, I think creating a hardlink is still a required step so that we aren't using up more disk space than we need to. Regarding how to speed up the process, I think we should override writeToTmpFileAndGenerateChecksums so that it only reads into a buffer to calculate the checksums (skipping the writing process as it is not necessary).
RE: Alternative Approach & CephFS
Unfortunately... it looks like libcephfs for Java is no longer actively supported/maintained. Leveraging CephFS's API would have to be done in Python - which would mean this tool would now exist outside of Metacat's control. I think that this upgrade process should be coordinated by Metacat, otherwise it could become confusing for Metacat operators to have to download a separate tool to upgrade to HashStore, in addition to upgrading Metacat itself.
I will let this simmer in my mind for a bit, your feedback has been extremely helpful!

Updated Proposed Process:

FileHashStoreLinks.storeHardLink

Calls a new method generateChecksums that generates the default 5 checksums (no writing)
Create hard link afterwards
Call tagObject

zenuml Syntax Converter Process Updated
zenuml
    title HashStoreConverter Process
    Client->HashStoreConverter.convert(Path existDoc,pid, Stream sysmeta) {
      "new InputStream existDocStream"
      FileHashStoreLinks.storeHardLink(existDoc, existDocStream, pid) {
        FileHashStore.generateChecksums {
          return hexDigests/Checksums
        }
        FileHashStore.getHashStoreDataObjectPath(cid) {
            return cidExpectedObjectPath
        }
        createHardLink(existDoc, cidExpectedObjectPath)
        // -
        // 'tagObject' is synchronized/thread safe
        FileHashStore.tagObject(pid, cid)

        return ObjectMetadata
      }
        // -
        // Close object stream
        "existDocStream.close()"
        FileHashStoreLinks.storeMetadata(sysmeta, pid) {
        // -
        // If sysmeta fails to store,
        // an exception will be thrown.
        // -
        // The hard link/tags created
        // for the data obj will remain.
        FileHashStore.storeMetadata(sysmeta, pid) {
            return pathToSysmeta
        }
            return pathToSysmeta
        }
        // -
        // - Close sysmeta stream
        "sysmeta.close()"
        return ObjectMetadata
    }

@doulikecookiedough thanks. In my comments above, I made a pretty big calculation error on my data throughput rates, which I edited above. So instead of 3982 hours for our 140TB cp, that should have been 39.82 hours. Pretty huge difference. Wanted to point this out as it changes my thinking on the scale of this conversion.

Thank you again @mbjones for sharing your thoughts with us and the update. I realize that I may have created some confusion regarding our approach to the conversion process as well.
While Metacat will likely set up the process with a loop, the execution phase will utilize Java's concurrency APIs (TBD, perhaps via the Java Collection object's .parallelStream().forEach(...) route) to handle the data efficiently.
The out-of-band process described sounds quite promising - but Metacat will eventually need to store the system metadata for each data object as well. Given this, along with how we want to store a hard link to optimize disk usage, I feel that the convert process outlined in this issue feels appropriate/suitable for our needs. Also, confirming that we aren't writing any bytes to disk anymore.

Update:
After further discussion with @taojing2002 and @artntek, we have agreed to combine the DB Upgrade & HashStore conversion process together.

For smaller repositories, this combined process should be relatively quick and minimally impact the user experience.

When operators configure the DB or initiate the process, it will also automatically trigger the HashStore conversion.

Operators will be redirected to the properties configuration page, where the status will be displayed as in progress along with placeholder text Please refresh for updates where the DB version is usually shown.

RE: Jumbo Repositories (KNB, DataONE, ADC) - Minimize Downtime (<2 minutes or less?)

First, we will investigate how long the existing process may take with test data on the knbvm server and continue our discussions from there.

A note on the jumbo repos comment -- the reason that I think the downtime can be kept to a few minutes is that the conversion of files from the existing store to HashStore can be done ahead of time without loss (and hopefully without re-writing existing data), recording the checksum results for later update in Metacat. A rough pseudo algorithm might be:

Start parallel system process to convert HashStore files
1a) Batch query metacat to get list of all identifiers, key metadata, and their systemMetadata, and note the time this query was made
for each file:
2a) read file to calculate checksums, and save checksums for later use
2b) hard link file into HashStore location
2c) write sysmeta to HashStore location
2d) write PID annotations and other metadata needed to HashStore
With completed list of PID/Checksums, start Metacat upgrade process (read-only mode), and BATCH copy the metadata into appropriate postgres tables (e.g., don't do iterative INSERTS, rather do SQL COPY and batch updates)
Sweep up any new identifiers that were touched after query time of step 1a and convert those few objects before resuming read-write operations

Hope this is helpful. Other approaches would be feasible too, this is just one possible path.

@taojing2002 I have tested the HashStoreConverter with the data found in knbvm, specifically the first 10000 rows retrieved from metacat's pg db where the size < 1 gb:

2791 data objects
2791 cid reference files
4830 pid reference files

For the normal storeObject process, it takes approximately 17 minutes to write a data file and calculate the default checksums.
For the hardlinking process convert / storeHardLink, it takes approximately 8 minutes to calculate the default checksums and create a hard link.

This has been completed via Feature-95: HashStoreConverter & FileHashStoreLinks - however, leaving this issue open to continue discussing plans/strategy to reduce downtime for jumbo repos.

The hard link conversion process was run for all data objects on knbvm via the hashstore client and took approximately 22 hours and 8 minutes (~5.9T)
For quick reference:
mok@knbvm:/var/metacat$ du -h -d 1
0       ./benchmark
0       ./inline-data
du: cannot read directory './.metacat': Permission denied
0       ./.metacat
1.5G    ./documents
512     ./users
178K    ./logs
5.9T    ./hashstore
du: cannot read directory './config': Permission denied
0       ./config
910M    ./solr-home3
du: cannot read directory './tdb/tdb16272684195302037564': Permission denied
409G    ./tdb
4.5K    ./dataone
507G    ./data
421M    ./temporary
57K     ./certs
6.8T    .

Closing this issue as the team has developed a plan to upgrade.


[jssrc2cpg][swiftsrc2cpg] Improved logging of files being ignored by default

astgen and swiftastgen do not log ignored files.
That may result in an empty CPG without the user even noticing. With this PR, we info log any file that was ignored in that particular case.

As far as I can tell, this change would log each individual ignored file at the info level. For large ignored directories (test directories, for example), this could get very noisy. Perhaps we should keep the individual logs at debug level, but the number of files parsed at info level or warn if it's 0 (or just info/warn on 0)?

Yeah, good point. I changed it.

this duplicates the list of ignore patterns in two places, right? Once here, once in the astgen implementations? That seems brittle - it's only a matter of time until those lists (or their interpretation) diverge.
Would be better to either have the astgens return the list of files they've decided to ignore as part of their output, or get their ignore patterns as input.

@maltek
Yes, thats right and may become a problem in the future once they diverge.
I am working on a solution for that: all the astgen tools will produce a single .json file describing parsed files, un-parsable files (with the cause for that), and ignored files.
With this approach, no frontend will have to parse the stdout/stderr output anymore and we have standard for handing over information from astgen -> frontend.
This PR is just a intermediate solution to show at least some information why your CPG is empty.

OK - no objection from me if this is just meant as a temporary solution :+1:


refactor: split aa-alchemy into account-kit packages

BREAKING CHANGE: this removes the @alchemy/aa-alchemy package in favor of @account-kit/*
Pull Request Checklist

[ ] Did you add new tests and confirm existing tests pass? (yarn test)
[ ] Did you update relevant docs? (docs are found in the site folder, and guidelines for updating/adding docs can be found in the contribution guide)
[ ] Do your commits follow the Conventional Commits standard?
[ ] Does your PR title also follow the Conventional Commits standard?
[ ] If you have a breaking change, is it correctly reflected in your commit message? (e.g. feat!: breaking change)
[ ] Did you run lint (yarn lint:check) and fix any issues? (yarn lint:write)
[ ] Did you follow the contribution guidelines?

Everything here is basically just a refactor moving things around:

@alchemy/aa-alchemy -- deleted
@account-kit/core -- contains everything that aa-alchemy had in it except for the things in the packages below
@account-kit/react -- this is everything in @alchemy/aa-alchemy/react + @alchemy/aa-alchemy/tailwind which enables the UI components
@account-kit/state -- this is everything that used to be in @alchemy/aa-alchemy/config which is basically the underlying storage layer that hooks are built on. This package would allow any dev to essentially rebuild the hooks in @account-kit/react

TODO in a follow up PR:

split the AlchemySigner out of infra into its own package
take the create[SmartAccountType]AlchemyClient methods out of the infra and move them into smart-accounts


Update all poetry.lock dependencies

Dependabot seems to only update the dependencies listed in pyproject.toml, and not the ones in poetry.lock.
For example, this repo is configured to check for Python package updates daily, and last checked 7 hours ago. But when I run poetry update a whole bunch of packages are updated:
$ poetry update
Updating dependencies
Resolving dependencies... (11.7s)

Writing lock file

Package operations: 0 installs, 14 updates, 0 removals

  • Updating six (1.15.0 -> 1.16.0)
  • Updating distlib (0.3.1 -> 0.3.2)
  • Updating lazy-object-proxy (1.4.3 -> 1.6.0)
  • Updating attrs (20.2.0 -> 21.2.0)
  • Updating identify (1.5.9 -> 1.6.2)
  • Updating nodeenv (1.5.0 -> 1.6.0)
  • Updating packaging (20.4 -> 21.0)
  • Updating pathspec (0.8.0 -> 0.8.1)
  • Updating py (1.9.0 -> 1.10.0)
  • Updating pyyaml (5.3.1 -> 5.4.1)
  • Updating regex (2020.10.28 -> 2021.7.6)
  • Updating typed-ast (1.4.1 -> 1.4.3)
  • Updating typing-extensions (<IP_ADDRESS> -> <IP_ADDRESS>)
  • Updating virtualenv (20.1.0 -> 20.4.7)

I would instead expect all the locked dependencies to be kept up to date.

👋 Sorry for the slow response.
Dependabot does support bumping deps in the poetry.lock file as demonstrated by this test (and other tests in this file):
https://github.com/dependabot/dependabot-core/blob/c9df0dd1bfb671589560584c5b162861da0d5ac5/python/spec/dependabot/python/file_updater/poetry_file_updater_spec.rb#L227-L240
However, Dependabot isn't smart enough to unlock multiple deps if multiple packages have to bump simultaneously in order to get past version constraint issues. Ie, if bumping one direct dep would violate constraints in another direct dep, that update will fail. There's a number of tickets to tackle that, #1190 is probably the catch-all, but it's a very tricky problem when you dig into the edge-cases. We've made some progress on it in npm, but not for python.
If you're not seeing pyproject.toml bumps from Dependabot result in poetry.lock bumps, please file a new issue with specific examples and a reproducible public repo.
At this point, it's been nearly two years since the example in the original issue happened... does it still reproduce for single dependencies that aren't held back due to the known multi-constraint problem or should we close this?

Dependabot does support bumping deps in the poetry.lock file as demonstrated by this test (and other tests in this file):

As far as I can tell that test at least is irrelevant for this case, because it checks that requests and pytest are updated when pyproject.toml references both of these. The original issue is that a package listed in poetry.lock and not listed in pyproject.toml, that is, a dependency of one of the packages listed in pyproject.toml, is not being updated. Is there a test where pyproject.toml contains only package X, poetry.lock contains package X and its dependency, package Y, and an update to package Y is updated?

Hei @l0b0!
By default Dependabot only updates direct dependencies. To update indirect dependencies too, you need to explicitly configure it by using
allow:
  - dependency-type: all

See https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#allow.
I'm closing this since I believe this is expected behavior, but let me know if you run into any issues or you disagree.


Document metrics/alerts required for multicluster

What are new metrics or recommended alerts that we may need for multicluster?

There are some metrics that should be provided by MCS API implementation rather than multicluster service mesh. But I'm putting them all together here.
Metrics
In participating cluster

Number of ServiceImport ServiceExport resources. This should be equivalent to number of services imported and exported
Rate of sync request with the broker, group by status
Cross cluster request metrics, group by service, source cluster, destination cluster.

Request rate
Error rate
Latency

For each imported services

number of endpoint slices, by exporting cluster
total number of endpoints

Gateway data transfer rate
Gateway request rate

In broker

Number of joined clusters, group by status
Broker sync request rate

Alerts

Cross cluster request error rate is higher than the threshold, e.g. 1% of total requests or 1 rpm
Cross cluster request latency higher than normal (e.g. recent 5m p99 is higher than 50% of average p99 in the last 24h)
Sync request error rate is higher than the threshold
Cluster in connecting status for too long
Cluster failed to join the multicluster mesh

References
https://submariner.io/operations/monitoring/

ping @steeling for comments here.


add timesync_chrony_custom_settings variable for free-form local configs

Added custom settings for chrony.conf

[citest commit:68e955b5f7499ec9ed605f6b86f23bbc5289aa45]

@mlichvar @pcahyna This seems like a good feature - just wondering if this is the preferred way to handle provider specific settings.

I had some questions about the ntp settings - I think they apply to the chrony settings as well - see https://github.com/linux-system-roles/timesync/pull/54/files#r469487750 and https://github.com/linux-system-roles/timesync/pull/54/files#r469489022

[citest]

travis ci is failing due to yamllint error:
yamllint run-test: commands[0] | yamllint .

./tests/tests_ntp.yml

  35:16     error    trailing spaces  (trailing-spaces)

you can use tox -e yamllint locally to debug this error.  You can use tox -e molecule to also look for ansible lint and other issues.

I had some questions about the ntp settings - I think they apply to the chrony settings as well - see https://github.com/linux-system-roles/timesync/pull/54/files#r469487750 and https://github.com/linux-system-roles/timesync/pull/54/files#r469489022

If user wants to remove a specific setting he just needs to remove it from the list. Custom settings will literally apply any string in the list. In example, something like:
timesync_chrony_custom_settings:
  - "initstepslew 60 pstime1.psi.ch pstime2.psi.ch pstime3.psi.ch"
  - "logdir /var/log/chrony"
  - "log measurements statistics tracking"

will add the above lines at the end of chrony.conf. If we want to remove the logging part:
timesync_chrony_custom_settings:
  - "initstepslew 60 pstime1.psi.ch pstime2.psi.ch pstime3.psi.ch"

This will remove the logdir and log lines. Same if you just want to apply a change to these lines, as soon as one updates the line new line will apply. Same rule will apply here: https://github.com/linux-system-roles/timesync/pull/54 I just wrote something very similar in order to keep coherency. The difference is that on my pull request custom settings are added to the end of the file, will on #54 are added to the top (I really prefer to keep predefined settings on top and custom settings at the end, as it can be a random list of settings). The only problem is that no comments will be added, unless one tries to do something like:
timesync_chrony_custom_settings:
  - "# Allow chronyd to make a rapid measurement of the system and correct clock error at boot time"
  - "initstepslew 60 pstime1.psi.ch pstime2.psi.ch pstime3.psi.ch"
  - "# Select which information is logged."
  - "logdir /var/log/chrony"
  - "log measurements statistics tracking"

The way for adding comments can be improved by enforcing a comment for each new line of settings (it is, instead of a list of strings, a list of hashes with 2 string elements), but in general terms it would really make code more complex and more difficult to understand. In general, predefined settings are in general enough, but on specific use cases one needs to use advanced settings (this is our case for instance, we are managing few datacenters as well user workstations in a complex network setup, so we need to have tools for debugging and optimizing time synchronization).

I had to create a specific test tests_chrony.yml for chrony and leave tests_ntp.yaml as it was in the master branch: looks like for RHEL6-like systems the default provider is NTP, so tests_ntp.yaml with custom settings was failing with the extended chrony settings (as these systems were using NTP by default for this test).

If user wants to remove a specific setting he just needs to remove it from the list.

There is no support for incremental settings.  The list defined in timesync_chrony_custom_settings will replace whatever already exists in the config file.  The template module is idempotent if you run the playbook again with the exact same timesync_chrony_custom_settings.  As far as comments go, I think adding them to timesync_chrony_custom_settings is fine - I don't think any custom handling for comments is necessary.
How to remove custom settings?  For example, if I have already run the role with
timesync_chrony_custom_settings:
  - "initstepslew 60 time1.domain.ch time2.domain.ch time3.domain.ch"
  - "logdir /var/log/chrony"
  - "log measurements statistics tracking"

Now I want to remove all of these settings.  I suppose one way would be to do something like this:
timesync_chrony_custom_settings:
  - "# no custom settings"

which isn't great, but it works.  Any suggestions?

Hi,
If one wants to remove all settings, by do not defining
timesync_chrony_custom_settings would be enough. It is, if variable is not
defined or list length is 0, should write nothing in the config file.
Setting is an empty list by default.
On Thu, 13 Aug 2020, 15:58 Richard Megginson<EMAIL_ADDRESS>wrote:

If user wants to remove a specific setting he just needs to remove it from
the list.
There is no support for incremental settings. The list defined in
timesync_chrony_custom_settings will replace whatever already exists in
the config file. The template module is idempotent if you run the
playbook again with the exact same timesync_chrony_custom_settings. As
far as comments go, I think adding them to timesync_chrony_custom_settings
is fine - I don't think any custom handling for comments is necessary.
How to remove custom settings? For example, if I have already run the role
with
timesync_chrony_custom_settings:

"initstepslew 60 time1.domain.ch time2.domain.ch time3.domain.ch"
"logdir /var/log/chrony"
"log measurements statistics tracking"

Now I want to remove all of these settings. I suppose one way would be to
do something like this:
timesync_chrony_custom_settings:

"# no custom settings"

which isn't great, but it works. Any suggestions?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/linux-system-roles/timesync/pull/80#issuecomment-673495267,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB3IE3AG5JMQMZLY2TKLJHTSAPWQZANCNFSM4P4TNK2A
.

Hi, If one wants to remove all settings, by do not defining timesync_chrony_custom_settings would be enough. It is, if variable is not defined or list length is 0, should write nothing in the config file. Setting is an empty list by default.

OK.  Right, because the entire file would be created from scratch every time the role is run.

Hi, If one wants to remove all settings, by do not defining timesync_chrony_custom_settings would be enough. It is, if variable is not defined or list length is 0, should write nothing in the config file. Setting is an empty list by default.

OK. Right, because the entire file would be created from scratch every time the role is run.

Hi,
below there is an example from switching from this variable configuration:
# linux-system-roles.timesync settings
timesync_chrony_custom_settings:
  - "# Allow chronyd to make a rapid measurement of the system and correct clock error at boot time"
  - "initstepslew 60 pstime1.psi.ch pstime2.psi.ch pstime3.psi.ch"
  - "# Select which information is logged."
  - "logdir /var/log/chrony"
  - "log measurements statistics tracking"
timesync_ntp_provider: chrony
timesync_ntp_servers:
  - hostname: pstime1.psi.ch
  - hostname: pstime2.psi.ch
  - hostname: pstime3.psi.ch

to this configuration (removing custom settings):
# linux-system-roles.timesync settings
# timesync_chrony_custom_settings:
#   - "# Allow chronyd to make a rapid measurement of the system and correct clock error at boot time"
#   - "initstepslew 60 pstime1.psi.ch pstime2.psi.ch pstime3.psi.ch"
#   - "# Select which information is logged."
#   - "logdir /var/log/chrony"
#   - "log measurements statistics tracking"
timesync_ntp_provider: chrony
timesync_ntp_servers:
  - hostname: pstime1.psi.ch
  - hostname: pstime2.psi.ch
  - hostname: pstime3.psi.ch

Is just a diff and looks like is properly removing only that part of the config file. I guess is what you expected, is it correct?
TASK [timesync : Generate chrony.conf file] *******************************************************************************************************************************************************************************************************************************************************************************************************************************************
--- before: /etc/chrony.conf
+++ after: /root/.ansible/tmp/ansible-local-11683pw2yr63/tmpmy0upwh_/chrony.conf.j2
@@ -13,9 +13,3 @@
 # Record the rate at which the system clock gains/losses time.
 driftfile /var/lib/chrony/drift

-# Custom settings
-# Allow chronyd to make a rapid measurement of the system and correct clock error at boot time
-initstepslew 60 pstime1.psi.ch pstime2.psi.ch pstime3.psi.ch
-# Select which information is logged.
-logdir /var/log/chrony
-log measurements statistics tracking

changed: [rhel-8-dev-7a95e9bb.psi.ch]

Otherwise let me know.

Thanks.  Looks good.

[citest]

@mlichvar ok to merge?

Looks good to me. Thanks.

But please squash the commits before merge.

@mcaubet please squash the commits into a single commit

looks like you squashed your commits into a merge commit?  try this instead: https://github.com/wprig/wprig/wiki/How-to-squash-commits

@mcaubet looks like you squashed your commits into a merge commit? try this instead: https://github.com/wprig/wprig/wiki/How-to-squash-commits

Hi, I just resolved some conflicts. I will try to locally squash it (I already did, but my master branch is not in sync with yours). However, I thought squashing commits + merging was possible when merging a pull request, isn't it?

There still was a conflict that I had to resolve. Is it ok now?

There still was a conflict that I had to resolve. Is it ok now?

You still have two commits here.  And one is a merge commit.   Yes, I am not very used to these formalities, but I completely agree with you: is important to keep it right. Thanks for pointing this out. I hope now is fine otherwise let me know.
Best regards and thanks for your help,
Marc

[citest commit:28a71a3ecdeb157501747b5d475ab38b7025609b]


Using docker pennylane-lightning-gpu on GH200

Hello,
I want to test my GH200 (Grace-Hopper by Nvidia) by executing a simulation of a "heavy-weight" quantum algorithm with multiple qubits, and I thought PennyLane-Lightning could be a great tool for this.
I want to run the simulation via a docker, but I a saw you have archived the repo of [pennylane-lightning-gpu].
So what do you recommend I should do?

Hi @giladqm
The repo https://github.com/PennyLaneAI/pennylane-lightning-gpu and all of its contents were migrated to https://github.com/PennyLaneAI/pennylane-lightning so all of the lightning.gpu device components can now be installed from this repository. For aarch64 system, we do not currently provide wheels through PyPI, but there are multiple options to install the package (see https://pennylane.ai/install/#high-performance-computing-and-gpus for more details)

Install from Conda-Forge: lightning.gpu for aarch64 is available through Conda-Forge asconda install pennylane-lightning-gpu

Though, this package is provided mostly with community support, and will likely not support all other standard devices (e.g. you'll need to manually install lightning.qubit and other packages too).
Build from source: this should work from the docker container you are also discussing here, assuming you have nvcc and supporting libraries installed. For simplicity, we can first install pennylane and pennylane-lightning, and all required build dependencies, then build and install pennylane-lightning-gpu as:python -m venv pyenv && source ./pyenv/bin/activate
python -m pip install pennylane
git clone https://github.com/PennyLaneAI/pennylane-lightning --branch latest_release --single-branch
cd pennylane-lightning

# requirements-dev.txt does not have wheels for all packages so we can explicitly list these out
python -m pip install cmake ninja custatevec_cu12 pip~=22.0
PL_BACKEND="lightning_gpu" python -m pip install . --verbose

The package should build and install a natively built version of the libraries into your python environment.

Feel free to let us know if the above doesn't work. Shipping PyPI wheels for aarch64 is on our roadmap, but we have no current timeline to provide yet.

Hi @giladqm
Are you running this on the Nvidia cuQuantum appliance docker image?
I have verified this with docker run --platform aarch64 --rm -it nvcr.io/nvidia/cuquantum-appliance:24.03-arm64 to spin up the container locally, and then  I have run exactly:
python -m venv pyenv && source ./pyenv/bin/activate
python -m pip install pennylane
git clone https://github.com/PennyLaneAI/pennylane-lightning --branch latest_release --single-branch
cd pennylane-lightning

# requirements-dev.txt does not have wheels for all packages so we can explicitly list these out
python -m pip install cmake ninja custatevec_cu12 pip~=22.0
PL_BACKEND="lightning_gpu" python -m pip install . --verbose

and the installation completes successfully. Is there some other environment or custom modifications you have made to your environment, or are you working on a different container image than nvcr.io/nvidia/cuquantum-appliance:24.03-arm64?
If you are using a different env, there may be missing packages --- in this instance, it looks like setuptools isn't available in your environment. I'd recommend installing this package, since it is likely the cause of the failure in your env. Let us know if this helps.

I followed you instructions and indeed the installation was successful.
Unfortunately it looks like the GPU isn't being used.
This is the code I'm running:
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"  # Specify the index of the GPU you want to use

import time
import logging
import torch
import pennylane as qml
from matplotlib import pyplot as plt
from pennylane import numpy as np

# Define the directory to save outputs
output_dir = "output"
if not os.path.exists(output_dir):
    os.makedirs(output_dir)

# Set up logging
log_file_path = os.path.join(output_dir, "output_log.txt")
logging.basicConfig(filename=log_file_path, level=logging.INFO,
                    format='%(asctime)s %(message)s', filemode='w')
console = logging.StreamHandler()
console.setLevel(logging.INFO)
formatter = logging.Formatter('%(asctime)s %(message)s')
console.setFormatter(formatter)
logging.getLogger('').addHandler(console)

def circuit0_basic(params, wires):
    n_qubits = len(wires)
    n_rotations = len(params)
    if n_rotations > 1:
        n_layers = n_rotations // n_qubits
        n_extra_rots = n_rotations - n_layers * n_qubits

        for layer_idx in range(n_layers):
            layer_params = params[layer_idx * n_qubits: layer_idx * n_qubits + n_qubits, :]
            qml.broadcast(qml.Rot, wires, pattern="single", parameters=layer_params)
            qml.broadcast(qml.CNOT, wires, pattern="ring")

        extra_params = params[-n_extra_rots:, :]
        extra_wires = wires[: n_qubits - 1 - n_extra_rots: -1]
        qml.broadcast(qml.Rot, extra_wires, pattern="single", parameters=extra_params)
    else:
        qml.Rot(*params[0], wires=wires[0])

def circuit0(params, wires):
    n_qubits = len(wires)
    n_rotations = len(params)

    if n_rotations > 1:
        n_layers = n_rotations // n_qubits
        n_extra_rots = n_rotations - n_layers * n_qubits

        for layer_idx in range(n_layers):
            layer_params = params[layer_idx * n_qubits: layer_idx * n_qubits + n_qubits, :]
            qml.broadcast(qml.Rot, wires, pattern="single", parameters=layer_params)
            qml.broadcast(qml.CNOT, wires, pattern="ring")

        extra_params = params[-n_extra_rots:, :]
        extra_wires = wires[: n_qubits - 1 - n_extra_rots: -1]
        qml.broadcast(qml.Rot, extra_wires, pattern="single", parameters=extra_params)
    else:
        qml.Rot(*params[0], wires[wires[0]])

def runtest(H, cfg, show=False):
    max_iterations = cfg.max_iterations
    num_qubits = len(H.wires)
    num_param_sets = (2 ** num_qubits) - 1

    params = np.random.uniform(low=-np.pi / 2, high=np.pi / 2, size=(num_param_sets, 3))
    params = np.array(params, requires_grad=True)

    dev = qml.device("lightning.gpu", wires=num_qubits, batch_obs=1)
    logging.info(f"Using device: {dev}")

    @qml.qnode(dev, interface='autograd')
    def cost_fn(params):
        circuit0_basic(params, wires=H.wires)
        return qml.expval(H)

    @qml.qnode(dev, interface='autograd')
    def state_fn(params):
        circuit0_basic(params, wires=H.wires)
        return qml.state()

    # Check if GPU is available and log it after device is set
    if torch.cuda.is_available():
        logging.info("CUDA is available. Using GPU.")
    else:
        logging.info("CUDA is not available. Using CPU.")
        logging.info("CUDA check details:")
        logging.info(f"CUDA available: {torch.cuda.is_available()}")
        logging.info(f"CUDA device count: {torch.cuda.device_count()}")
        if torch.cuda.is_available():
            logging.info(f"CUDA device name: {torch.cuda.get_device_name(0)}")

    opt = qml.AdamOptimizer(stepsize=0.1)
    conv_tol = 1e-06
    energy_plot = []
    prev_energy = cost_fn(params)

    for n in range(max_iterations):
        params, energy = opt.step_and_cost(cost_fn, params)
        logging.info(f"Iteration {n + 1}/{max_iterations}: Energy = {energy:.6f}")
        energy_plot.append(energy)

        if np.abs(energy - prev_energy) <= conv_tol:
            logging.info("Convergence reached!")
            break
        prev_energy = energy

        # Print progress
        progress = (n + 1) / max_iterations * 100
        logging.info(f"Progress: {progress:.2f}%")

    logging.info("Optimization completed.")

    # Using the params, find the ground state vector
    best_params = params
    ground_state = state_fn(best_params)

    # Plot energies
    plt.clf()
    plt.plot(energy_plot)
    plt.xlabel("Iterations")
    plt.ylabel("Energy")
    plt.title("Energy at each iteration")
    energy_plot_path = os.path.join(output_dir, "energy_plot.png")
    plt.savefig(energy_plot_path)
    if show:
        plt.show()

    return energy, ground_state

class Params0:
    pass

class Hamiltonian:

    def __init__(self, N):
        self.ham = qml.Hamiltonian([], [])
        self.energies = None
        self.states = None
        self.gs_energy = None
        self.gs_state = None

        self.N = N  # Number of qubits

class MyHamiltonian0(Hamiltonian):

    def __init__(self, N, A, b, P=1, flag0=False):
        n = N // P
        super().__init__(N)
        self.flag0 = flag0
        self.P = P
        self.n = n
        self.A = A
        self.b = b
        self.y = self.A @ self.b.reshape(-1, 1)
        self.y = np.matrix(self.y)
        self.set_hamiltonian()

    def get_evec(self):
        return self.noise_std * np.random.normal(size=self.n, requires_grad=False)

    def set_hamiltonian(self):

        def func1():
            param1, param2 = self.A.shape
            w1 = np.zeros((param2, param2))
            w2 = np.zeros(param2)
            for m in range(param1):
                for i in range(param2):
                    w2[i] += -2 * self.A[m, i] * self.y[m, 0]
                    for j in range(param2):
                        w1[i, j] += self.A[m, i] * self.A[m, j]
            return w1, w2

        def func2():
            w1, w2 = func1()
            min_val = -2 ** (self.P - 1) + 1
            param1, param2 = w1.shape
            v1 = np.zeros((param2 * self.P, param2 * self.P))
            v2 = np.zeros(param2 * self.P)
            for i in range(param1):
                for s in range(self.P):
                    v2[self.P * i + s] += (2 ** s) * w2[i]
                    for j in range(param2):
                        v2[self.P * i + s] += (2 ** s) * 2 * min_val * w1[i, j]
                        for p in range(self.P):
                            v1[self.P * i + s, self.P * j + p] += (2 ** (s + p)) * w1[i, j]
            return v1, v2

        v1, v2 = func2()

        H = qml.Hamiltonian([], [])
        for i in range(self.n):
            for s in range(self.P):
                xadded = False
                x = i * self.P + s
                fact = - (sum(v1[x, :]) + sum(v1[:, x]) + 2*v2[x])
                if fact != 0:
                    xadded = True
                    H += fact * qml.PauliZ(x)
                for j in range(self.n):
                    for p in range(self.P):
                        y = j*self.P + p
                        fact = v1[x, y]
                        if fact != 0:
                            xadded = True
                            H += fact * qml.PauliZ(x) @ qml.PauliZ(y)
                if not xadded:
                    H += 0.0 * qml.PauliZ(x)

            if self.flag0:
                x = i * self.P
                H += -(1/2)*qml.PauliZ(x) + (1/4)*qml.PauliZ(x) @ qml.PauliZ(x)

        self.ham = H

def main():
    tstart = time.time()

    logging.info('Running experiment0')
    cfg = Params0()
    cfg.ensemble_num = 10
    P = 1
    n = 21
    cfg.num_of_qubits = n * P
    cfg.max_iterations = 10

    cfg.experiment_name = 'experiment0'
    cfg.hamiltonian_type = 'Hamiltonian0'

    A = np.random.randn(n, n)
    b = np.random.randint(2 ** P, size=n, requires_grad=False)
    flag0 = False

    res_vec = []
    H = MyHamiltonian0(cfg.num_of_qubits, A=A, b=b, P=P, flag0=flag0)
    logging.info("Starting the optimization...")
    gs_energy, __ = runtest(H.ham, cfg, show=False)
    res_vec.append(gs_energy)
    logging.info("Optimization finished.")
    logging.info("TIME= %d [sec]", int(np.round((time.time() - tstart))))
    logging.info("Ground state energy: %s", res_vec)
    return
    # -------------------------------------

if __name__ == '__main__':
    main()

and the output:
(pyenv) (base) cuquantum@7806514dd020:~$ python Gilad_Test.py
2024-05-28 05:48:21,924 Running experiment0
2024-05-28 05:48:26,198 Starting the optimization...
2024-05-28 05:48:26,788 Using device: Lightning GPU PennyLane plugin
Short name: lightning.gpu
Package: pennylane_lightning
Plugin version: 0.36.0
Author: Xanadu Inc.
Wires: 21
Shots: None
2024-05-28 05:48:26,789 CUDA is not available. Using CPU.
2024-05-28 05:48:26,790 CUDA check details:
2024-05-28 05:48:26,790 CUDA available: False
2024-05-28 05:48:26,790 CUDA device count: 0

I built the torch from source and now I get:
(cuquantum-24.03) cuquantum@7806514dd020:~$ python Gilad_Test.py
2024-05-28 07:11:14,614 Running experiment0
2024-05-28 07:11:18,819 Starting the optimization...
2024-05-28 07:11:19,331 Using device: Lightning GPU PennyLane plugin
Short name: lightning.gpu
Package: pennylane_lightning
Plugin version: 0.36.0
Author: Xanadu Inc.
Wires: 21
Shots: None
2024-05-28 07:11:19,332 CUDA is available. Using GPU.

But it doesn't seem the it's really working...

I tried also updating the code:
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"  # Specify the index of the GPU you want to use

import time
import logging
import torch
import pennylane as qml
from matplotlib import pyplot as plt
from pennylane import numpy as np

# Define the directory to save outputs
output_dir = "output"
if not os.path.exists(output_dir):
    os.makedirs(output_dir)

# Set up logging
log_file_path = os.path.join(output_dir, "output_log.txt")
logging.basicConfig(filename=log_file_path, level=logging.INFO,
                    format='%(asctime)s %(message)s', filemode='w')
console = logging.StreamHandler()
console.setLevel(logging.INFO)
formatter = logging.Formatter('%(asctime)s %(message)s')
console.setFormatter(formatter)
logging.getLogger('').addHandler(console)

def circuit0_basic(params, wires):
    n_qubits = len(wires)
    n_rotations = len(params)
    if n_rotations > 1:
        n_layers = n_rotations // n_qubits
        n_extra_rots = n_rotations - n_layers * n_qubits

        for layer_idx in range(n_layers):
            layer_params = params[layer_idx * n_qubits: layer_idx * n_qubits + n_qubits, :]
            qml.broadcast(qml.Rot, wires, pattern="single", parameters=layer_params)
            qml.broadcast(qml.CNOT, wires, pattern="ring")

        extra_params = params[-n_extra_rots:, :]
        extra_wires = wires[: n_qubits - 1 - n_extra_rots: -1]
        qml.broadcast(qml.Rot, extra_wires, pattern="single", parameters=extra_params)
    else:
        qml.Rot(*params[0], wires=wires[0])

def circuit0(params, wires):
    n_qubits = len(wires)
    n_rotations = len(params)

    if n_rotations > 1:
        n_layers = n_rotations // n_qubits
        n_extra_rots = n_rotations - n_layers * n_qubits

        for layer_idx in range(n_layers):
            layer_params = params[layer_idx * n_qubits: layer_idx * n_qubits + n_qubits, :]
            qml.broadcast(qml.Rot, wires, pattern="single", parameters=layer_params)
            qml.broadcast(qml.CNOT, wires, pattern="ring")

        extra_params = params[-n_extra_rots:, :]
        extra_wires = wires[: n_qubits - 1 - n_extra_rots: -1]
        qml.broadcast(qml.Rot, extra_wires, pattern="single", parameters=extra_params)
    else:
        qml.Rot(*params[0], wires[wires[0]])

def runtest(H, cfg, show=False):
    max_iterations = cfg.max_iterations
    num_qubits = len(H.wires)
    num_param_sets = (2 ** num_qubits) - 1

    # Initialize parameters directly on the GPU
    params = torch.tensor(np.random.uniform(low=-np.pi / 2, high=np.pi / 2, size=(num_param_sets, 3)), requires_grad=True, device='cuda', dtype=torch.float32)

    dev = qml.device("lightning.gpu", wires=num_qubits, batch_obs=True)
    logging.info(f"Using device: {dev}")

    @qml.qnode(dev, interface='torch')
    def cost_fn(params):
        circuit0_basic(params, wires=H.wires)
        return qml.expval(H)

    @qml.qnode(dev, interface='torch')
    def state_fn(params):
        circuit0_basic(params, wires=H.wires)
        return qml.state()

    # Check if GPU is available and log it after device is set
    if torch.cuda.is_available():
        logging.info("CUDA is available. Using GPU.")
    else:
        logging.info("CUDA is not available. Using CPU.")
        logging.info("CUDA check details:")
        logging.info(f"CUDA available: {torch.cuda.is_available()}")
        logging.info(f"CUDA device count: {torch.cuda.device_count()}")
        if torch.cuda.is_available():
            logging.info(f"CUDA device name: {torch.cuda.get_device_name(0)}")

    opt = torch.optim.Adam([params], lr=0.1)
    conv_tol = 1e-06
    energy_plot = []
    prev_energy = cost_fn(params).item()

    for n in range(max_iterations):
        opt.zero_grad()
        energy = cost_fn(params)
        energy.backward()
        opt.step()
        energy = energy.item()
        logging.info(f"Iteration {n + 1}/{max_iterations}: Energy = {energy:.6f}")
        energy_plot.append(energy)

        if np.abs(energy - prev_energy) <= conv_tol:
            logging.info("Convergence reached!")
            break
        prev_energy = energy

        # Print progress
        progress = (n + 1) / max_iterations * 100
        logging.info(f"Progress: {progress:.2f}%")

    logging.info("Optimization completed.")

    # Using the params, find the ground state vector
    best_params = params
    ground_state = state_fn(best_params)

    # Plot energies
    plt.clf()
    plt.plot(energy_plot)
    plt.xlabel("Iterations")
    plt.ylabel("Energy")
    plt.title("Energy at each iteration")
    energy_plot_path = os.path.join(output_dir, "energy_plot.png")
    plt.savefig(energy_plot_path)
    if show:
        plt.show()

    return energy, ground_state

class Params0:
    pass

class Hamiltonian:

    def __init__(self, N):
        self.ham = qml.Hamiltonian([], [])
        self.energies = None
        self.states = None
        self.gs_energy = None
        self.gs_state = None

        self.N = N  # Number of qubits

class MyHamiltonian0(Hamiltonian):

    def __init__(self, N, A, b, P=1, flag0=False):
        n = N // P
        super().__init__(N)
        self.flag0 = flag0
        self.P = P
        self.n = n
        self.A = A.to(torch.float32).to('cuda')  # Move to GPU and ensure float32 dtype
        self.b = b.to(torch.float32).to('cuda')  # Move to GPU and ensure float32 dtype
        self.y = self.A @ self.b.reshape(-1, 1)
        self.y = self.y.clone().detach().requires_grad_(True)  # Properly construct tensor from existing tensor
        self.set_hamiltonian()

    def get_evec(self):
        return self.noise_std * torch.normal(mean=0, std=1, size=(self.n,), device='cuda', requires_grad=False)

    def set_hamiltonian(self):

        def func1():
            param1, param2 = self.A.shape
            w1 = torch.zeros((param2, param2), device='cuda', dtype=torch.float32)
            w2 = torch.zeros(param2, device='cuda', dtype=torch.float32)
            for m in range(param1):
                for i in range(param2):
                    w2[i] += -2 * self.A[m, i] * self.y[m, 0]
                    for j in range(param2):
                        w1[i, j] += self.A[m, i] * self.A[m, j]
            return w1, w2

        def func2():
            w1, w2 = func1()
            min_val = -2 ** (self.P - 1) + 1
            param1, param2 = w1.shape
            v1 = torch.zeros((param2 * self.P, param2 * self.P), device='cuda', dtype=torch.float32)
            v2 = torch.zeros(param2 * self.P, device='cuda', dtype=torch.float32)
            for i in range(param1):
                for s in range(self.P):
                    v2[self.P * i + s] += (2 ** s) * w2[i]
                    for j in range(param2):
                        v2[self.P * i + s] += (2 ** s) * 2 * min_val * w1[i, j]
                        for p in range(self.P):
                            v1[self.P * i + s, self.P * j + p] += (2 ** (s + p)) * w1[i, j]
            return v1, v2

        v1, v2 = func2()

        H = qml.Hamiltonian([], [])
        for i in range(self.n):
            for s in range(self.P):
                xadded = False
                x = i * self.P + s
                fact = - (sum(v1[x, :]) + sum(v1[:, x]) + 2*v2[x])
                if fact != 0:
                    xadded = True
                    H += fact * qml.PauliZ(x)
                for j in range(self.n):
                    for p in range(self.P):
                        y = j*self.P + p
                        fact = v1[x, y]
                        if fact != 0:
                            xadded = True
                            H += fact * qml.PauliZ(x) @ qml.PauliZ(y)
                if not xadded:
                    H += 0.0 * qml.PauliZ(x)

            if self.flag0:
                x = i * self.P
                H += -(1/2)*qml.PauliZ(x) + (1/4)*qml.PauliZ(x) @ qml.PauliZ(x)

        self.ham = H

def main():
    tstart = time.time()

    logging.info('Running experiment0')
    cfg = Params0()
    cfg.ensemble_num = 10
    P = 1
    n = 21  # Increase number of qubits
    cfg.num_of_qubits = n * P
    cfg.max_iterations = 100  # Increase the number of iterations for better GPU utilization

    cfg.experiment_name = 'experiment0'
    cfg.hamiltonian_type = 'Hamiltonian0'

    A = torch.randn(n, n, device='cuda', dtype=torch.float32)  # Move to GPU and ensure float32 dtype
    b = torch.randint(2 ** P, size=(n,), device='cuda', dtype=torch.float32, requires_grad=False)  # Move to GPU and ensure float32 dtype
    flag0 = False

    res_vec = []
    H = MyHamiltonian0(cfg.num_of_qubits, A=A, b=b, P=P, flag0=flag0)
    logging.info("Starting the optimization...")
    gs_energy, __ = runtest(H.ham, cfg, show=False)
    res_vec.append(gs_energy)
    logging.info("Optimization finished.")
    logging.info("TIME= %d [sec]", int(np.round((time.time() - tstart))))
    logging.info("Ground state energy: %s", res_vec)
    return
    # -------------------------------------

if __name__ == '__main__':
    main()

But the GPU Memory Usage is very low:

I think this is a MIG issue, trying to figure it out.

If you try swapping the device for default.qubit and use PyTorch with CUDA-mapped tensors, does the GPU work? You can likely pick a smaller scale workload for this (e.g. something from the Torch GPU tests at https://github.com/PennyLaneAI/pennylane/blob/59a1e0586e707d057a0c92d4239036afa5312b73/tests/interfaces/test_torch.py#L399).
If this runs on the GPU without issue, it may be a runtime issue with LGPU. If not, then most likely the MIG/some CUDA driver issue on the node.

Thanks @CatalinaAlbornoz.
One thing I find really weird is that for the same code where n=7 ,default.qubit takes 4 seconds but with lightning.gpu it takes 100-140 seconds. What is the reason for this?

Hi @giladqm, lightning.gpu is optimized to work with over 20 qubits. There's a big overhead in spinning up all of the processes needing and passing the information from CPU to GPU and viceversa. So for smaller circuits default.qubit or lightning.qubit will work better.
If you go to pennylane.ai/performance you'll notice that at the end of the page we have a table to help you choose between the simulators depending on your circuit. Definitely take a look at this page, you may find some new insights. 😃

I appreciate the explanation, Thank you @CatalinaAlbornoz

@CatalinaAlbornoz is there somewhere I can read more about the types of devices?