Coordinate running calculations

Are we ready to start submitting calculations?  We have three molecule collections ready (solvated amino acids, dipeptides, and DES370K).  We have agreement on the level of theory to use and what quantities to compute.  I think that means we're ready to put together a draft of the first submission?
The solvated amino acids are probably the best one to start with.  It's a very small collection (only 1300 conformations), although they're individually on the large side (about 75-95 atoms).

CCing @pavankum @dotsdl @simonboothroyd @trevorgokey to see who might have available bandwidth to submit these to QCFractal.

I can help with preparing the submissions, @jthorton already made a template so I can start from that. Just to clarify, these are all optimization datasets, right? Or single point calculations?
Another question is about the name to prepend to every dataset, such as ML Dipeptides, ML Solvated Amino Acids, etc. @peastman already posed it here #9, it would be great to add that acronym in the dataset names.

These are single point calculations.  For each one we want to compute the energy and forces, as well as the quantities listed in https://github.com/openmm/qmdataset/issues/7#issuecomment-926209812.  Also the orbital coefficients and eigenvalues, if the storage requirements aren't prohibitive.

Maybe we can compute some with GPUGRID as well.
g
On Fri, Oct 1, 2021 at 4:02 AM Peter Eastman @.***>
wrote:

These are single point calculations. For each one we want to compute the
energy and forces, as well as the quantities listed in #7 (comment)
https://github.com/openmm/qmdataset/issues/7#issuecomment-926209812.
Also the orbital coefficients and eigenvalues, if the storage requirements
aren't prohibitive.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/openmm/qmdataset/issues/11#issuecomment-931834488,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB3KUOWDZLZORTY3U36RF7DUEUJEXANCNFSM5FC6N7RA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Is there anything we can do to get this going?  @pavankum you said you were waiting on a new release of QCFractal.  Is that release expected soon?  I notice they seem to do infrequent releases.  The last one was in June, and the one before that was November of last year.  If we're just waiting for the next regularly scheduled release, this project could be stuck on hold for months.

I already pinged Ben Pritchard from Molssi ten days ago on openff qcfractal channel and offered help, and he said he would push to make a release last week, maybe @jchodera can inquire about the current status.

Great, thanks!

MolSSI just had a major advisory board meeting in Washington DC. That just concluded yesterday.
Now is probably a good time to inquire about status.

@pavankum : It looks like @bennybp released QCFractal 0.15.7 and QCPortal 0.15.7, and the QCArchive server was upgraded yesterday, so you may be clear to proceed now.
When submitting these conformer datasets, can we also submit a standard OptimizationDataset alongside them as well starting from the list of molecule SMILES? That will help us establish which conformer generation scheme(s) are actually useful, since it's not clear which approach provides the most efficient use of computational resources at this point.

@jchodera Sure, will make the submission PRs today.

@peastman I got the ball rolling on submissions, made the PRs to qca-dataset-submission repo (pubchem sets - 243, 245, 246, 247, 248, 249, des370k - 244, solvated amino acids - 239, dipeptides - 251) and will push to compute once they get reviewed by David Dotson/Josh Horton.
I used SPICE as a placeholder name, is there a consensus on the naming convention? For example, a subset of pubchem molecules (2501-5000) is named as "SPICE PubChem Set 2 Single Points Dataset", does this look okay?
@jchodera do you want optimization datasets for a particular subset for comparison, or for all of these sets?

@pavankum: Thank you so much!

I used SPICE as a placeholder name, is there a consensus on the naming convention? For example, a subset of pubchem molecules (2501-5000) is named as "SPICE PubChem Set 2 Single Points Dataset", does this look okay?

I think this means you officially get to name this dataset since you did the submission work. :)

@jchodera do you want optimization datasets for a particular subset for comparison, or for all of these sets?

Let's just do dipeptides (251) for now---that should provide an excellent comparison set without adding much to the compute burden.
@peastman : I notice we skipped a datast of monomers extracted from des370k---could we add those in with higher priority than the dimers? All you have to do is extract the unique set of monomers and conformations.
Also, I notice that many of those PubChem sets look completely nuts:

I'll prioritize trying to get approval to redistribute the other datasets we discussed under nonrestrictive licenses.

Let's just do dipeptides (251) for now---that should provide an excellent comparison set without adding much to the compute burden.

@jchodera Thank you for the feedback, will add that to the submission list

I think this means you officially get to name this dataset since you did the submission work. :)

Congratulations on your excellent choice of name!

Also, I notice that many of those PubChem sets look completely nuts:

There are definitely some odd ones in there.  This is partly due to how I ordered the molecules: it tries to choose ones that are maximally different from anything that has come before, so if a molecule is really unusual, it gets put very early in the dataset.  If you only look at the first few pages of the first set, you'll get the impression this collection is full of things that don't look much like drugs, but it quickly settles down into much more ordinary ones.

I notice we skipped a dataset of monomers extracted from des370k

Is there any reason to think they'll be useful?  Back when I was trying to train models on DES370K and nothing else, I found that training just on dimers was difficult and adding in a few monomer conformations helped it to learn.  But in this case it already has tons of data for single molecules.  A few hundred extra molecules isn't likely to make much difference.

Is there any reason to think they'll be useful? Back when I was trying to train models on DES370K and nothing else, I found that training just on dimers was difficult and adding in a few monomer conformations helped it to learn. But in this case it already has tons of data for single molecules. A few hundred extra molecules isn't likely to make much difference.

It's very quick to run
It's a minimal set covering a lot of biomolecular chemical space
Many tools aren't yet set up to run on multiple molecules (e.g. espaloma, for building MM force fields)
There's issues like BSSE that mean the monomer set is actually a bit different than well-separated molecules
We'll easily be able to compare different strategies for generating conformers on this small standard dataset
As you point out, it really helps learning to separate intermolecular from intramolecular interactions to include monomers

Hey all, we are currently executing SPICE PubChem Set 1 Single Points Dataset v1.1. Based on the growth rate of storage usage on QCArchive, at about 5MiB per calculation with wavefunction stored, we will go beyond QCArchive's storage capacity if we proceed in this way with the other 5 PubChem sets.
Is it known now whether or not wavefunctions (orbitals and eigenvalues) will be needed for the downstream use case of these datasets? If this is not known, can we begin using set 1 for that downstream case to arrive at a decision?
If wavefunctions are not needed at all, we can switch off wavefunction storage for the remaining 5 and proceed immediately. If wavefunctions are or may be required, we will need at least a 5TiB storage expansion of some kind on QCArchive.

Wavefunctions aren't needed for any of the applications I'm interested in.  I think the argument for saving them was that it could save time if we later decided we wanted to compute additional quantities, or redo the computation with a more accurate method (https://github.com/openmm/qmdataset/issues/7#issuecomment-924970376).  But if it causes storage problems, I don't think it's necessary.

Thanks for clarifying, @peastman!
@dotsdl : Since the PubChem Set 1 is only 10% done after 4-5 days of compute, maybe it makes sense to purge the dataset and start over without wavefunctions? 5 MiB x 11K calculations is 55 GiB of data that is probably unnecessary, and the dataset would eventually consume 595 GiB for no reason.

@dotsdl : One other quick question: Are the molecules sorted in order of increasing size? It may make sense to do so, since this would allow the highest throughput initially, enabling us to catch other issues earlier (rather than later) in dataset generation.

@peastman @jchodera A naive question, I am wondering whether Orbnet used any orbital information in their model, and you think that might be something relevant to your ML models as well, apart from forces and energies

@pavankum : It's a great question! I don't doubt that information derived from wavefunction/orbital data would be valuable in training advanced machine learning potentials, but I don't believe any of the architectures we are considering now would make use of this information.
@jeherr: This is a really interesting idea---something to think about!

I am wondering whether Orbnet used any orbital information in their model

Not from the training data, if that's what you mean.  Their model begins with a semi-empirical calculation, and the outputs of it become the input to the model.  So in that sense, it likely does involve orbital information (I haven't looked at the details to see exactly what values they use).  But the dataset just has energies, no orbitals or even forces.  That's what they fit to.

Thank you very much for the clarifications!!

Thanks all! I just spoke to @bennybp, and I propose we proceed as follows:

We will let SPICE PubChem Set 1 Single Points Dataset v1.1 run as is, with wavefunctions attached. This will allow experimentation with wavefunctions if you are inclined in the near future.
We will submit sets 2 through 6 without wavefunctions attached.

Please let me know if you object to any of this.

If that works for you, go for it!

Where can I find the completed datasets?

Looks like even the SPICE PubChem Set 1 Single Points Dataset v1.1 has a long ways to go:

The data is all available in real time through the QCPortal API---see the example usage here, though I think this dataset is a new BasicDataset type that is not yet shown in the examples. You can use the example code in that notebook to browse and retrieve the available data.
The QCPortal API is not very performant for bulk downloads yet---@dotsdl has been working with @bennybp on speeding this up, and both MolSSI is recruiting a new postdoc and OpenFF is hiring a contractor (we're still searching) to make improvements to this infrastructure. Another goal is to make the data available via monolithic HDF5 files on the machine learning datasets dashboard---this currently has to be prepared by hand.

What about https://github.com/openforcefield/qca-dataset-submission/pull/254?  It claims to be complete.  I just want to look it over to make sure all the data content and organization looks right.

I launched a MyBinder notebook using the OptimizationDataset example and tweaked it a bit:
Browsing the SPICE datasets notebook
Takeaways:

The SPICE DES Monomers Single Points Dataset v1.0 appears as a Dataset, which I thought was the base class type and not a specific single-point dataset type
Browsing the collection for this datasest appears to give an empty dataframe
None of the methods for browsing or retrieving data seem to work

It's possible these issues are caused by the mybinder image being out of date (it has QCPortal v0.15.7), but we're going to need some help from @pavankum @bennybp here.

@peastman and @jchodera, putting together a code snippet for how to access the data elements in each dataset. Will post here today.

@dotsdl: If you want to do this within a Jupyter notebook, it can probably be added as a PR directly into the QCPortal examples folder so that others can benefit from it immediately!

Can do, and agreed. Access of Dataset records are woefully undocumented and those dataset types in particular are very user-unfriendly. @bennyp and I are actively working to remedy this.

Sorry for the delay in response, here's one small snippet to look at data from a basic dataset
import qcportal

client = qcportal.FractalClient()
ds = client.get_collection(collection_type='Dataset', name='SPICE PubChem Set 1 Single Points Dataset v1.1')
print(ds.list_values())
method = 'wb97m-d3bj'
basis = 'def2-tzvppd'

# dataframe of records, this may take a while depending on the size of dataset
df = ds.get_records(method=method, basis=basis)

print(len(df))

for i in range(len(df)):
     if not isinstance(df.iloc[i].record, float):
             if df.iloc[i].record.status == 'COMPLETE':
                     rec = df.iloc[i]
                     break

print(rec.dict())

@dotsdl may add more

@peastman you can use this jupyter notebook as a starting point: Using single point Datasets.zip
Please start with SPICE PubChem Set 1 Single Points Dataset v1.1 for your experiments; where present, you'll want to use the v1.1 version of datasets on QCArchive, since these include the wcombine=False fix.

@pavankum from experience you should specify program and keywords like I do in the notebook, otherwise it is possible to get records that aren't part of the dataset submission back.

I've added this example as a PR to MolSSI/QCFractal#703. I'll capture any feedback on this there to expand its usefulness before merge.

Also, I notice that many of those PubChem sets look completely nuts:

Odd molecules like this are good for regularization of trained models. In general, it's good to try not to target anything specific too much, and sometimes these molecules can be really information rich when you consider how much the other data all shares overly common substructures ad infinitum.

This is a really interesting idea---something to think about!

I feel like I have seen some work along these lines before, but I can't recall anything right now. I'll try to look around and see what I can find.

Should've done this earlier, here is a list of datasets with their updated names and completion status of each. Right now the only bottleneck is compute resources since openff datasets are prioritized over these datasets and if you would like bump the priority of any single set, or order them in priority please let us know. If you're not satisfied with the pace of computation it would be great if you can contribute any compute resources, we would be happy to help you set up the qcfractal compute managers specific to these datasets. I can also keep updating this table (here or in a different issue) every few days so that you can catch up on the progress easily.

Dataset name
Complete
Remaining
Comments

SPICE DES Monomers Single Points Dataset v1.1
18700
-
COMPLETE

 SPICE Solvated Amino Acids Single Points Dataset v1.1
1291
9
Almost there!

SPICE Dipeptides Single Points Dataset v1.1
57
33793
Running now

SPICE Dipeptides Optimization Dataset v1.0
610
32717
Low priority full optimization set

SPICE PubChem Set 1 Single Points Dataset v1.1
42802
75804
Running now

SPICE PubChem Set 2 Single Points Dataset v1.0
0
121540
In queue

SPICE PubChem Set 3 Single Points Dataset v1.0
0
122226
In queue

SPICE Pubchem Set 4 Single Points Dataset v1.0
0
122750
In queue

SPICE Pubchem Set 5 Single Points Dataset v1.0
0
123150
In queue

SPICE Pubchem Set 6 Single Points Dataset v1.0
0
123800
In queue

SPICE DES370K Single Points Dataset v1.0
0
345682
In PR review

cc: @dotsdl @jthorton @peastman @jchodera

Thanks for the great summary, @pavankum!

If you're not satisfied with the pace of computation it would be great if you can contribute any compute resources, we would be happy to help you set up the qcfractal compute managers specific to these datasets.

We should be using all available MSK CPU resources for this.
I'll investigate whether something could be set up at Stanford as well.

Thanks!  We'll look into what other resources might be available.

We also might be underutilizing MSK resources at the moment. I'm investigating.

Reposting here from OpenFF slack:
@jchodera  we are currently limiting our usage of Lilac to avoid filling disks on the compute nodes. I worked with @bennybp on Friday to devise a solution for manager cleanup on termination that is both reliable and acceptable in what it requires from QCEngine. Apologies this has taken so long, but there are several process boundaries and software layers to cross here. There are many poor solutions that fail to fully solve the problem, and so it has taken time to converge on one that does.
I am implementing what we devised on Friday today in MolSSI/QCFractal#700.
This issue on Lilac wasn't apparent in the past due to our use of a basis set that didn't sometimes require up to 250GiB of memory and/or ~70GiB of scratch space from the compute nodes they landed on. The SPICE datasets have challenged us in ways that are new.

Thanks so much for the detailed update, @dotsdl, and glad to hear that progress is being made so we can make use of all available resources soon!

I've gotten an account on a cluster at Stanford.  Can you provide instructions on what I need to do to start running calculations on it?

@peastman I think we should meet up for a working session to do this. The best approach depends on the configuration of the cluster, and we'll be able to arrive at this quickly in an interactive call.

It's now running on five nodes, each with 32 cores and 256 GB memory.  According to the logs, it has completed about 30 tasks so far.  Is there a way to confirm you're receiving the results?

@peastman with the information you just shared, that is sufficient. The manager would complain loudly if it wasn't able to send results back to public QCArchive.
Thanks again for your help with this!

That's great! As long as the logs show the task submission and success rate you're good. I think it's difficult to monitor the jobs from different servers (fractal-managers of a certain kind, lilac or pacific-research-platform or sherlock ...). A bit tedious way is to check dataset by dataset and grabbing the manager_name metadata in the result record. Since these are huge datasets querying for the records takes quite a bit of time (half an hour to one hour or more for each), among the currently computing sets I checked jobs associated with a manager for pubchem sets 1, 2 & 3, and the dipeptide single points, and I can see pubchem set 2 has jobs on sherlock, assuming that's the stanford cluster
Pubchem set 1: ({'LilacQM': 13082, 'PacificResearchPlatformQM': 30707,  'vulkan': 73, 'NewCastlePsi4': 290, 'tscc': 28})
Pubchem set 2: {'PacificResearchPlatformQM': 92096, 'LilacQM': 5490,  'tscc': 14317, 'NewCastlePsi4': 51,
         'sherlock': 141}
Pubchem set 3: ({'PacificResearchPlatformQM': 26503, 'LilacQM': 2113, 'tscc': 1490})
Dipeptides single points: ({'PacificResearchPlatformQM': 31360, 'tscc': 1348, 'UCI': 1136})

Thanks!  That's really helpful information.
Based on those numbers, it looks like PacificResearchPlatformQM does far more than everything else put together.  But I'm not sure that's really true.  For example, the above says it has done 92,096 calculations for pubchem 2.  But looking up the latest results for that dataset, only 10,449 have actually been completed.  It looks like those numbers mostly reflect it producing a lot of errors very quickly.
It looks like Sherlock is currently completing about 600 calculations a day.  At that rate, it would take it about three years to get through the whole dataset.  I'll see if I can get a few more nodes, but it's still going to be a pretty minor contribution to what needs to be done.  I guess every bit helps though.

But I'm not sure that's really true.

Yeah, I agree, digging into the errors, will report back.

@peastman: The MSK computing resources are heavily underutilized right now because @dotsdl where running more jobs causes havoc for other users by not freeing up scratch space when done. @dotsdl is nearly finished addressing this issue and will then be able to scale up further on the MSK resources.

It looks like Sherlock will let me use eight nodes at a time.  Additional jobs beyond that remain pending in the queue with the status "QOSMaxCpuPerUserLimit".  That should let it complete about 1000 of the PubChem calculations per day.  Once we get to the DES370K dimers, they should go faster since they're smaller.

I checked the errors on PRP with dipeptide single points set (259) and most of them are all similar to an old bug (fixed) with py-cpuinfo, wonder why it is popping up again, I think @dotsdl noticed this before. @dotsdl when you have a moment can you please check the pods on PRP and if we need to pin the version of py-cpuinfo.

Hey all, I made a huge mistake this week. In submitting pubchem sets 2 - 5 through local infrastructure (our Github Action times out after 6 hours; this is our current workaround for such large sets), I failed to pull latest master of qca-dataset-submission that included the submittable dataset.json.bz2s without orbitals_and_eigenvalues set for wavefunction storage. This means all records corresponding to these datasets will carry wavefunctions, and this will be too much for the current storage solution on public QCArchive.
I have parked the computations for these submissions under the openff-defunct compute tag; this will keep them from being computed for now while @pavankum, @bennybp, and I devise a solution early next week. It may be possible to delete these records directly in the DB, allowing us to re-submit with wavefunction: None, but we will have to game this and other options out before we proceed.
 We do have other sets for compute to work on while we solve this problem, and we will try to minimize time lost.

RE: cpuinfo, I've created a PR against QCEngine to address this: https://github.com/MolSSI/QCEngine/pull/339
From our PRP pod logs, there appear to be multiple nodes on which this problem can arise. I will try to mark each of these with anti-affinity to avoid pods landing there, but this is more of a compensating control than anything until we get a fix like the above deployed in QCEngine.

Progress as of 2022-01-18 13:03 UTC

Dataset name
Complete
Remaining
Comments

SPICE DES Monomers Single Points Dataset v1.1
18700
-
COMPLETE

 SPICE Solvated Amino Acids Single Points Dataset v1.1
1300
-
COMPLETE

SPICE Dipeptides Single Points Dataset v1.1
19746
14104
Running now - 58% done

SPICE Dipeptides Optimization Dataset v1.0
610
32717
Low priority full optimization set

SPICE PubChem Set 1 Single Points Dataset v1.1
50954
67652
Running now - 43% done

SPICE PubChem Set 2 Single Points Dataset v1.0
11165
110375
On hold - sorting out storage issue

SPICE PubChem Set 3 Single Points Dataset v1.0
3642
118584
On hold - sorting out storage issue

SPICE Pubchem Set 4 Single Points Dataset v1.0
474
122276
On hold - sorting out storage issue

SPICE Pubchem Set 5 Single Points Dataset v1.0
652
122498
On hold - sorting out storage issue

SPICE Pubchem Set 6 Single Points Dataset v1.0
0
123800
In queue

SPICE DES370K Single Points Dataset v1.0
0
345682
In queue

@dotsdl is working with Ben from QCA to resolve the pubchem sets 2-5 submission issue, and as of now we still have lot of work for the compute nodes to chew on, so this won't hinder any progress. Also, @dotsdl is coordinating with qcengine team to get his fix for pre-emptible manager jobs leaving behind temp files, and there will be a minor release as soon as the PR gets reviewed.

Thanks!  By my count, it completed 43,783 calculations in the last week.  A lot of those were dipeptides, which on average are a bit larger than the PubChem molecules.  Once it finishes those, I'll estimate it should be able to do about 60,000 per week.  At that rate, it will take about three months to get through all the PubChem molecules.  That leaves the DES370K dimers, but they're much smaller so they should go very quickly.
That's just to give us a very vague idea of where we are.  We can update those estimates as we get more data in the coming weeks.

An update: despite our efforts to mitigate storage increases, @bennybp is observing ~30 GB/d increase in utilization on public QCArchive. At this rate MolSSI will run out of storage in ~2 weeks. As an additional mitigation step, we have shut off OpenFF workers executing openff-spice; this allows us to continue executing datasets critical to FF fitting while we work on near-term solutions to the storage issue.
@peastman you may continue running managers on your resources so we still make forward progress here. I don't think execution rates are high enough to add significantly to the problem on the receiving end.

What about the ones that weren't set to store wavefunctions.  Could we start the DES370K dimers going?

We should be able to start DES370K without wavefunctions. We can also start pubchem set 6 without wavefunctions, too.
Realized also that we were executing SPICE Dipeptides Single Points Dataset v1.1 (https://github.com/openforcefield/qca-dataset-submission/pull/259) under the default openff compute tag, and this is also storing wavefunctions. I have changed the compute tag to openff-defunct to park this for now as well.

Thanks!  DES370K is much higher priority than pubchem 6.

Also note that we've hardly run any calculations on PubChem 4 and 5, so if it's easier to just delete them and start over, we won't be losing much.  And the amount done on PubChem 2 and 3 isn't that much more.

@dotsdl : I'm a bit confused here. Storing wavefunctions seems physically incompatible with QCArchive's current hardware situation. The only solution is to halt all datasets that are storing wavefunctions, re-create them without storing wavefunctions, and purge the datasets with wavefunctions from the QCArchive database directly. Is that the current plan?

@jchodera we have halted all datasets that are storing wavefunctions, and I got confirmation today from @bennybp that the growth rate is no longer alarming.
Our next step is to submit datasets that we have not yet submitted, but without wavefunctions. We have:

SPICE DES370K Single Points Dataset v1.0 - this is currently being submitted via local infrastructure due to its size
SPICE PubChem Set 6 Single Points Dataset v1.0 - we will submit this next

As for pubchem sets 1 - 5 as well as SPICE Dipeptides Single Points Dataset v1.1, which we have submitted with wavefunctions attached,  we have parked these for now with the openff-defunct compute tag. We cannot simply resubmit these with the existing dataset.json.bz2 submission artifacts, even with changing store_wavefunction from orbitals_and_eigenvalues to none, as the destination wavefunction key on a single point record is part of the protocols object, and is not included in the deduplication hashing. The server will view the submission as unchanged, and will not create new records with the wavefunction: none protocol we desire here. We also cannot yet delete records and guarantee internal consistency in the DB.
One way around the deduplication machinery is to translate each molecule by e.g. 1 angstrom, and we can do this if necessary, but we can buy some time with the two submissions above and the still-executing SPICE Dipeptides Optimization Dataset v1.0. The cleaner approach in my view would be to attempt record deletion with the upcoming new server deployment, which does support deletion, then resubmit with the existing artifacts.
I recognize there is a lot packed in the above. Happy to discuss live or elaborate further if needed. We are trying to push forward where we can without exacerbating the storage issues on public QCA further.

@dotsdl : Maybe this is a good time to decouple the infrastructure issues from what the highest-value science goals are, since we're now five months into the project with almost no usable data to show for it.
@peastman : Can you rank all the datasets in order of importance? My guess at priority (and rationale) is:

SPICE DES Monomers Single Points Dataset - minimum required coverage for organic molecules
SPICE Dipeptides Single Points Dataset - more extensive coverage for proteins
SPICE Dipeptides Optimization Dataset - comparison of whether MD-generated single points vs OptimizationDataset is better
SPICE DES370K Single Points Dataset - noncovalent interactions for organic molecules
SPICE Solvated Amino Acids Single Points Dataset - impact of solvent
SPICE PubChem - extensive coverage of druglike space

@dotsdl : I suggest writing off anything that's already been run as dead, excising it in the future when supported by QCArchive, and restarting the new datasets in order of priority (with the 1A shift hack) with settings that will not kill the QCArchive infrastructure.
Sorry for the odyssey in pain, folks. There are always growing pains in scaling any new technology.

The DES monomers and the solvated amino acids are already done, so there's no reason to rerun them.  Here is how I would rank the others.

DES370 dimers - this is where most of our information about nonbonded interactions comes from
Dipeptides single points - essential for anything involving proteins
PubChem, in order - they're sorted so the earliest ones have the most diversity
Dipeptides optimization - I don't have any use for this, and I don't believe it will contribute significant information that isn't in the other datasets

Just as a note, in the last few days some crazy expensive calculations have started appearing.  I have nodes that have been running for nearly 24 hours and only completed 3 tasks.  I don't believe anything in SPICE should take anywhere close to that long.  Maybe they're from some other project?  But my nodes are supposed to be configured to prioritize SPICE calculations over other ones.  It would be really helpful if there were some way I could determine what was going on.

@peastman from the logs can you see the task id, something like Task 15081373 : sent / success, you can use that to query results
from qcportal import FractalClient
client = FractalClient()
record = client.query_results(15081373)[0]
print(record.dict())

Ahh, I might be wrong, please wait for @dotsdl to take a look

my guess is that some of the low priority tasks from dipeptide optimization set made it to the queue, checking the managers on completed jobs, meanwhile re-tagged the set to not send any jobs to any queue.

Thank you @jchodera and @peastman. I will prepare new versions of the following (all without wavefunctions, with molecule coordinates translated), and submit in order:

[ ] SPICE Dipeptides Single Points Dataset v1.2
[ ] SPICE PubChem Set 1 Single Points Dataset v1.2
[ ] SPICE PubChem Set 2 Single Points Dataset v1.1
[ ] SPICE PubChem Set 3 Single Points Dataset v1.1
[ ] SPICE PubChem Set 4 Single Points Dataset v1.1
[ ] SPICE PubChem Set 5 Single Points Dataset v1.1
[ ] SPICE PubChem Set 6 Single Points Dataset v1.0 (no need for a new version; ready to submit existing version)

SPICE DES370K Single Points Dataset v1.0 is computing now, and this is the highest priority in @peastman's list.
@peastman do you have the --verbose flag running for your managers? If so, you should see the full task specs in the logs for your managers. These won't tell you what dataset the task is from, but if it has procedure: geometric in the content then this means it is an optimization, not a single point, calculation.

@dotsdl SPICE Dipeptides Single Points Dataset is a small one with 33K calculations and it is 60% done, do you think the storage space we get back from removing pubchem set 1 would be more than enough to compensate for the remaining calculations on this set? We can save some compute time if we do so.

@pavankum since it's small I'd actually rather proceed with a v1.2 resubmission and halt all compute on the v1.1 submission for the Dipeptides Single Point Set. It appears wavefunction storage really does grow space utilization fast with this large basis set, and I'd rather we eliminate it entirely from the sets above. We also don't plan on attempting deletion soon, since it technically isn't supported by the server code and would in effect be direct DB surgery (with potentially unintended negative consequences) until we have the new QCA server code deployed.
A hard halt on wavefunction storage where we don't need them is the approach I'd like to take here. This saves us the space for cases where we do want it, such as OpenFF datasets for ESP work.

@pavankum SPICE Dipeptides Single Points Dataset v1.2 is ready for review! Please do examine it with suspicion; feel free to squash and merge when satisfied!

All PRs to openforcefield/qca-dataset-submission for the above have been created. Submission Action is running now for SPICE Dipeptides Single Points Dataset v1.2 and SPICE PubChem Set 1 Single Points Dataset v1.2.

Thanks!

It's been about two weeks since we restarted the computations without wavefunctions.  This seems like a good time for another look at how we're doing overall.
The DES370K dimers are about 99% done.  At this point it's cycling errors.  For whatever reason, some of these structures seem to be very difficult.  I see error rates over 50% on some nodes.  I don't know if there's anything we can do about that.
The dipeptides are about half done, completing about 2000 computations per day.  At that rate, it should take about another week to complete all of them.
Based on experience the first time we ran them, the PubChem molecules tend to go about twice as fast as the dipeptides, though with a higher error rate (around 10%, compared to no errors at all for the dipeptides).  If we continue at the current rate, it will probably take about one month each for the PubChem sets, or about six months to get through all of them.  We can start experimenting with fitting models before all of them are done, though.

I've deployed QCFractal <IP_ADDRESS> managers to Lilac and PRP, and have upped the deployment count for each substantially. I haven't observed indications of the issues we had previously, so assuming we get decent allocations of jobs I expect we'll get higher throughput this week.

That's great news, thanks!

I'm not seeing any sign of the extra compute resources being used.  It only seems to be completing about 1000 tasks per day.

I see 1329 calculations in the last 24-hour difference reported for the https://github.com/openforcefield/qca-dataset-submission/pull/269.

It has typically been getting through about 2000 per day, so it's down rather than up.  I haven't been able to get as many nodes on Sherlock the last few days, which accounts for the decrease.  But there's no sign of any new resources being added.

I'm not currently getting any pods on PRP despite my requested resources there, and I do have Lilac prioritized for OpenFF biopolymer work over SPICE. We are running with the cpuqueue mostly on Lilac, which keeps each execution limited to no more than 6 hours. Do we anticipate many of these calculations taking longer than that? If so I can switch off cpuqueue requests and stick to preemptible there, which have a 157 hour limit.

That depends on the hardware.  On Sherlock it's averaging about 6 core-hours per calculation.  Since each node has 32 cores, the clock time for each calculation is far less than that.

You can still submit cpuqueue jobs with up to 72 hour wall clock limits---it will just exclude some nodes with 6-hour wall clock limits.

Ah cool, thanks @jchodera! Doing you think upping to the 72 hour wall clock limits for cpuqueue would significantly reduce our footprint of workers on Lilac, with preemptible jobs in the queue as well?

Doing you think upping to the 72 hour wall clock limits for cpuqueue would significantly reduce our footprint of workers on Lilac, with preemptible jobs in the queue as well?

If you can check the logs of the 6h jobs and see how much time is wasted by early termination, that would help us make a rational decision here!

Looking to the logs I think we're fine; I see many multiples (up to ~50) completed with largely 100% success rates across our cpuqueue workers. I don't think we're losing much with a 6-hour limit, or at least it's not apparent at this time.

Since PubChem set 1 was submitted four days ago, it has averaged just over 2000 calculations per day.  Compare that to 5000 per day it was averaging in mid January.  And that included dipeptides, which were larger than the PubChem molecules.  We still seem to be getting very little resources from most of the clusters.

@dotsdl : Can we fully dedicate lilac to the OpenMM calculations?
Also, any luck with the HPC folks in helping optimize the lilac deployment? Let me know if you need help with that.

@dotsdl : It looks like you're also asking for one core per thread-slot, which means that we're wasting hyperthreads:
affinity[core(1)*1:distribute=pack]

Was this left over from my previous attempts (which was a mistake!) or the result of new profiling?
If we don't do this, we can fit 2x as many workers on the nodes at once.
If this is not the result of profiling, maybe we could try removing the whole affinity[core(1)*1:distribute=pack] block and see if that helps throughput?

@jchodera I have not worked with the Lilac HPC folks for any profiling on these workloads. Part of the challenge is the wide range of memory requirements that can come down the pipe, so we have chosen a memory configuration that is reasonably high (providing 70GiB to psi4 execution) and a number of cores that give us reasonable coverage of nodes based on bhost output. If there are ways to optimize our usage of Lilac further, I'm happy to work with you on it.
Over this last week we have de-prioritized SPICE workloads to give more time time to ESP optimizations and protein capped 1-mer sidechain torsiondrive across all OpenFF compute resources, as these are key for FF development right now at OpenFF. I will ask to shift priority on Lilac to SPICE, however, on OpenFF#qcfractal-compute.

@dotsdl hyperthreading is supported by psi4 and I think it is mostly handled by MKL subroutines, but switching it off is recommended for improved performance, here is one post where Holger Kruse comments on that, I think your current lilac deployment is good but you can test it with and without the options John specified to confirm again.

I'm trying to inspect the progress so far by using the code from the notebook posted above.
ds = fc.get_collection('Dataset', 'SPICE PubChem Set 1 Single Points Dataset v1.2')
spec = ds.list_records().iloc[0].to_dict()
recs = ds.get_records(method=spec['method'], basis=spec['basis'], program=spec['program'], keywords=spec['keywords'])

That returns 118606 records, which is correct.  But every one of them has its status set to 'INCOMPLETE'.  Is that not the correct way to find tasks that have been finished?  I also searched for the ID of a task from the Sherlock logs, and it isn't present.  Does that mean it's actually working on some other dataset?

But every one of them has its status set to 'INCOMPLETE'.

I just checked it. I could see

spice

COMPLETE
9621

ERROR
17

INCOMPLETE
108968

NaN
0

RUNNING
0

I also searched for the ID of a task from the Sherlock logs, and it isn't present. Does that mean it's actually working on some other dataset?

Task ids are different from record ids so you may not see it here.
@dotsdl bumped the priority on Lilac and we would see improved throughput this week, apologies for the slow pace.

Task ids are different from record ids so you may not see it here.

How can I determine what dataset a task ID is for?

I think adding a verbose tag --verbose to the qcfractal manager would print out more info about the job in the logs, other than that I don't know a direct way since tasks vanish after they complete calculation, if they're in queue we can see using client.query_tasks(id), @dotsdl do you have any idea?
A crude way is to check the temp files created on the node but that is the last resort.
For the completed records, these are the hosts,
{'sb08': 136,
         'sb07': 42,
         'sh03': 1496,
         'hpc3': 4427,
         'sb09': 38,
         'tscc': 3288,
         'sb01': 40,
         'sb10': 54,
         'lt22': 2,
         'lx01': 2,
         'lt13': 3,
         'lt14': 6,
         'lt17': 5,
         'lx10': 4,
         'ls13': 2,
         'ls04': 3,
         'lt15': 2,
         'lu08': 4,
         'ls17': 5,
         'lu04': 2,
         'ls12': 3,
         'lt04': 1,
         'ls15': 3,
         'lu05': 3,
         'lt18': 1,
         'ls01': 2,
         'lu02': 3,
         'lx08': 3,
         'lt09': 4,
         'lt03': 2,
         'ls16': 4,
         'ls11': 1,
         'ls02': 3,
         'lt05': 1,
         'lt20': 2,
         'lt19': 3,
         'lx07': 1,
         'lx04': 3,
         'ls18': 2,
         'lu01': 1,
         'lx12': 2,
         'lu03': 3,
         'lt07': 1,
         'ls09': 1,
         'lx14': 2,
         'lx13': 1,
         'lt21': 1,
         'lx03': 1,
         'lt11': 1,
         'lt01': 1}

I'm trying to inspect the progress so far by using the code from the notebook posted above.
ds = fc.get_collection('Dataset', 'SPICE PubChem Set 1 Single Points Dataset v1.2')
spec = ds.list_records().iloc[0].to_dict()
recs = ds.get_records(method=spec['method'], basis=spec['basis'], program=spec['program'], keywords=spec['keywords'])

That returns 118606 records, which is correct. But every one of them has its status set to 'INCOMPLETE'. Is that not the correct way to find tasks that have been finished? I also searched for the ID of a task from the Sherlock logs, and it isn't present. Does that mean it's actually working on some other dataset?

If you then do the following, you can get the status counts:
> recs.record.apply(lambda x: x.status).value_counts()
RecordStatusEnum.incomplete    107931
RecordStatusEnum.complete       10616
RecordStatusEnum.error             59

I again confirmed with David and Ben this morning and there is no easy way to map a task id to a dataset name or record id because of some database management quirks (that I'm not fully aware of), so the alternative is to add --verbose flag in the qcfractal-manager invocation and we will see something the task input dict in the output, which shows the record id too
[D 220301 02:38:51 base_adapter:164] Submitted Task:
    {'id': '15096884', 'spec': {'function': 'qcengine.compute', 'args': [{'id': '98058346', 'schema_name': 'qcschema_input', 'schema_version': 1, 'molecule': {'schema_name': 'qcschema_molecule', 'schema_version': 2, 'validated': True, 'symbols': array(['F', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'N',
           'N', 'C', 'C', 'C', 'C', 'C', 'C', 'C', 'H', 'H', 'H', 'H', 'H',
           'H', 'H', 'H', 'H', 'H', 'H', 'H', 'H', 'H', 'H', 'H', 'H', 'H',
           'H', 'H'], dtype='<U1'), 'geometry': array([[ 1.32277753, 12.13673029,  4.58676732],
           [ 2.74117171, -3.37247372,  0.46497419],
           [ 3.27167927, -2.22108451, -1.82781208],
           [-0.81905666,  8.44658523,  6.25844414],
           [ 0.60807723,  9.64776561,  4.37540051],
           [-0.99468595,  1.04103464,  0.37783542],
           [-1.59205677,  6.08150083,  5.78128413],
           [-1.67002485,  2.07815276,  2.87353666],
           [ 1.23046849,  8.28151831,  2.20489806],
           [ 0.37337192,  2.62507495, -1.35773555],
           [-1.01973743,  4.67038141,  3.43013398],
           [ 0.60990498,  5.97078145,  1.60663804],
           [-3.18447086,  0.89929457,  4.54248295],
           [ 1.22906084,  4.95347152, -0.68222319],
           [ 7.56334422, -3.86248014,  1.40401581],
           [ 4.7457325 , -3.9224541 ,  2.42733197],
           [ 0.89290639,  1.66545696, -3.98121248],
           [-0.03038161, -3.44810955,  1.35883972],
           [-1.19262695, -2.46693249, -3.14191095],
           [ 1.33341306, -1.26060621, -3.78262135],
           [-1.6771054 , -1.73739699, -0.34291917],
           [ 5.16845801, -1.89009057, -2.5896373 ],
           [-1.39385626,  9.40361123,  7.97385228],
           [-2.5774034 ,  4.94398437,  7.1378082 ],
           [ 2.43946359,  9.28987724,  1.11636554],
           [-3.59373722,  1.52785283,  6.25631486],
           [-3.93972667, -0.72186064,  4.11937379],
           [ 2.12675111,  6.07606139, -1.95205617],
           [ 7.86036565, -1.86980069,  1.37390696],
           [ 8.9414188 , -4.68455055,  2.76633661],
           [ 7.88820577, -4.63389298, -0.46430376],
           [ 4.5194578 , -2.80679542,  4.08871573],
           [ 4.52894742, -5.95220626,  3.04864625],
           [-0.6408944 ,  2.17607349, -5.34266868],
           [ 2.59052034,  2.67382299, -4.75858333],
           [-0.71635947, -5.44205323,  1.4498604 ],
           [-0.37342337, -2.75657353,  3.40176762],
           [-1.15721878, -4.60840064, -3.12953896],
           [-2.63695024, -1.75157612, -4.5019209 ],
           [ 1.96146072, -2.10062906, -5.62731947],
           [-3.73724102, -2.07906278,  0.05693225]]), 'name': 'C18FH20N2', 'identifiers': {'molecule_hash': 'd6e0aa3d71845b8474721ba17333eefe13f2e8e0', 'molecular_formula': 'C18FH20N2'}, 'molecular_charge': 1.0, 'molecular_multiplicity': 1, 'connectivity': [[0, 4, 1.0], [1, 2, 2.0], [1, 15, 1.0], [1, 17, 1.0], [2, 19, 1.0], [2, 21, 1.0], [3, 4, 1.0], [3, 6, 2.0], [3, 22, 1.0], [4, 8, 2.0], [5, 7, 2.0], [5, 9, 1.0], [5, 20, 1.0], [6, 10, 1.0], [6, 23, 1.0], [7, 10, 1.0], [7, 12, 1.0], [8, 11, 1.0], [8, 24, 1.0], [9, 13, 2.0], [9, 16, 1.0], [10, 11, 2.0], [11, 13, 1.0], [12, 25, 1.0], [12, 26, 1.0], [13, 27, 1.0], [14, 15, 1.0], [14, 28, 1.0], [14, 29, 1.0], [14, 30, 1.0], [15, 31, 1.0], [15, 32, 1.0], [16, 19, 1.0], [16, 33, 1.0], [16, 34, 1.0], [17, 20, 1.0], [17, 35, 1.0], [17, 36, 1.0], [18, 19, 1.0], [18, 20, 1.0], [18, 37, 1.0], [18, 38, 1.0], [19, 39, 1.0], [20, 40, 1.0]], 'fix_com': True, 'fix_orientation': True, 'fix_symmetry': 'c1', 'provenance': {'creator': 'QCElemental', 'version': 'v0.22.0', 'routine': 'qcelemental.molparse.from_schema'}, 'id': '91493641', 'extras': {'canonical_isomeric_explicit_hydrogen_mapped_smiles': '[H:24][c:7]1[c:4]([c:5]([c:9]([c:12]2[c:11]1[c:8]([c:6]3[c:10]([n+:14]2[H:28])[C:17]([C@:20]4([C:3](=[C:2]([C:18]([C@@:21]3([C:19]4([H:38])[H:39])[H:41])([H:36])[H:37])[C:16]([H:32])([H:33])[C:15]([H:29])([H:30])[H:31])[H:22])[H:40])([H:34])[H:35])[N:13]([H:26])[H:27])[H:25])[F:1])[H:23]'}}, 'driver': 'gradient', 'model': {'method': 'wb97m-d3bj', 'basis': 'def2-tzvppd'}, 'keywords': {'maxiter': 200, 'scf_properties': ['dipole', 'quadrupole', 'wiberg_lowdin_indices', 'mayer_indices', 'mbis_charges'], 'wcombine': False}, 'protocols': {'wavefunction': 'none'}, 'extras': {'_qcfractal_tags': {'program': 'psi4', 'keywords': '31'}}, 'provenance': {'creator': 'QCElemental', 'version': 'v0.23.0', 'routine': 'qcelemental.models.results'}}, 'psi4'], 'kwargs': {'local_options': {'memory': 180.0, 'ncores': 48, 'scratch_directory': '/tmp/11036375', 'retries': 5, 'nnodes': 1, 'cores_per_rank': 1}}}, 'parser': 'single', 'status': 'RUNNING', 'program': 'psi4', 'procedure': None, 'manager': 'UCI-HPC3-hpc3-19-19-2359d0ec-13e8-40e3-ba67-b0be4a79d1c1', 'priority': 1, 'tag': 'openff-spice', 'base_result': '98058346', 'modified_on': '2022-03-01T10:38:51.001480', 'created_on': '2022-01-28T23:24:14.058398'}

Tasks don't know which dataset they belong to, and neither do records. The mapping in QCFractal is datasets -> records -> tasks, not the other way around. This is because multiple datasets can in principle point to the same records.

For the completed records, these are the hosts,

Wow---lilac nodes (l??? hostnames) have completed a vanishingly small fraction of this dataset. I wonder if there is something pathologically wrong with our deployment.

@jchodera I made the switch to the compute tags on Lilac yesterday morning and allowed the workers to continue until they died naturally and new ones took their place. After having a day to work, the completions by host for Pubchem Set 1 are:
tscc-13-16.sdsc.edu    2064
tscc-13-20.sdsc.edu    1224
hpc3-20-22              549
hpc3-20-21              412
hpc3-17-05              340
sh03-01n19.int          308
sh03-01n17.int          288
sh03-01n16.int          255
sh03-01n64.int          229
sh03-01n46.int          209
hpc3-20-19              197
hpc3-17-07              195
hpc3-20-16              189
hpc3-20-17              172
sh03-01n27.int          170
sh03-01n03.int          164
sh03-01n66.int          159
hpc3-17-06              156
hpc3-19-00              156
hpc3-22-08              148
sh03-01n45.int          132
hpc3-19-01              128
hpc3-19-06              126
hpc3-19-02              115
hpc3-19-16              113
hpc3-17-04              107
sh03-01n25.int          105
sh03-01n65.int          100
sh03-01n58.int           97
sh03-01n21.int           88
hpc3-20-20               83
hpc3-l18-00              83
hpc3-21-01               80
hpc3-21-30               66
hpc3-21-06               63
hpc3-19-05               59
hpc3-19-04               57
hpc3-19-18               56
lu08                     56
hpc3-21-12               54
sb085.cluster            54
hpc3-19-11               54
hpc3-19-03               52
hpc3-19-10               51
sb080.cluster            51
lt14                     50
lu04                     50
hpc3-21-09               50
hpc3-19-07               49
lx12                     49
ls15                     47
hpc3-19-08               47
lt18                     46
ls16                     46
sh03-01n67.int           46
lx03                     45
hpc3-19-19               45
hpc3-21-18               45
sb101.cluster            45
ls10                     44
lt06                     44
lt17                     44
ls09                     44
hpc3-19-09               43
lt20                     42
lu03                     42
lx10                     42
ls04                     42
sb077.cluster            42
lt13                     41
ls02                     41
hpc3-21-05               41
sb019.cluster            40
lx01                     39
sh03-01n24.int           39
hpc3-21-16               39
ls06                     39
lu01                     38
lt04                     38
hpc3-19-17               38
sb090.cluster            38
hpc3-l18-03              37
lt08                     37
lt09                     37
hpc3-21-13               36
lx13                     35
ls12                     35
lt03                     35
lt10                     35
lu06                     34
lu02                     33
lx14                     33
lu05                     32
lt19                     32
lt11                     32
ls18                     32
sb088.cluster            31
sh03-01n22.int           30
lt07                     30
lx04                     30
lt21                     29
ls13                     29
lt22                     29
hpc3-21-31               29
lt05                     29
lt01                     29
lx11                     29
lt15                     29
ls07                     28
ls03                     28
ls17                     28
lx08                     28
lx07                     26
lt02                     26
hpc3-21-20               26
lu07                     24
ls11                     24
ls01                     21
ls08                     17
lt12                     15
hpc3-17-12               12
hpc3-20-24               11
sb103.cluster             9
hpc3-17-15                8
hpc3-18-02                7
hpc3-18-00                7
hpc3-17-17                5
hpc3-20-18                3
hpc3-17-13                2
hpc3-22-03                1
hpc3-22-05                1
hpc3-17-14                1
hpc3-15-20                1

So the contributions from Lilac are climbing as we would expect.

What's the correspondence between those names and the different clusters?

I'm not sure about the others, but all the hostnames matching the regex l[stuvwxy]\d\d belong to MSKCC's lilac cluster.

@dotsdl @bennybp @pavankum : Would any of you have time to help us work with this dataset by converting some of the completed (or nearly complete) datasets into HDF5 files similar to what you have for the QCArchive dashboard, or the TorchMD-Net HDF5 format?
@jeherr can then dive in and start building models.

Single point datasets do have built-in support for export to HDF5 and I tried this for the Monomers single point dataset but hit an error, see the issue here.

See openforcefield/openff-qcsubmit#196 for @jthorton's implementation of HDF5 export. This is the better route in my opinion for getting what we need here.
@jchodera, @jeherr can you enumerate the data elements you need included in this kind of export?

We need a generic exporter that can include arbitrary information.  An important part of this dataset is that we're including not just forces and energies, but also other useful quantities like MBIS multipoles, bond orders, etc.  See #7.  Rather than hardcoding particular fields, we should be able to store everything contained in the records.
We also need to be sure the molecule IDs from the original input files are included in a clear way.  Those are all meaningful identifiers, such as PubChem substance IDs.  They're present in the QCArchive data, but only in an obfuscated form.  When you call get_records(), the values in the index column are of the form <molecule id>-<conformation index>.  The molecule IDs aren't present in the individual records.

I decided to try writing my own exporter as a proof of concept.  It appears to me that the data available through FractalClient is missing a lot of information.  Elements?  SMILES strings?  Atom positions?  How do I retrieve those?

this would give the molecule details,
import qcportal as ptl

client = ptl.FractalClient.from_file()
ds = client.get_collection("Dataset", "SPICE Solvated Amino Acids Single Points Dataset v1.1")
spec = ds.list_records().iloc[0].to_dict()
print(spec)
recs = ds.get_records(method=spec['method'], basis=spec['basis'], program=spec['program'], keywords=spec['keywords'])
print(len(recs))

rec = recs.iloc[0]
print("cmiles:", rec.record.get_molecule().extras)
print("Elements:", rec.record.get_molecule().symbols)
print("coordinates in bohr:", rec.record.get_molecule().geometry)

I think @jthorton made a sample exporter in qcsubmit here and he can update it accordingly to meets your needs, and can also map back to the original hdf5 you created for the molecules.

Thanks!

What units are all the quantities in?  Including positions, energies, forces, charges, multipole moments, etc.

all are in atomic units

Here is a proof of concept script for building the HDF5 file.  Does it look like I'm doing everything correctly?  Am I selecting the correct fields?
from qcportal import FractalClient
from collections import defaultdict
import numpy as np
import h5py

client = FractalClient()
ds = client.get_collection("Dataset", "SPICE Solvated Amino Acids Single Points Dataset v1.1")
spec = ds.list_records().iloc[0].to_dict()
recs = ds.get_records(method=spec['method'], basis=spec['basis'], program=spec['program'], keywords=spec['keywords'])
recs_by_name = defaultdict(list)
for i in range(len(recs)):
    rec = recs.iloc[i]
    index = recs.index[i]
    name = index[:index.rfind('-')]
    recs_by_name[name].append(rec.record)
outputfile = h5py.File('output.hdf5', 'w')
for name in recs_by_name:
    group = outputfile.create_group(name)
    group_recs = recs_by_name[name]
    molecules = [r.get_molecule() for r in group_recs]
    qcvars = [r.extras['qcvars'] for r in group_recs]
    group.create_dataset('smiles', data=[molecules[0].extras['canonical_isomeric_explicit_hydrogen_mapped_smiles']], dtype=h5py.string_dtype())
    group.create_dataset("atomic_numbers", data=molecules[0].atomic_numbers, dtype=np.int16)
    conformations = group.create_dataset('conformations', data=np.array([m.geometry for m in molecules]), dtype=np.float32)
    conformations.attrs['units'] = 'bohr'
    energies = group.create_dataset('energies', data=np.array([vars['DFT TOTAL ENERGY'] for vars in qcvars]), dtype=np.float32)
    energies.attrs['units'] = 'hartree'
    gradients = group.create_dataset('gradients', data=np.array([vars['DFT TOTAL GRADIENT'] for vars in qcvars]), dtype=np.float32)
    gradients.attrs['units'] = 'hartree/bohr'
    mbis_charges = group.create_dataset('mbis_charges', data=np.array([vars['MBIS CHARGES'] for vars in qcvars]), dtype=np.float32)
    mbis_charges.attrs['units'] = 'e'
    mbis_dipoles = group.create_dataset('mbis_dipoles', data=np.array([vars['MBIS DIPOLES'] for vars in qcvars]), dtype=np.float32)
    mbis_dipoles.attrs['units'] = 'e*bohr'
    mbis_quadrupoles = group.create_dataset('mbis_quadrupoles', data=np.array([vars['MBIS QUADRUPOLES'] for vars in qcvars]), dtype=np.float32)
    mbis_quadrupoles.attrs['units'] = 'e*bohr^2'
    mbis_octupoles = group.create_dataset('mbis_octupoles', data=np.array([vars['MBIS OCTUPOLES'] for vars in qcvars]), dtype=np.float32)
    mbis_octupoles.attrs['units'] = 'e*bohr^3'
    scf_dipoles = group.create_dataset('scf_dipoles', data=np.array([vars['SCF DIPOLE'] for vars in qcvars]), dtype=np.float32)
    scf_dipoles.attrs['units'] = 'e*bohr'
    scf_quadrupoles = group.create_dataset('scf_quadrupoles', data=np.array([vars['SCF QUADRUPOLE'] for vars in qcvars]), dtype=np.float32)
    scf_quadrupoles.attrs['units'] = 'e*bohr^2'
    group.create_dataset('wiberg_lowdin_indices', data=np.array([vars['WIBERG LOWDIN INDICES'] for vars in qcvars]), dtype=np.float32)
    group.create_dataset('mayer_indices', data=np.array([vars['MAYER INDICES'] for vars in qcvars]), dtype=np.float32)

It's truly absurd how slow it is.  The solvated amino acids dataset is tiny: just 26 molecules with 50 conformations each.  It takes over 12 minutes to run.  To process everything at that rate would take about week.
We may also want to reconsider what data to include in the HDF5 file.  It comes out to about 100 MB for this tiny dataset.  The full SPICE v1 dataset will probably add up to around 50 GB.  Is that too big?  The largest fields are the bond indices (size is O(n^2) in the number of atoms) and the MBIS octupoles (27 elements per atom per conformation).

One caution about the energy and gradient, the functional 'wb97m' and dispersion correction 'd3bj' are calculated separately and we need to add them to get the final energy as well as gradient. Their fields are 'DISPERSION CORRECTION ENERGY', and 'DISPERSION CORRECTION GRADIENT'. Other than that this looks good to me.
Runtime wise we can try to run this server side through @bennybp or there are some tricks @dotsdl or @jthorton may know.

According to the Psi4 documentation, DFT TOTAL ENERGY includes the dispersion correction.  DFT FUNCTIONAL TOTAL ENERGY is the one that doesn't include it.

@peastman one possible reason for the slowness is the individual calls to get_molecule() in molecules = [r.get_molecule() for r in group_recs] I have found it is much faster to batch all queries to the server in batches the size of the query limit which is 1000. So I would collect together all of the molecule ids you which to query and then do it in large batches as is done in qcsubmit.
The example I put together was supposed to just unblock fitting but if you need more information in the hdf5 then we can add it based on your proof of concept. I think it might be nice to have a flexible interface that defaults to extract common information like smiles, elements, return_energy, return_gradient, return_hessian etc then also allow users to pass an optional dictionary of qcvars they wish to include in the file along with the units they are in. So combining your example with the proposed interface in qcsubmit would be something like
qcvars = {"MBIS CHARGES": "e", "SCF DIPOLE": "e*bohr"}
dataset.to_hdf5(filename="my_dataset.hdf5", qcvars=qcvars)

With this kind of use, users can extract all of the information they want to use in their training sets which lets us avoid having to distribute these large files which just duplicate the information in QCFractal and in many cases may be redundant as users only want a subsection of the data. Instead, we just release each version of the SPICE dataset as a collection of record ids in qcfractal, as collections are mutable and can grow after release as calculations finish or more or added by mistake this is currently the safest way to build a fixed dataset. This is something we have done with the fitting of our sage force field and example datasets are here these datasets are JSON serialisations of QCSubmit result objects and can easily be loaded up and converted to hdf5 as needed
from openff.qcsubmit.results import BasicResultCollection
dataset = BasicResultCollection.parse_file("dataset.json")
# get the raw records to use them directly
records_and_molecules = dataset.to_records()
# or build a hdf5 file for later use
qcvars = {"MBIS CHARGES": "e", "SCF DIPOLE": "e*bohr"}
dataset.to_hdf5(filename="my_dataset.hdf5", qcvars=qcvars)

50 GB is ok in my view.
On Thu, Apr 21, 2022 at 10:38 AM Josh Horton @.***>
wrote:

@peastman https://github.com/peastman one possible reason for the
slowness is the individual calls to get_molecule() in molecules =
[r.get_molecule() for r in group_recs] I have found it is much faster to
batch all queries to the server in batches the size of the query limit
which is 1000. So I would collect together all of the molecule ids you
which to query and then do it in large batches as is done in qcsubmit.
The example I put together was supposed to just unblock fitting but if you
need more information in the hdf5 then we can add it based on your proof of
concept. I think it might be nice to have a flexible interface that
defaults to extract common information like smiles, elements,
return_energy, return_gradient, return_hessian etc then also allow users
to pass an optional dictionary of qcvars they wish to include in the file
along with the units they are in. So combining your example with the
proposed interface in qcsubmit would be something like
qcvars = {"MBIS CHARGES": "e", "SCF DIPOLE": "e*bohr"}dataset.to_hdf5(filename="my_dataset.hdf5", qcvars=qcvars)
With this kind of use, users can extract all of the information they want
to use in their training sets which lets us avoid having to distribute
these large files which just duplicate the information in QCFractal and in
many cases may be redundant as users only want a subsection of the data.
Instead, we just release each version of the SPICE dataset as a collection
of record ids in qcfractal, as collections are mutable and can grow after
release as calculations finish or more or added by mistake this is
currently the safest way to build a fixed dataset. This is something we
have done with the fitting of our sage force field and example datasets are
here
https://github.com/openforcefield/openff-sage/blob/main/data-set-curation/quantum-chemical/data-sets/1-2-0-hess-set-v2.json
these datasets are JSON serialisations of QCSubmit result objects and can
easily be loaded up and converted to hdf5 as needed
from openff.qcsubmit.results import BasicResultCollectiondataset = BasicResultCollection.parse_file("dataset.json")# get the raw records to use them directly records_and_molecules = dataset.to_records()# or build a hdf5 file for later use qcvars = {"MBIS CHARGES": "e", "SCF DIPOLE": "e*bohr"}dataset.to_hdf5(filename="my_dataset.hdf5", qcvars=qcvars)
—
Reply to this email directly, view it on GitHub
https://github.com/openmm/spice-dataset/issues/11#issuecomment-1104879379,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB3KUOW6TXEKS7GPFVSM4MLVGEHYRANCNFSM5FC6N7RA
.
You are receiving this because you commented.Message ID:
@.***>

@peastman Ahh yeah, never mind, I completely forgot that we're not splitting for these calculations. No need to do any extra modifications.

I replied over at #21.  We're getting pretty far off topic for this issue!

All my workers have started reporting errors: "Acquisition of new tasks was not successful."  And some of them are stopping early with this message:
[W 220423 10:00:48 managers:501] Could not post jobs from 1 ago and over attempt limit, marking jobs as stale.
[E 220423 10:00:48 managers:516] Exceeded number of stale updates allowed! Attempting to shutdown gracefully...
[E 220423 10:00:48 managers:516] Exceeded number of stale updates allowed! Attempting to shutdown gracefully...

@peastman I think there is a problem with an intermediate server. I am looking into it

Progress seems to have slowed way down.  Over the last two weeks, it has averaged less than 3500 calculations per day.  In the most recent 24 hour period, it only did 837.

Yeah, sorry about that, for the last one month there's been slow throughput because of drop in node availability,

Lilac: had to scale down because of an issue with their job monitoring tool with large slurm arrays, David sorted this out with the hpc admins and bumped the job count yesterday,
UCI-hpc3: had to reserve some nodes for testing openff-toolkit stuff, they're back to running qca jobs now
David is also pushing to get some extra compute from Max-Planck clusters, courtesy of Bert de Groot's lab, so we may see some improvement in throughput this week.

Nearly all the jobs I had running yesterday exited with errors:
[W 220612 04:10:16 managers:694] Acquisition of new tasks was not successful.
[W 220612 04:12:47 managers:501] Could not post jobs from 1 ago and over attempt limit, marking jobs as stale.
[E 220612 04:12:47 managers:516] Exceeded number of stale updates allowed! Attempting to shutdown gracefully...
[E 220612 04:12:47 managers:516] Exceeded number of stale updates allowed! Attempting to shutdown gracefully...

Does something need to be fixed on the server?

I think the server is being a bit overloaded at the moment. Let’s wait a bit and see if it clears up.
some stuff is getting through, but with delays.

We're almost there!  PubChem set 6 has now completed its initial pass through the data.  We'll need a little more time for error cycling, and then we can mark it as complete and release the dataset!

And we are done!  The release is at https://github.com/openmm/spice-dataset/releases/tag/1.0, including an HDF5 file with the most commonly used data fields.  Congratulations everyone!
I'm now training a model on the finished dataset.  I should have a complete first draft of the paper ready to review in a few days.

@peastman: For posterity (or potentially for the paper), I totaled up the number of core-hours consumed, and came up with 4,057,659 core-hours.
This only counts completed ResultRecords (failures are not counted) and the cores actually used (since we have to pad the batch queue requests with more thread-slots than we actually use).
Here's the script I used to pull this information from QCPortal. It ran overnight.
import qcportal as ptl
client = ptl.FractalClient()
client

import yaml
with open('config.yaml') as input:
    config = yaml.safe_load(input.read())

import numpy as np
wall_time = 0.0
for subset in config['subsets']:
    # Download the next subset.

    print('Processing', subset)
    ds = client.get_collection('Dataset', subset)
    all_molecules = ds.get_molecules()
    for row in ds.list_records().iloc:
        spec = row.to_dict()
        if spec['method'] == 'wb97m-d3bj':
            recs = ds.get_records(method=spec['method'], basis=spec['basis'], program=spec['program'], keywords=spec['keywords'])
            break

    nrecs = len(recs)
    for i in range(nrecs):
        rec = recs.iloc[i].record
        wall_time += rec.provenance.wall_time
        print(f'{wall_time / 60 / 60} core-hours used')

Thanks, that's useful information to have.  It means an average of just under four core-hours per conformation, which sounds about right.  Are you sure wall_time is the right value to use?

Here's an example of the provenance information available in a ResultRecord:
Provenance(creator='Psi4', version='1.4.1', routine='psi4.schema_runner.run_qcschema', cpu='Intel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz', module='scf', qcengine_version='v0.22.0', username='dotson', nthreads=10, memory=70.0, wall_time=2.985147714614868, hostname='lt04')

What I'm reporting here (wall_time * nthreads) is the only sensible measure of effort that it being captured at the moment.

Whoops, I realize I neglected to multiply by nthreads in the version I ran (and pasted)! I had been accumulated wall_time * nthreads in a MyBinder session, but it kept timing out, and I must have neglected to copy over the correct version to my laptop to run locally.
We may be off by a factor of 10x. I'll rerun the corrected script.

Update: I had run the correct script and the measure should be accurate. I had just pasted the wrong script here.
The script I ran was:
import qcportal as ptl
client = ptl.FractalClient()
client

import yaml
with open('config.yaml') as input:
    config = yaml.safe_load(input.read())

import numpy as np
wall_time = 0.0
for subset in config['subsets']:
    # Download the next subset.

    print('Processing', subset)
    ds = client.get_collection('Dataset', subset)
    all_molecules = ds.get_molecules()
    for row in ds.list_records().iloc:
        spec = row.to_dict()
        if spec['method'] == 'wb97m-d3bj':
            recs = ds.get_records(method=spec['method'], basis=spec['basis'], program=spec['program'], keywords=spec['keywords'])
            break

    nrecs = len(recs)
    for i in range(nrecs):
        rec = recs.iloc[i].record
        try:
            wall_time += rec.provenance.wall_time * rec.provenance.nthreads
        except AttributeError as e:
            pass
        if i % 100 == 0:
            print(f'{wall_time / 60 / 60} core-hours used')

    print(f'{wall_time / 60 / 60} core-hours used')

print(f'{wall_time / 60 / 60} core-hours used')

Apologies for the confusion.

That's probably the best estimate we can get, but keeping in mind it's only an estimate.  Wall clock time can be influenced by a lot of things.  For example, jobs that ended up failing may have caused other jobs running at the same time to take longer to complete, so they're still influencing the results.


I think it's technical, but may be a case of fake profile

Subject: I think it's technical, but may be a case of fake profile
Description:
nothing, found as is
Results:
on the Buenos Aires host find page there were 3 profiles with exactly the same text, I have screenshots if you like.
Backend version: develop-7c33e18d
Frontend version:
User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:105.0) Gecko/20100101 Firefox/105.0
Page: https://couchers.org/search?location=Buenos+Aires%2C+Autonomous+City+of+Buenos+Aires%2C+Argentina&lat=-34.6075682&lng=-58.4370894#loc=1/0/0
User: saralesarale (9761)

This is the default text; but it's the translated version, not the English one, hence the confusion!


feat(table): adds delta cell example

Description
depends on https://github.com/uber-web/baseui/pull/941

going to close this pr so that design only needs to review 1 screener diff
see https://github.com/uber-web/baseui/pull/948


Scale bar

Scale bar for indicating the unit length (let's do this better than the old 3D viewer)

note that imglib2 calibration is just a double[]
https://github.com/imglib/imglib2-algorithm/blob/master/src/main/java/net/imglib2/algorithm/dog/DogDetection.java


Disable log compression in pipeline.groovy

It's currently incompatible with the changes Jenkins is making as part of JEP-210: issues.jenkins-ci.org/browse/JENKINS-54680

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

scripts/pipeline.groovy
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Thanks for the PR.

@mattleibow sigh, turns out you actually need to make an empty properties ( [ ] ) instead otherwise it'll keep the previous value: https://github.com/mono/SkiaSharp/pull/709

Thanks. I merged it.

Does it automatically end up in the development branch or do I need another PR?

Automatic! Almost... I use the development branch for larger work items so I can have a "stable" master and an "almost stable" master :)


Support IF NOT EXISTS in create statement

Description
Add IF NOT EXISTS in create index statement. This is useful along with checkpoint_location option introduced in https://github.com/opensearch-project/opensearch-spark/pull/39. We just submit the exact same job (create with if not exists statement) to Spark to restart index refreshing, instead of submitting new job with REFRESH statement.
Doc: https://github.com/dai-chen/opensearch-spark/blob/support-if-not-exists/docs/index.md
Issues Resolved
https://github.com/opensearch-project/opensearch-spark/issues/27
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

In general, what is preferred REFRESH INDEX statement or CREATE INDEX with IF NOT EXISTS.

In general, what is preferred REFRESH INDEX statement or CREATE INDEX with IF NOT EXISTS.

I think run REFRESH statement directly is more clear. CREATE INDEX with IF NOT EXISTS is for use case like EMR job which we want to just restart with same job info without looking into it. Anyway, these options are only in SQL. In either case, only FlintSpark.refreshIndex API will be triggered underlying. Thanks!


Return Valid JSON for empty data

For commands that ask for JSON results, if the input to the Go JSON
marshaller is empty, it will return a byte array with a literal
"null" in it.  If that is the case, we should output [] instead
as at least that is valid JSON and will not break consumers of the
data.
Signed-off-by: baude<EMAIL_ADDRESS>

LGTM

What about mount.go, history,go diff.go, info.go?

@rhatdan those should be OK because for any of them to work, an image must be present.

@baude What about an image with no differences?

tested a diff ...
[bbaude@localhost cri-o (kpod_json)]$ sudo ./kpod diff --format json fb040950889d
{}

Looks like we are good...rebased to most recent as well.

[retest]

@baude Could you rebase this so we can get it merged.

/test all

Tests pass/merging.


TypeError: 'NoneType' object is not callable for basic example

Describe the bug
When executing the basic example from https://pyattck.readthedocs.io/en/latest/ics/icsattck.html, an Exception is thrown.
To Reproduce
>>> from pyattck import Attck
>>> attack_mitre = Attck()
>>> technique_list = attack_mitre.ics.techniques
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jan/.local/lib/python3.8/site-packages/pyattck/attck.py", line 289, in ics
    from .ics import ICSAttck
  File "/home/jan/.local/lib/python3.8/site-packages/pyattck/ics.py", line 7, in <module>
    class ICSAttck(Base):
  File "/home/jan/.local/lib/python3.8/site-packages/pyattck/ics.py", line 42, in ICSAttck
    __attck = MitreAttck(**Base.config.get_data("ics_attck_json"))
  File "/home/jan/.local/lib/python3.8/site-packages/pyattck_data/attack.py", line 55, in __init__
    raise te
  File "/home/jan/.local/lib/python3.8/site-packages/pyattck_data/attack.py", line 53, in __init__
    self.__attrs_init__(**kwargs)
  File "<attrs generated init pyattck_data.attack.MitreAttck>", line 14, in __attrs_init__
  File "/home/jan/.local/lib/python3.8/site-packages/pyattck_data/attack.py", line 66, in __attrs_post_init__
    raise te
  File "/home/jan/.local/lib/python3.8/site-packages/pyattck_data/attack.py", line 62, in __attrs_post_init__
    data = TYPE_MAP.get(item['type'])(**item)
TypeError: 'NoneType' object is not callable

Referred in Issue #125 in PyAttck as well...
Expected behavior
No error
Reason
It seems like the past month was released a new STIX object. The campaigns
What I found is that is not yet implemented on the pyattck-data lib.
https://github.com/swimlane/pyattck-data/blob/e704a5c46d916ee3fb898b8c16902d8e7035fbe4/src/pyattck_data/attack.py#L27
(- by @kastellanos)

Related to question


ARTEMIS-994 Support Netty Native Epoll on Linux

The following changes are made to support Epoll.

Refactored SharedNioEventLoopGroup into renamed SharedEventLoopGroup to be generic (as so we can re-use for both Nio and Epoll)
Add support and toggles for Epoll in NettyAcceptor and NettyConnector (with fall back to NIO if cannot load Epoll)
Removal from code of PartialPooledByteBufAllocator, caused bad address when doing native, and no longer needed - see jira discussion

New Connector Properties:
useEpoll - toggles to use epoll or not, default true (but we failback to nio gracefully)
epollRemotingThreads = same behaviour as nioRemotingThreads but for Epoll.
useEpollGlobalWorkerPool = same behaviour as useNioGlobalWorkerPool but for Epoll.
New Acceptor Properties:
useEpoll - toggles to use epoll or not, default true (but we failback to nio gracefully)
useEpollGlobalWorkerPool = same behaviour as useNioGlobalWorkerPool but for Epoll.

I am running the testsuite, and if everything is ok I should merge it tomorrow...
I have 50% of tests running and so far so good.. will know in 2 hours...
BTW: can you run these commands please?
git pull -rebase upstream master
git push origin -f {your branch name}

and then:
git rebase -i upstream/master

And squash these committs into one, with a description that matches it?
I could do this myself, but I would rather have you finding the best commit name once you squash them.

Awesome work! Testsuite is good...

@clebertsuconic I have just now rebased and squashed, hope this is what you were wanting me to do.

@michaelandrepearce I don't think you really need three parameters for this... just one would do...

useEpoll...

the other two you introduced could just use the same parameter, being the same semantic.. being just epoll.
That way you would configure to use epoll or not with a single switch. If you need to configure different settings.. than you can just update the values.. it gets easier on users IMHO.
WDYT?

@clebertsuconic I though about this but annoyingly theyre named nioBlah as such just using that would be not directly indicate you're affecting epoll. Like wise if we were to strip the nio part then they would be generic but would break any compatibility with existing clients as they'd have to change their properties if used.
This is why I introduce the extra duplicate props. If you think renaming the existing properties to generic in nature I can do this but as noted wouldn't be back compatible.

@michaelandrepearce I will add a new generic parameter, and deprecate the old one.
Will do that on a separate commit. I just wanted to know if you had any reason.

@clebertsuconic sounds good.

@clebertsuconic I notice this didn't get merged still. Did I miss understand and you expected me to make the configuration changes? if so no worries, just let me know.

@michaelandrepearce nope.. I just procrastinated for an afternoon :) give me till tomorrow

@clebertsuconic I was thinking this afternoon has 2.0.0 been cut for tagging? If not then maybe worth trying to get the property change done before that as a 2.0.0 is a major breaking release anyhow would avoid need for deprecating and creating a new generic we just would need a rename which would be cleaner

@michaelandrepearce 2.0.0 (Artemis) is already cut. 3 days ago.. it was just the vote on...
So this will be for 2.0.1.
I was thinking about setting the default as NIO at least for a while, and having a --epoll --no-epoll property through the CLI.

@michaelandrepearce I did some performance tests with https://github.com/ssorj/quiver and the test hung... I will have to take double care before merging this.. it may take some extra time, as I don't want to make the broker unstable now.
I would like to investigate what's happening before merging this.

@clebertsuconic thanks, ill try look into also see if i spot anything.
I assume this is repeatable.
Was using the amqp client, looking at the quiver dependencies not the core or jms client.
Also i assume the master currently doesn't suffer this?
Lastly does this occur when set to nio also in the branch? or only when epoll is on?

@michaelandrepearce I couldn't check other options.. I was using AMQP..
so, I was using this command line on quiver to replicate it:
quiver q0 --sender qpid-messaging-cpp --receiver qpid-messaging-cpp --messages 1m --body-size 100 --credit 1000 --timeout 10

you may need to use a few snapshots on quiver. but most stuff is available on fedora. Look at the list of dependencies if you like to try it.

Hi @clebertsuconic i managed to get it running.
Some notes:
I was not able to run your command as is, using the version that comes via dnf / repo, seems some options are supported or something :S
I got the below output.
$ quiver q0 --sender qpid-messaging-cpp --receiver qpid-messaging-cpp --messages 1m --body-size 100 --credit 1000 --timeout 10
usage: quiver [-h] [-m COUNT] [--impl NAME] [--body-size COUNT]
[--credit COUNT] [--timeout SECONDS] [--output DIRECTORY]
[--init-only] [--quiet] [--verbose]
ADDRESS
quiver: error: unrecognized arguments: --sender qpid-messaging-cpp --receiver qpid-messaging-cpp
I did however successfully run using just:
$ quiver q0 --messages 1m --body-size 100 --credit 1000 --timeout 10
I am using fedora 25.
It ran fine with both server acceptor set to epoll and nio for me.
Attached is the output i got for both runs.
quiver.epoll.txt
quiver.nio.txt

the only comment is with epoll, i notice the producer was producing faster than the consumer during the test as such some queue depth occured, with nio producer and consumer speeds were closer as such less queue depth occured.
epoll producer rate > nio producer rate by approx 200 messages a second where as consumer rates differed only by approx 50.

@michaelandrepearce thanks for the results!

@michaelandrepearce you have to follow all the dependencies on quiver. probably the cpp didn't finish compilation on make?
There's something going on.. I will review it this week. if you run the cpp module, which generates a bit more load on AMQP and you will have a few weird errors. it just needs some testing.

@clebertsuconic i have re-run with march larger run size, and now able to reproduce.
So it seems there is a direct memory leak, this occurs with both epoll and nio set, as suchi assume this is around removal of PartialPooledByteBufAllocator for the default netty allocator rather than epoll itself. I will dig into this area first.

i turned on the netty leak detector:
22:55:59,492 SEVERE [io.netty.util.ResourceLeakDetector] LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 4
#4:
io.netty.buffer.AdvancedLeakAwareByteBuf.readBytes(AdvancedLeakAwareByteBuf.java:504)
io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:661)
org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.inputBuffer(ProtonHandler.java:182)
org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.inputBuffer(AMQPConnectionContext.java:110)
org.apache.activemq.artemis.protocol.amqp.broker.ActiveMQProtonRemotingConnection.bufferReceived(ActiveMQProtonRemotingConnection.java:134)
org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:629)
org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:68)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
io.netty.handler.codec.ByteToMessageDecoder.handlerRemoved(ByteToMessageDecoder.java:219)
io.netty.channel.DefaultChannelPipeline.callHandlerRemoved0(DefaultChannelPipeline.java:631)
io.netty.channel.DefaultChannelPipeline.remove(DefaultChannelPipeline.java:468)
io.netty.channel.DefaultChannelPipeline.remove(DefaultChannelPipeline.java:428)
org.apache.activemq.artemis.core.protocol.ProtocolHandler$ProtocolDecoder.decode(ProtocolHandler.java:185)
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
org.apache.activemq.artemis.core.protocol.ProtocolHandler$ProtocolDecoder.channelRead(ProtocolHandler.java:128)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1018)
io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
java.lang.Thread.run(Thread.java:745)

Also:
#1:
io.netty.buffer.AdvancedLeakAwareByteBuf.writeBytes(AdvancedLeakAwareByteBuf.java:600)
io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:829)
io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616)
io.netty.buffer.AdvancedLeakAwareByteBuf.readBytes(AdvancedLeakAwareByteBuf.java:469)
io.netty.handler.codec.ByteToMessageDecoder.handlerRemoved(ByteToMessageDecoder.java:217)
io.netty.channel.DefaultChannelPipeline.callHandlerRemoved0(DefaultChannelPipeline.java:631)
io.netty.channel.DefaultChannelPipeline.remove(DefaultChannelPipeline.java:468)
io.netty.channel.DefaultChannelPipeline.remove(DefaultChannelPipeline.java:428)
org.apache.activemq.artemis.core.protocol.ProtocolHandler$ProtocolDecoder.decode(ProtocolHandler.java:185)
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
org.apache.activemq.artemis.core.protocol.ProtocolHandler$ProtocolDecoder.channelRead(ProtocolHandler.java:128)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1018)
io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
java.lang.Thread.run(Thread.java:745)

@michaelandrepearce that's what I was going after.. some sort of leak while dealing with AMQP.
I will review it this week.

I think I got to the bottom of this... there was a leak.. and some proton miss use.

@michaelandrepearce did you ammend anything on your PR? I can't build it any longer (especially outside of Linux.. like on a mac).

No haven't touched it

@clebertsuconic i just did a complete fresh clone and rebuild of this PR/My branch which i rebased the other day. It still built ok for me. Should i rebase again? Maybe some commit in the last few days might have broken something?

Please review master ?

Many thanks Clebert.
Had a brief look over tonight. Seems good. Will spend a bit more time tomorrow and also take the chance to update my fork.
Sent from my iPhone

On 24 Mar 2017, at 00:47, clebertsuconic<EMAIL_ADDRESS>wrote:
Please review master ?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.


perf(lexer): introduce AtomStoreCell

Introduces an AtomStoreCell type which internally uses an UnsafeCell, but enforces a safe contract. This allows skipping the borrow check of RefCell.
> cargo "bench" "--package" "swc_ecma_parser" "--bench" "parser" "--" "three"
Main: ~38.2ms
#8231: ~37.8ms
This PR: ~37.4ms
Waiting on https://github.com/swc-project/swc/pull/8231

Maybe we can move this to swc_atoms


[MINOR][SQL]Support mutable expression unit test with codegen projection

Currently the ExpressionEvalHelper.checkEvaluation always create new projection for codegen expression, this is not helpful in testing the mutable expression, which is stateful internally.

Merged build triggered.

Merged build started.

This is the follow up of discussion on #7468

cc @davis @rxin @cloud-fan

Test build #37930 has started for   PR 7566 at commit 50dd01d.

Sorry, it's @davies, not @davis :)

Test build #37930 has finished for   PR 7566 at commit 50dd01d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait ExpectsInputTypes extends Expression
trait ImplicitCastInputTypes extends ExpectsInputTypes
trait Unevaluable extends Expression
trait Nondeterministic extends Expression
trait CodegenFallback extends Expression
case class Hex(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
case class Unhex(child: Expression) extends UnaryExpression with ImplicitCastInputTypes

Merged build finished. Test FAILed.

retest this please

Merged build triggered.

Merged build started.

Test build #50 has started for   PR 7566 at commit 50dd01d.

Merged build triggered.

Merged build started.

Test build #38038 has started for   PR 7566 at commit 50dd01d.

Test build #50 has finished for   PR 7566 at commit 50dd01d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test build #38038 has finished for   PR 7566 at commit 50dd01d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #38170 has started for   PR 7566 at commit d97a8de.

Merged build triggered.

Merged build started.

Test build #38170 has finished for   PR 7566 at commit d97a8de.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test build #38171 has started for   PR 7566 at commit ef01d47.

Test build #38171 has finished for   PR 7566 at commit ef01d47.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #38174 has started for   PR 7566 at commit 3da3bef.

Test build #38174 has finished for   PR 7566 at commit 3da3bef.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Merged build finished. Test FAILed.

retest this please

Merged build triggered.

Merged build started.

Test build #88 has started for   PR 7566 at commit 24f43ea.

Merged build triggered.

Merged build started.

Test build #38338 has started for   PR 7566 at commit 24f43ea.

Test build #88 has finished for   PR 7566 at commit 24f43ea.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test build #38338 has finished for   PR 7566 at commit 24f43ea.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

@rxin @davies any comments on this?
It's the follow up of #6762(Use the copyMemory instead of the System.arrayCopy) and #7468 (For testing the mutable status for regex_replace/extract codegen)

LGTM

Merged into master, thanks!


First service dependency of WebviewEditorInput at position 8 conflicts with 8 static arguments

Just open a markdown preview running out of sources.

I believe that this was fixed at some point as I can no longer repo it in master


How to access the memory of a RPC agent?

I am trying access the memory of a distributed agent, but it consistently results in an empty list.  How to resolve this problem? Thanks.
Example:
assistant = DialogAgent(name='assistant', sys_prompt="You are a helpful assistant.", model_config_name='llama3').to_dist()
msg = assistant(Msg(name='user', role='user', content="hi"))
print(assistant.memory.get_memory())

# print result: []

After using to_dist, the agent is transferred to other processes, and only a proxy (assistant in your example) is left locally, which is empty and doesn't have a memory field. It will only forward the call method (assistant(x) in your example) to the original agent.

Currently, getting the memory of distributed agents is not supported. After using to_dist, the agent is transferred to other processes, and only a proxy (assistant in your example) is left locally, which is empty and doesn't have a memory field. It will only forward the call method (assistant(x) in your example) to the original agent.
If you really want to get the memory of the distributed agent, you can check your local log dir in ./runs/{runtime id}. All messages in your application are saved in the log.

Got it. Thanks


Iotedge daemon version obsolete

Hello I am having a problem concerning iotedge daemon on the Vision AI Dev Kit.
After updating the firmware, I run iotedge check and I got the following errors:

I can't update the daemon because there is no package manager in Yocto. Is it possible to override the actual Linux image and install a new one that has the iotedge daemon up to date ?

@othmane-alaoui yes, you can try that out.

Thank you @v-prsasa for your answer, is there any tutorial or manual that can help me with that ?

@othmane-alaoui We don't have particular tutorial as of now. below link may help you.
https://azure.github.io/Vision-AI-DevKit-Pages/docs/Get_Started/

@othmane-alaoui Should we go ahead and close this case?

@othmane-alaoui Can we close this case now? Is the issue resolved?

@v-prsasa this issue is not resolved yet, I am still looking for a solution

@othmane-alaoui -

Please find the latest firmware released by Altek for the Vision AI DevKit - https://azure.github.io/Vision-AI-DevKit-Pages/docs/Firmware/
In order to communicate with the Vision AI DevKit using a command line, including updating the firmware when needed, you will need the Android SDK Platform Tools. The Vision AI DevKit specifically uses the Android Debug Bridge (ADB) and Fastboot - https://azure.github.io/Vision-AI-DevKit-Pages/docs/platform_tools/
Use this guide for setting up Azure resources - https://azure.github.io/Vision-AI-DevKit-Pages/docs/Setup_Azure_resources/

Closing this case as we don't see any response since past 14 days.


chore update Croaring for good performance

Proposed changes
update Croaring for good performance, according to https://github.com/RoaringBitmap/CRoaring/pull/320
Problem Summary:
Describe the overview of changes.
Checklist(Required)

Does it affect the original behavior: (Yes/No/I Don't know)
Has unit tests been added: (Yes/No/No Need)
Has document been added or modified: (Yes/No/No Need)
Does it need to update dependencies: (Yes)
Are there any changes that cannot be rolled back: (Yes/No)

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

LGTM

cool


Update dependencies and other small fixes

Okay, phew, I was just updating deps and then I found a thing or two to fix so... :sweat_smile:
@jcgruenhage do you mind taking a quick look if things look good here?
I'll merge and release 0.2.0 afterwards (major bump due to error type being fixed).

lgtm!

Aaand... it's done: https://crates.io/crates/clap_allgen thanks for your cooperation! :bow:


@urql/svelte adds nearly 60kb to the bundlesize of sveltekit

If I do a clean sveltekit installat, simply add @urql/sveltekit and instantiate the client, the bundle-size for the vendor-chunk jumps up from ~7kb to 65kb uncompressed. After compression the situation is much better, but it seems to be too high in the first place.
Without @urql/svelte:
.svelte-kit/output/client/_app/chunks/vendor-08a3096e.js        6.97 KiB / brotli: 2.60 KiB

With @urql/svelte:
.svelte-kit/output/client/_app/chunks/vendor-7e63fcf0.js        65.22 KiB / brotli: 16.49 KiB

urql version & exchanges:
latest @urlq/svelte, no exchanges
Steps to reproduce
Simply run the following commands, choose "svelte skeleton" to get a clean project.
 pnpx create-svelte@next sveltekit-urql-filesize && cd sveltekit-urql-filesize && pnpm i && pnpm build && pnpm add @urql/svelte graphql && echo "
Welcome to SvelteKit
" > src/routes/index.svelte && pnpm build

Expected behavior
I expected a max of ~30kb of bundlesize (@urql/svelte adds around 22kb according to bundlephobia).
Actual behavior
65kb of bundlesize

we've got a reply here for the exact same question: https://github.com/FormidableLabs/urql/discussions/1905#discussioncomment-1248775
Also, this is more of an open ended question rather than an issue, so I'll convert it over to a discussion thread ✌