Store record count in S3 object metadata to avoid unnecessary transfers

Based on https://github.com/nextstrain/ncov-ingest/pull/313 by
@rajivshah3, with slight modifications and several commits collapsed to
one.
Co-authored-by: Rajiv Shah<EMAIL_ADDRESS>Co-authored-by: Thomas Sibley<EMAIL_ADDRESS>Testing

[ ] CI
[ ] Verify metadata on objects produced by CI

I started a test run to see it work in full pipeline context.

Test run launched AWS Batch job id b3198dfa-de99-48a2-8a47-ee7c9728b58e which successfully set the recordcount metadata on the objects under s3://nextstrain-staging/files/ncov/open/branch/trs/record-count-in-metadata/, for example:
$ aws s3api head-object --bucket nextstrain-staging --key files/ncov/open/branch/trs/record-count-in-metadata/genbank.ndjson.xz
{
    "AcceptRanges": "bytes",
    "Expiration": "expiry-date=\"Fri, 01 Jul 2022 00:00:00 GMT\", rule-id=\"Cleanup dev files under files/ncov/open/branch/\"",
    "LastModified": "Wed, 15 Jun 2022 04:50:30 GMT",
    "ContentLength":<PHONE_NUMBER>,
    "ETag": "\"e0cbd5128d7c4584699d958815aee1cc-147\"",
    "VersionId": "null",
    "ContentType": "application/x-xz",
    "Metadata": {
        "recordcount": "5570514",
        "sha256sum": "5591feb7f971f44fd61a44a4cd0a2203e47869713e2305a51296048a57d6a38d"
    }
}

Thanks so much for this contribution, @rajivshah3!


refactor(portal): separate backend from portal

Separate portal-server from portal-web.

Codecov Report
Base: 70.29% // Head: 70.63% // Increases project coverage by +0.34% :tada:

Coverage data is based on head (6313e1f) compared to base (fb00738).
Patch has no changes to coverable lines.

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #230      +/-   ##
==========================================
+ Coverage   70.29%   70.63%   +0.34%
==========================================
  Files          93       78      -15
  Lines        2380     1948     -432
  Branches      304      237      -67
==========================================
- Hits         1673     1376     -297
+ Misses        660      535     -125
+ Partials       47       37      -10

Impacted Files
Coverage Δ

...portal-server/src/clusterops/slurm/bl/submitJob.ts

apps/portal-server/src/utils/turbovnc.ts

apps/portal-server/tests/file/utils.ts

apps/portal-web/src/pages/api/file/copy.ts

apps/portal-web/src/pages/api/file/createFile.ts

apps/portal-web/src/pages/api/file/deleteDir.ts

apps/portal-web/src/pages/api/file/deleteFile.ts

apps/portal-web/src/pages/api/file/download.ts

apps/portal-web/src/pages/api/file/list.ts

apps/portal-web/src/pages/api/file/mkdir.ts

... and 4 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


README describes 'b e' to edit last block in page - but config file has it set as 'b l'

Unless I'm missing something, there's a difference in the docs and implementation. The docs should say b l to edit the last block, shouldn't it?
Thanks for the plugin; it scratches an itch I've had for keyboard shortcuts since I started with logseq a few months ago.

You're completely correct! Good catch. Should be fixed now.
I'm glad you're finding the plugin useful!


CharacterAPI support

DLL can be moved to an outside archive used in actions, but since I don't have access to that I had to add it to repository.

This should probably be retargeted into develop - a lot o similar code is there.
Is the source for this assembly public? I would much prefer to add it as a submodule in the libs/ folder.

You can redirect it into develop or I can submit pull request.
The source is just CharacterAPI as a whole. You can use it however you want.

I'll close this and add it into develop for you - thank you for the contribution, though!

Actually, I've encountered a small issue - depending on the CharacterAPI assembly itself will need it in the build directory and thus loaded as a BepInEx plugin, setting it as a hard dependency. You will probably need to adopt the separate assembly strategy like in SlopCrew.API/CrewBoomAPI.

It is set as soft dependency, you don't need it for the plugin to work. You obviously need it for a build.
Unless you have some github action issue that can't remove the plugin from build artefact I don't see how it can be an issue.

On build, because of it being referenced, the assembly lands in the output directory:
$ ls -1 SlopCrew.Plugin/bin/Debug/net462/
CharacterAPI.dll
SlopCrew.API.dll
SlopCrew.API.pdb
SlopCrew.Common.dll
SlopCrew.Common.pdb
System.ValueTuple.dll
websocket-sharp.dll

BepInEx will then scan the plugin installation directory recursively, meaning it will see CharacterAPI.dll and load it. It would likely be needed to hack something together with something like Costura, but the best approach is to move it into a separate assembly that BepInEx won't attempt to load.
Two copies of the same assembly with the same version number resolve to the same instance, so as long as the version numbers match, you can have an API assembly you can set values in your plugin and read from another plugin.

It only lands there because, once again, completely on me, I forgot to set "Copy Local" to "No" for the dll when commiting to repository. You can either do it via VS Studio or just edit *.csproj file manually
<Reference Include="CharacterAPI">
  <HintPath>..\Assemblies\CharacterAPI.dll</HintPath>
  <Private>False</Private>
</Reference>```
Since you use csproj file for building via powershell script it solves the problem. I've even checked myself and it works now.

I feel like at this point it would be better for me to redo both pull requests, fix the issue regarding build, commits and do it on correct branch. I can do tomorrow if you want.

Sure!


Feature: Churn analysis using random forest algorithm

Is your feature request related to a problem? Please describe.
This feature is related to the problem of customers opting new products and leaving old products

Describe the solution you'd like
I will apply Random forest on the customer dataset and classify the customers based o the score. The customers who are more likely to leave can be approached with better discounts.

Describe alternatives you've considered
I am planning to apply random forest , the alternative would be using SVM.

Additional context
@keshavsingh4522 - Assign this issue to me.
Thanks

I want to work on this


Update ref based rna-seq

This PR in based on #3272
Additional changes in this PR:

Use collection instead of individual datasets
Use last up to date version of tools (except STAR which I could do).
Add myself as contributor
Update some phrasing: changing 'absolute fold-change > 2' to 'fold-change > 2 or < 0.5' etc.
Change color scale of log2(counts)
Change some images / text on strandness
Add a step to check Pasilla gene status
Propose an alternative with counts with STAR
Use a choose your own adventure (CYOA) style thanks to @hexylena
Use last version of heatmap2 and plot z-score without computing them. Moved the part on how to compute z-scores in a detailed section and do it on all genes.
Use multiQC on cutadapt thanks to @bebatut
Aside:
I updated create_new file to set name at creation.
@hexylena added a post on CYOA and updated 'Create new tutorial'.
@bebatut updated copy datasets between histories

TODO:

Keep group tag but say it is an alternative to collections.
Check Pasilla gene with ID just after DESeq2

TODO if time:

update heatmap2 tool to allow Z-score computation before clustering.

Ready for review @hrhotz
@bebatut would you mind to check that the changes on strandness are ok for you (if you have time I am also interested in your review for the rest but this is the biggest change I made on what you wrote)?

Thanks @nsoranzo

I am still working on this PR (just the heatmap part because I just updated heatmap2 tool so it can compute the z-score).

Heatmap2 changes done.
Waiting for last reviews before changing the workflow.

One last comment. Could you add the level to the metadata? Thanks a lot

Which one would you put? Introductory?

I would say intermediate.

I would have said Introductory for the level. We usually use it at the introduction to RNA-Seq analysis

I still need to update the workflow for DEG but I am a bit fighting...

Also what should I do with the tour?
How can I test it?

So I guess it can be merged, isn't it?

Please wait that I update the workflow with tags.

Please wait that I update the workflow with tags.

Ah, ok sorry!

What do we do with the tour? I update it?

@shiltemann @hexylena is it possible to hide the tour so I have time to update it after my course?

I think no worries about the tour. I am quite sure it is a lot out of date. We should probably remove it

Then let's merge like this? Or I remove the tour?

Hi,
My last changes are:

Add a sentence about the discrepancies between the results of the first part and the count files on zenodo (this may disappear if we update zenodo).
Give some hints on how to run the single-read datasets. In my opinion, the single reads datasets should be run with the option in STAR Length of the genomic sequence around annotated junctions to 74 as we explain above that it should be read length - 1. However, the counts in zenodo corresponds to STAR ran with 36 like in PE datasets. What do you think?
Thanks

Hi there,
I am back from holidays and our class is on Wednesday.

Should I remove the tour or I keep like this?
Should I remove the DEXseq part which is commented but out of date?
Thanks

Let's just remove the tour.  I think ok to leave commented out portions.

Thanks @hexylena
Ready for review / merge

The updated tutorial is online :tada:
I just noticed sth. If nothing is selected for CYOA, then both content is displayed

@hexylena is there a way to select by default one of them?

Yes, that is expected. I'm not sure there's a good solution for that, picking one by default seems sub-optimal, and having both displayed at least the user can see both simultaneously which feels useful? (Espeially when it's e.g. two separate hands-on boxes). Less so when it's really inline stuff.

We could write some JS to select one (randomly or set a default.)

I think we can set a default (at least for this tutorial).

What should the default be?

@bebatut @gallardoalba @hexylena @nsoranzo @jennaj  Thank You very much for all your work and help getting @lldelisle  PR into the GTN

Yes, thank you very much, I am really happy that we have an up to date tutorial.

Let see if I have time to do this.


Look at synchrony in SqsNotificationListener tests

This PR looks at problems with the current  WhenMessageHandling** tests on SqsNotificationListener.
The [When] in BaseQueuePollingTest fires off a test scenario and does not wait for it to complete. After this, the [Then] methods all loop and wait for the specified condition to be met.
But when this is e.g. Monitor.DidNotReceiveWithAnyArgs().HandleException("Godot will come")  how long should you wait to be sure that this will not happen?
What actually happens is that Patiently.VerifyExpectationAsync checks a function repeatedly and it will pass the first time, regardless of whether an exception is thrown later or not.
So the test does not verify what it claims to, and could start failing due to timing issues in other [Then] methods if they are run, and in what order. Or it could exhibit behaviour dependant on how the test runner works. We use 3 test runners: Resharper, NCrunch and AppVeyor. Differences have been observed - appveyor breaks completely on tests that pass locally.
My strategy for fixing this is:

Use the AsyncBehaviourTest base class. The [When] method is now async and must not complete until the system under test has completed the test. This is the only async or "pateint" part of the test.

We strongly prefer to not guess at this test being complete by using long delays or retry loops. Instead we send a "done" signal using inter-thread communication mechanisms (Semaphore, Monitor, Mutex), or in this case a TaskCompletionSource when the work is done. Where we need to wait for one thread to get ahead of another, the wait should be short. The expected pass should be almost as fast as the system under test.

The [Then] methods inspect the terminal state of the system at a point in time, and do not wait. I do not recommend using the "Patiently" class to retry: the implementation is fine, but the idea is not useful here.

To an extent this is yak-shaving, but also we need to be confident in these tests before we can make other changes.
We still have a lot of noisy messages Issue in message handling loop for queue , region eu-west-1 in the appveyor build log - this seems to be because in some tests, the message loop spins around quite  few times before it gets shut down, but it's no longer breaking things and the build passes in under 2 minutes.

This addresses https://github.com/justeat/JustSaying/issues/171


Add more powerful query and scan ops

Closes #266
The current query and scan APIs return lists of results, discarding some information that may be valuable to the user, e.g. the last evaluated key. I introduced the pagination APIs queryFrom and scanFrom, but it only tried (and failed) to solve one problem. These APIs were ultimately unusable.
In here they both are removed, and more powerful versions are introduced: query0 and scan0 both yield the raw DynamoDB results, allowing users to do whatever they want with it, e.g. accessing the last evaluated key. They could be used to defined scan and query and replace the current "streaming" logic, but that is out of scope for this PR (let's discuss in #275 ).
A useful follow-up will be to provide facilities to extract UniqueKey[K] values from the results, but again something perhaps for v1.1.

[x] add tests

@philwills oh and I forgot the most important 😅 This PR introduces from which allows users to define a starting key for their scan/query ops:
table.from('key -> value).scan()

so now pagination works just like any other existing API (consistently, limit, etc.)
cc /@jcohen

@TBonnin added the tests back here

@philwills well I'm hoping I can keep on soliciting your valuable feedback on key design decisions 😃
On scan0 and query0: I don't like it either, it was a decision motivated by the following reasoning:

the scanFrom and queryFrom were broken, you could not call them without a key, but the only way to get a key was to use them ☯ 😲 Introducing them in the first place felt like a drawback.
people have expressed the need to access information available in the results a couple of times, I think it is a fair ask
I'd like to get v1 out, leaving improvements to the design until v2
there is precedent in delete and deleteAll

Not ideal, I agree with you the raw data types should be hidden away, it felt this was a reasonable compromise to get us there.


libs/libc/stdio: replace double_t to double

The CI check failed because, include *.c not in "Include field",   ignore it.
#ifdef CONFIG_LIBC_FLOATINGPOINT
#  include "stdio/lib_libdtoa.c"
#endif

Modify reason:
When build Nuttx SIM, in x86_64 system:
Compile with gcc option '-m64' (default):
sizeof(double_t) = 8
sizeof(double)   = 8
Compile with gcc option '-mx32':
sizeof(double_t) = 8
sizeof(double)   = 8
Compile with gcc option '-m32':
sizeof(double_t) = 12       // long double
sizeof(double)   = 8
When use '-m32', and print sth. like this:
printf("%f\n", (double)3.0);
SIM will print out: nan
This is because sizeof(double_t) is not equal with double.
Resolve:
replace all double_t to double in libs/libc/stdio.
As a user of '-m32', you should know double_t is one type
long double, and len is 12. And you use use '%lf' to print.
like:
printf("%lf\n", (double_t)3.0);
Currently we don't support '%lf'.
Change-Id: I9b9d11853140d5296dd80416c8ed6a260a9d2d9c
Signed-off-by: ligd<EMAIL_ADDRESS>Summary
Impact
Testing

I've seen this failure before, when downloading source cache. My attempt to handle this is in #1762

@v01d is correct.  We should just drop the cache with changes made on GitHubs side.

@GUIDINGLI  Please rebase against master, this should fix the build failure.

@v01d done


SleepSchedeule refactored and fixed

Refactored code
Fixes #5171
Fixes issue when bot wakes and sleeping again immediately (same sleep entry, but now < next_end because of random offset)
I will rewrite failing test later, for now just skipping it.

👍


new release (v2.2.2)

For https://github.com/Microsoft/LightGBM/issues/985#issuecomment-433717181
@StrikerRUS any warnings we should remove ?

@guolinke There are no warnings removing scheduled for 2.2.2.
However, I suppose that fixes from nomp-err branch should be in master before 2.2.2 (https://github.com/Microsoft/LightGBM/pull/1780#issuecomment-433675009).

@StrikerRUS it this ready now ?


add BTC jam (BTC Lending site)

img is their favicon converted to real png.

only didnt pass Travis due to timeout of travis....

it looks like you added 2 images. One in the PNG format and one in JPEG.

@Carlgo11 you're right, there's only one jpg now 32x32 in the PR.

although we prefer png

@mikeatlas at 32x32 PNG is 99% of the time smaller than jpg.
Your image is jpg, but your code refers to png. Please make it consistent.

I give up trying to help contribute with PRs when you spend more time nitpicking me than accepting them and making tiny tweaks yourself. Appreciate your project immensely but I'll opt out of helping further.
FWIW, here's the same png/jpg, I ran the png version through ImageOptim I went from 1kb JPG to 2kb PNG. So I opted for jpeg. But messed up between push/pull PRs in the interim, my apologies.

@mikeatlas We are not nitpicking, just making sure that data is accurate :)

That picture doesn't look like any logo they use. Their favicon is  and while googling for bitjam I see this logo

I regret trying to help out. Their favicon is only 16x16 and you require 32x32, I picked a logo that looks like what their site uses right now. You can disregard this PR.

While we do appreciate all contributions we need to do our job.
Our job is to make sure things follow the guidelines of this repo before we can merge anything.
It sometimes is a lengthy process for a PR to follow the guidelines but all 703 PRs that have been merged have gone through it.

I've seen your team fix "volunteered" PRs in the past before, instead of just nitpicking submissions
Example: https://github.com/jdavis/twofactorauth/pull/894
Good day.

I don't really see what you're pointing at?
Josh is the creator of this repo and knows best of all what our guidelines are. Even so a contributor did point out an error he made with formatting.

@mikeatlas Sorry about the trouble with your PR.
I'm in the process of rewriting the Contributing.md file to make it more friendly and easier to understand. If you have any suggestions of parts that were particularly confusing, feel free to tell me!
Sometimes we help out with PRs when people are having trouble with it or are new to GitHub. If that's the case, you can just tell us, and one of us can help. I think the reason we didn't do this in your case is that it seemed like you knew what you were doing.
If you don't want to make a new pr, I can take add it to my backlog of sites.
Thanks!

Please do add it to your backlog. http://help.btcjam.com/customer/portal/articles/1987686
I'm not forking a new repo, creating a branch, checking in changes, creating and submitting a PR for you again when you could have easily accepted this and my other PR by dotting a missed "i" or crossing a missed "t".

No problem.
I will talk to the contributors about fixing little things after they're merged instead of while the PR is opened.
I'll also make sure to be more descriptive in the Contributing.md and the wiki to make sure that there's nothing to fix in the first place. :smile:
Again, I'm sorry about how this PR went and I hope you consider contributing again.

my logo choice was fine by the way.
http://blog.btcjam.com/2015/05/07/how-to-use-authy-two-factor-authentication-with-btcjam/


PythonNET integration

Is your feature request related to a problem? Please describe.
Complement python integration and c#-python variable sharing feature request
Describe the solution you'd like
It is unclear when python integration will be implemented, it would be exploratory to evaluate the feasibility of integrating PythonNET as an intermediate step for python integration and c#-python variable use case support
Alternative
Ironypython use case

Hi @GeorgeS2019! Could you help me understand why you'd like to see python added/what use cases you'd like to be able to use c#-python sharing for?

This has been partially addressed here

I'm not seeing any details in the issue linked -- Hoping to understand what you're doing in c#, when you decide you need to move to python, what datatypes you'd expect to be able to share, what you'd then do in python etc. Looking for the description of the use cases themselves


Bump to LLVM green commit from Nov. 14, onnx-mlir main

Commit ce861d7401a9af54e92846001f9724c31f589d33 had to be reverted, since it messed up the build.ninja files, causing ninja to fail when building onnx-mlir
Implemented support for SoftmaxV11Op in ONNXToTorch as that is now showing up

 string(REPLACE "\n" "" SHORT_HASH_OUTPUT_STRIPPED "${SHORT_HASH_OUTPUT_STRIPPED}")
below
 string(REPLACE ".git" "" SHORT_HASH_OUTPUT_STRIPPED "${SHORT_HASH_OUTPUT}")
fixes the missing commit

 string(REPLACE "\n" "" SHORT_HASH_OUTPUT_STRIPPED "${SHORT_HASH_OUTPUT_STRIPPED}")
below  string(REPLACE ".git" "" SHORT_HASH_OUTPUT_STRIPPED "${SHORT_HASH_OUTPUT}") fixes the missing commit

Can we land that upstream?


Bump Dependencies #9320205123

Bumps dependencies for product-is. Link : https://github.com/wso2/product-is/actions/runs/9320205123

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Please refresh article

The original text at https://code-maze.com/csharp-async-enumerable-yield pre-ordered the Nov 2021 .NET latest release so the warning at the bottom "could still change" was valid. However Muhammad updated webpage in Dec 2021 after the release so the caveat no longer applies. Was a good article as topic is still only dimly understood, but could be improved now.
A reminder to clients using HTTP to consume such an API that code should request each piece and not the default "all at once" paradigm. Apart from the timeliness of the data, the piecemeal calls benefit from lower memory consumption.
HTH regards, Dick

Hey @DickBaker, thanks. You seem to read the articles in detail and have a good eye for noticing these things.
Would you consider applying for the Technical Editor role on Code Maze?
We could use someone like you with such a keen eye and willingness to help others.
We're constantly trying to improve the quality of content, so this might be the perfect role for you.


Missing [hash] for vendor.js

Is there a specific reason you don't append a hash to the vendor.js? Can't this lead to cache invalidation issues?
⬇ https://github.com/faceyspacey/redux-first-router-demo/blob/93fabd794b4fe8d599a600e67c092e3334758092/webpack/client.prod.js#L77
All the other outputs do have a hash, e.g.
• https://github.com/faceyspacey/redux-first-router-demo/blob/master/webpack/client.prod.js#L17-L18
• https://github.com/faceyspacey/redux-first-router-demo/blob/master/webpack/client.prod.js#L36
• https://github.com/faceyspacey/redux-first-router-demo/blob/master/webpack/client.prod.js#L51

It's a mistake. I'm pretty sure autodll only speeds up rebuilds on HMR and is there only meant to exist in development. ...but if not, yea there should be hash.


⚠ Jenkins Plugins Artifactory has degraded performance

In 7c8555c, Jenkins Plugins Artifactory (https://get.jenkins.io/plugins/artifactory/) experienced degraded performance:

HTTP code: 200
Response time: 1242 ms

Resolved: Jenkins Plugins Artifactory performance has improved in 7e8025e after 24 minutes.


Allow nested zones

Currently zone only allows figDesc and graphic as children.
TEI allows nested zones, which would be a preferable way to encode a clef's position within a measure box.
<zone type="measure">
  <zone type="clef" />
</zone>

This deviation from TEI was by design.  Nested zones don't add any information not already present in the content markup.  In this case, the MEI markup explicitly encodes the parent/child relationship between measure and clef.  Also, adding coordinates to nested zones muddies the water a bit as it becomes less clear if the coordinates of the child zone are relative to the parent or not.  Making the zones "un-nested", it is (hopefully) clear that the coordinates of the child are NOT relative to its parent zone, but to the surface or graphic containing the zone(s).

I second this deviation from the TEI model. And also think that keeping the facsimile tree a flat (two-dimensional) space is adequate and more clear for describing a two-dimensional space. Just out of curiosity: Does any of you have a use-case at hand for a graphic nested in zone (because graphic is also possible in surface and even in facsimile)?

You can provide images of details within a zone without specifying coordinates, e.g. compare https://tei-c.org/release/doc/tei-p5-doc/en/html/ref-graphic.html
What bothers me more is that it is possible in MEI to nest a zone within graphic. I would like to see a use case for this one.

Ok, that seems to make sense, e.g., if you mark, let's say, the zone of a watermark that is not visible in your primary image file, and you supply a  thermographic detail image of the watermark.

What bothers me more is that it is possible in MEI to nest a zone within graphic. I would like to see a use case for this one.

Picking up the watermark example again you could encode it the other way round, i.e.:
You have a regular RGB and a thermographic image of the whole page:
<surface>
    <graphic target="rgb.jpg">
        <zone label="some area of interest" […] />
    </graphic>
    <graphic target="thermo.jpg">
            <zone label="watermark only visible in this representation" […] />
    </graphic>
</surface>

I'm just exploring, not sure if we should definitely stick to all those possibilities.

I think that the notes on TEI's zone are quite explicit (and imho better than trying to force correct/logical encoding by design):

The position of every zone for a given surface is always defined by reference to the coordinate system defined for that surface.

A graphic element contained by a zone represents the whole of the zone.

And also better than allowing things like:
<facsimile>
  <surface>
    <zone>
      <graphic>
        <zone>
          <graphic>
            <zone>
              <graphic />
            </zone>
          </graphic>
        </zone>
      </graphic>
    </zone>
  </surface>
</facsimile>

There seemed to be consensus a long time ago. Discussed this again at 2022 dev meeting in Berlin. We will not be able to suppress bad use of MEI in all cases, so a simple "don't do that" is sufficient. Not even sure we need to update documentation for this…


BUG: interpolate.griddata: memory leak in linear mode under Python 3.12

Describe your issue.
When running a particular test in our testsuite, specifically under python 3.12, its memory use balloons up to 16GB, while it barely needs 1~2GB under any older Python version.
It's sadly relatively inconclusive, but with all kinds of memory leak testers/profilers, everything points at the griddata interpolation function. Particularly this instance: https://github.com/OGGM/oggm/blob/7489be15c644b54c5f20501c38e654a47c63c53c/oggm/core/gis.py#L797
I was unable to produce any minimal example on how to reproduce this, it only happens in our entire framework of libraries, which makes it really hard to pinpoint down what is leaking where here.
I have already tried running this against the latest 1.14.x branch, since there was a memory leak fixed there recently.
No change in memory balooning behaviour though compared to latest 1.13 version.
I've also tried the binary wheels from pypi vs. our self-compiled version. Both leak in the same way.
I have packaged this all up into a docker image, which is available here, with instructions on how to trigger the issue:
https://hub.docker.com/repository/docker/timoroth/oggm-latest-dev-debug/general
The python version in there is built in debug mode, so pytest-leaks works.
The likely culprit is linked above already.
One interesting and concerning thing I noticed when running this with pytest-leaks in action:
At the end, it exits with the following assertion:
test_merge_simulated_thickness: leaked references: [233775], memory blocks: [136886], file descriptors: [7]
...../include/python3.12d/object.h:798: _Py_NegativeRefcount: Assertion failed: object has negative ref count
<object at 0x7fc6e830f650 is freed>

Like stated before, this really only happens when running under python 3.12. The memory behaviour under python 3.11 or older is absolutely tame.
If you want to verify, run the same reproducer from the Docker-Hub image in this docker image: ghcr.io/oggm/untested_base:py3.11
Reproducing Code Example
See above, sadly I've so far been unable to find any minimal reproducer for this.

Error message
None, if the machine has enough RAM, the test runs to completion successfully.
However, the memory is never released until the python process exits.
When testing for leaks with pytest-leaks, it exits with an assertion about a negative ref count at the very end, after completing the test runs.

SciPy/NumPy/Python version and system information
>>> import sys, scipy, numpy; print(scipy.__version__, numpy.__version__, sys.version_info); scipy.show_config()
1.14.0rc2.dev0+1360.67d743e 1.26.4 sys.version_info(major=3, minor=12, micro=3, releaselevel='final', serial=0)
Build Dependencies:
  blas:
    detection method: pkgconfig
    found: true
    include directory: /usr/include/x86_64-linux-gnu/openblas-pthread/
    lib directory: /usr/lib/x86_64-linux-gnu/openblas-pthread/
    name: openblas
    openblas configuration: USE_64BITINT= DYNAMIC_ARCH=1 DYNAMIC_OLDER=1 NO_CBLAS=
      NO_LAPACK= NO_LAPACKE=1 NO_AFFINITY=1 USE_OPENMP=0 generic MAX_THREADS=64
    pc file directory: /usr/lib/x86_64-linux-gnu/pkgconfig
    version: 0.3.20
  lapack:
    detection method: pkgconfig
    found: true
    include directory: /usr/include/x86_64-linux-gnu/openblas-pthread/
    lib directory: /usr/lib/x86_64-linux-gnu/openblas-pthread/
    name: openblas
    openblas configuration: USE_64BITINT= DYNAMIC_ARCH=1 DYNAMIC_OLDER=1 NO_CBLAS=
      NO_LAPACK= NO_LAPACKE=1 NO_AFFINITY=1 USE_OPENMP=0 generic MAX_THREADS=64
    pc file directory: /usr/lib/x86_64-linux-gnu/pkgconfig
    version: 0.3.20
  pybind11:
    detection method: config-tool
    include directory: unknown
    name: pybind11
    version: 2.12.0
Compilers:
  c:
    commands: cc
    linker: ld.bfd
    name: gcc
    version: 11.4.0
  c++:
    commands: c++
    linker: ld.bfd
    name: gcc
    version: 11.4.0
  cython:
    commands: cython
    linker: cython
    name: cython
    version: 3.0.10
  fortran:
    commands: gfortran
    linker: ld.bfd
    name: gcc
    version: 11.4.0
  pythran:
    include directory: ../../pip-build-env-t7ypm4mq/overlay/lib/python3.12/site-packages/pythran
    version: 0.16.1
Machine Information:
  build:
    cpu: x86_64
    endian: little
    family: x86_64
    system: linux
  cross-compiled: false
  host:
    cpu: x86_64
    endian: little
    family: x86_64
    system: linux
Python Information:
  path: /usr/local/pyenv/versions/3.12-dev-debug/bin/python3.12
  version: '3.12'

It seems like most of the memory growth happens while reproject_gridded_data_variable_to_grid is running. That could potentially still implicate SciPy, though, since it uses Salem to reproject, and that reprojection uses a RegularGridInterpolator.

The reason I report this here is that I ran this through https://pypi.org/project/memory-profiler/ and got this result:

Filename: /root/oggm/oggm/core/gis.py

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
   747    398.0 MiB    398.0 MiB           1   @entity_task(log, writes=['gridded_data'])
   748                                         @profile
   749                                         def process_dem(gdir=None, grid=None, fpath=None, output_filename=None):
   750                                             """Reads the DEM from the tiff, attempts to fill voids and apply smooth.
   751
   752                                             The data is then written to `gridded_data.nc`.
   753
   754                                             Parameters
   755                                             ----------
   756                                             gdir : :py:class:`oggm.GlacierDirectory`
   757                                                 Where to write the data. If set it overrules grid and fpath.
   758                                             grid : :py:class:`salem.gis.Grid`
   759                                                 Grid of the DEM file. Needed if gdir is not provided.
   760                                             fpath : str
   761                                                 The filepath of the DEM file. Needed if gdir is not provided.
   762                                             output_filename : str
   763                                                 The filename of the nc file to add the DEM to. Defaults to gridded_data
   764                                             """
   765    398.0 MiB      0.0 MiB           1       if gdir is not None:
   766                                                 # open srtm tif-file:
   767                                                 dem = read_geotiff_dem(gdir)
   768                                                 # Grid
   769                                                 dem_grid = gdir.grid
   770                                                 grid_name = gdir.rgi_id
   771                                             else:
   772    398.0 MiB      0.0 MiB           1           if grid is None or fpath is None:
   773                                                     raise InvalidParamsError('If you do not provide a gdir you must'
   774                                                                              'define grid and fpath!
I've also ran this same profiling on Python 3.11, and it makes a similar allocation, but the difference is that it's near immediately freed again.
While on 3.12, that memory sticks around forever now, and is never freed.
When extending this profiling further, it just points at this line: https://github.com/scipy/scipy/blob/maintenance/1.14.x/scipy/interpolate/_ndgriddata.py#L323
After which, all code is .pyx, which cannot be profiled like this.
My very superficial guess as to what's happening is something, somewhere decreases a ref counter one too many times, either for some reason only on Python 3.12, or for some reason only Py3.12 is bothered by it.
Based on the assertion pytest-leaks throws.

OK, so it's not a memory leak in some C function from scipy, it's that python in general does not guarantee when things are deallocated, and python 3.12 does something differently from previous versions.
Not sure how much scipy can do here.
To double-check:

can you try replacing the griddata call with a direct call to LinearNDInterpolator (see the griddata source: it's a very thin wrapper).
does the behavior (with LinearNDInterpolator) depend on the scipy version? E.g., if you downgrade to scipy 1.12 or something like that, is there a change?

I'm not sure what it is, but Python is at least aware of the memory in some shape or form. Not sure if it'd account for a stray malloc in some C-Code.
But the fact that it's this different between 3.11 and 3.12 does strongly hint at it being Python-related for sure.
So far I have tried 1.13.x and 1.14.x. The behaviour is the same there. I have not tried 1.12.x yet, and older versions don't support Py3.12 I think.
I have already tried littering the code with calls to gc.collect(). It changed nothing.
Will try directly calling LinearNDInterpolator and on 1.12.x shortly.

Just to make sure, when you say you tried 1.14, the version you tried included https://github.com/scipy/scipy/pull/20404, right? If unsure, retry with the current main or a nightly wheel from https://anaconda.org/scientific-python-nightly-wheels/scipy/files.

after one test is done, its memory should get GC'd

I don't believe the timing is guaranteed by python VM though.
So from the scipy side I'd suggest working out two possibilities:

the cython changes from gh-20404 : it should improve memory consumption which went up in  https://github.com/scipy/scipy/pull/18376;
using LinearNDInterpolator instead of griddata:

interp = LinearNDInterpolator(...)
values = interp(....)

If this does not help by itself, try adding an explicit del interp.
The guess here is that what happens is that the 600mb that memory_profiler shows is the interp object and something keeps a reference to, thus keeping it for longer than strictly necessary.
Like you said, it's technically possible that there is a genuine stray malloc somewhere; the fact that the problem only shows under python 3.12 and not 3.11 makes other reasons more likely.

Just to make sure, when you say you tried 1.14, the version you tried included #20404, right? If unsure, retry with the current main or a nightly wheel from https://anaconda.org/scientific-python-nightly-wheels/scipy/files.

Yes, I explicitly tested 1.14 because of that PR. (See version info in initial report)
I also ran the Benchmark it included (in a loop). It does not leak, but shows a normal GC pattern of ramping up in memory, until the GC kicks in and cleans up.
I also ran that test against 1.13, and unsurprisingly, memory consumption quickly ran out of control.

after one test is done, its memory should get GC'd

I don't believe the timing is guaranteed by python VM though.

Definitely not, but it should be sooner than never/before the OS takes care of it via OOM kill.

So from the scipy side I'd suggest working out two possibilities:
* the cython changes from [BUG: interpolate: fix high memory usage for 2 classes #20404](https://github.com/scipy/scipy/pull/20404) : it should improve memory consumption which went up in  [ENH: CloughTocher2DInterpolator multiple times with different values #18376](https://github.com/scipy/scipy/pull/18376);

I tested this with latest 1.13, and the HEAD of the 1.14.x maintenance branch.
They both show the exact same memory consumption and leaky behaviour, almost down to individual bytes-sizes of allocations.

* using LinearNDInterpolator instead of griddata:

interp = LinearNDInterpolator(...)
values = interp(....)

If this does not help by itself, try adding an explicit del interp.
The guess here is that what happens is that the 600mb that memory_profiler shows is the interp object and something keeps a reference to, thus keeping it for longer than strictly necessary.

I'll try those in my next round of tests and report here.

Like you said, it's technically possible that there is a genuine stray malloc somewhere; the fact that the problem only shows under python 3.12 and not 3.11 makes other reasons more likely.

The wildest guess I have is something, somewhere not managing reference counts correctly.
And Python 3.12 introduces the new concept of "Immortal Objects", which are signaled by a magical refcount.
Though it's unlikely it'd hit that magic number by just decref'ing one too many times.

One thing to correct myself on:
/usr/local/pyenv/versions/3.12-dev-debug/include/python3.12d/object.h:798: _Py_NegativeRefcount: Assertion failed: object has negative ref count
<object at 0x7f3d68875f80 is freed>
Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: finalizing (tstate=0x00007f3e88b692a8)

This also happens without pytest-leaks, just purely by running the test to completion on a debug build of python 3.12.
When directly calling LinearNDInterpolator, not much changes:
   800    429.7 MiB      0.0 MiB           1           try:
   801    429.7 MiB      0.0 MiB           1               inputval = np.ravel(dem[pok])
   802   1101.8 MiB    672.1 MiB           1               ip = LinearNDInterpolator(points, inputval, fill_value=np.nan, rescale=False)
   803   1105.8 MiB      4.0 MiB           1               ipr = ip(inter)
   804   1105.8 MiB      0.0 MiB           1               dem[pnan] = ipr
   805                                                 except ValueError:
   806                                                     raise InvalidDEMError('DEM interpolation not possible.')
   807   1105.8 MiB      0.0 MiB           1           if gdir is not None:
   808                                                     log.info(gdir.rgi_id + ': DEM needed interpolation.')
   809                                                     gdir.add_to_diagnostics('dem_needed_interpolation', True)
   810                                                     gdir.add_to_diagnostics('dem_invalid_perc',
   811                                                                             len(pnan[0]) / (nx * ny))
   812                                                 else:
   813   1105.8 MiB      0.0 MiB           1               diagnostics_dict['dem_needed_interpolation'] = True
   814   1105.8 MiB      0.0 MiB           1               diagnostics_dict['dem_invalid_perc'] = len(pnan[0]) / (nx * ny)
   815
   816   1105.8 MiB      0.0 MiB           1       isfinite = np.isfinite(dem)

This is the same log with scipy downgraded to the 1.13.1 binary wheel from pypi:
   800    517.7 MiB      0.0 MiB           1           try:
   801    517.7 MiB      0.0 MiB           1               inputval = np.ravel(dem[pok])
   802   1187.1 MiB    669.4 MiB           1               ip = LinearNDInterpolator(points, inputval, fill_value=np.nan, rescale=False)
   803   1187.1 MiB      0.0 MiB           1               ipr = ip(inter)
   804   1187.1 MiB      0.0 MiB           1               dem[pnan] = ipr
   805                                                 except ValueError:
   806                                                     raise InvalidDEMError('DEM interpolation not possible.')
   807   1187.1 MiB      0.0 MiB           1           if gdir is not None:
   808                                                     log.info(gdir.rgi_id + ': DEM needed interpolation.')
   809                                                     gdir.add_to_diagnostics('dem_needed_interpolation', True)
   810                                                     gdir.add_to_diagnostics('dem_invalid_perc',
   811                                                                             len(pnan[0]) / (nx * ny))
   812                                                 else:
   813   1187.1 MiB      0.0 MiB           1               diagnostics_dict['dem_needed_interpolation'] = True
   814   1187.1 MiB      0.0 MiB           1               diagnostics_dict['dem_invalid_perc'] = len(pnan[0]) / (nx * ny)
   815
   816   1187.1 MiB      0.0 MiB           1       isfinite = np.isfinite(dem)

I also tried downgrading to 1.12.0 this time, and the results are practically identical to the 1.13.1 results.
With a del thrown in there, nothing changes either:
   800    581.8 MiB      0.0 MiB           1           try:
   801    581.8 MiB      0.0 MiB           1               inputval = np.ravel(dem[pok])
   802   1188.2 MiB    606.4 MiB           1               ip = LinearNDInterpolator(points, inputval, fill_value=np.nan, rescale=False)
   803   1188.2 MiB      0.0 MiB           1               ipr = ip(inter)
   804   1188.2 MiB      0.0 MiB           1               del ip
   805   1188.2 MiB      0.0 MiB           1               dem[pnan] = ipr
   806                                                 except ValueError:
   807                                                     raise InvalidDEMError('DEM interpolation not possible.')
   808   1188.2 MiB      0.0 MiB           1           if gdir is not None:
   809                                                     log.info(gdir.rgi_id + ': DEM needed interpolation.')
   810                                                     gdir.add_to_diagnostics('dem_needed_interpolation', True)
   811                                                     gdir.add_to_diagnostics('dem_invalid_perc',
   812                                                                             len(pnan[0]) / (nx * ny))
   813                                                 else:
   814   1188.2 MiB      0.0 MiB           1               diagnostics_dict['dem_needed_interpolation'] = True
   815   1188.2 MiB      0.0 MiB           1               diagnostics_dict['dem_invalid_perc'] = len(pnan[0]) / (nx * ny)
   816
   817   1188.2 MiB      0.0 MiB           1       isfinite = np.isfinite(dem)

The more I look at this, the more it seems like the GC has been completely short-circuited or something.
When I run the same kind of tests under python 3.11, there is the occasional line with a negative memory delta, indicating that the GC ran in the meantime.
But on 3.12, there are only 0 or positive numbers, with the VERY rare small negative number, likely caused by native code doing something.

OK, this is starting to look like a refcounting error somewhere indeed. The next step to find where this somewhere is to trim down the reproducer. Does it fail if you take pytest out of the equation and run a standalone script? If you trim it down to just calling LinearNDInterpolator with canned data?

Not sure if it might be related to this comment which mentions a recent (Sep 2023) Cython bug when used with Python 3.12

That bug should long be fixed though, and I'm relatively certain that I built everything with latest stable cython, which should have the fix.
What I'd like to stress is: nothing seems to be getting GC'd anymore. The graph of memory usage only knows one way over the entire runtime: up, it never goes down once.
So I'm not even sure if this really is scipy, or scipy is just the most visible victim of something else royally messing up the GC.
That would also explain why I've so far been completely unable to create any kind of minimal reproducer for this.
Every time I try to extract parts of this process with canned data, it works as expected.

@TimoRoth any luck further chasing down this issue?

Negative, but I recently verified it's still happening.
And it is not happening when using conda packages, which makes it all the more mysterious.

I don't know what got updated where, but in a fresh venv with everything fully updated, I am now unable to reproduce this on python 3.12 and 3.13. I don't know what fixed it, let alone what caused it.
Bit unsatisfying, but at least it works now, and we're not indefinitely stuck on 3.11.

in a fresh venv with everything fully updated, I am now unable to reproduce this on python 3.12 and 3.13.

So to double-check: once you use a fresh venv, there is no memory leak?
Meaning the OP was indeed some unfortunate glitch of unknown origin, and all is well unless that specific venv is used.
If so, we can declare that the balance is restored in the force and close this issue as a one-off.
Do feel free to discuss further and please reopen if the problem persists.

Yeah, some update of something must have fixed this.
I still have an environment with the issue stashes away as docker image, so I could slowly travel through time with it, and see which update fixes it. But so far I didn't have the motivation to do so. I'm just glad we're not stuck on 3.11.


Slimerjs no longer supporting downloads to Travis

Since a few days ago, Travis has stopped working on my branch, getting 403 errors trying to download the slimerjs package. This seems to be because the slimerjs server has blocked access from AWS and hence Travis; see https://github.com/graingert/slimerjs/issues/39.
I'm assuming that it stopped working on my branch because a cached copy expired, and that it will stop working on ol master whenever that cached copy expires.
The slimerjs people are saying that a copy should be available in the user dir instead of Travis downloading it every time from their server - it is a large package. Any ideas on how best to implement this?

I'd like to try moving back to Karma to run tests in various browsers.  Travis has Firefox installed (https://docs.travis-ci.com/user/gui-and-headless-browsers/#Using-xvfb-to-Run-Tests-That-Require-a-GUI).  We could configure Karma to run our tests on Firefox there.  It would be nice to get rid of the Phantom and Slimer dependencies.

I'll try to put up a branch with Karma later today.

Plus one for karma!

slimerjs 0.960.2 should fix your problem for now

Great.  If someone can solve any immediate problems with SlimerJS 0.960.2, please go ahead.  The Karma work is going to be more involved, so I'll just continue on that separately.

https://github.com/openlayers/ol3/pull/5479

Fixed with #5479.

yup, Travis now works again on my fork 👍


Neopixel Module Not Available

Apologies if you're already aware of this bug but it seems the neopixel module does not exist in MicroPython on V2.
Maybe an issue linked to https://github.com/microsoft/pxt-microbit/issues/3415

This module is also requested via micro:bit support in https://support.microbit.org/a/tickets/40468

@microbit-mark @microbit-carlos Needed for testing "display_and_neopixel.py"

Yes, this is something we wanted to implement in the CODAL layer, so I've opened https://github.com/lancaster-university/codal-microbit-v2/issues/19.

The older implementation doesn't co-exist with Bluetooth so I think it does make sense to use the PWM-based one. @finneyj any thoughts on merging that so we can ensure Neopixel in MicroPython?  Would it require a dedicated PWM channel?

Yes, I agree we should build a PWM-based driver codal side. It does require a dedicated PWM peripheral, but I have one reserved for this already, so no problem.
Sam's driver needs a bit of work as I recall to package so that it is memory safe during DMA and double buffers to avoid memory overhead/glitching. I'll happily do that when I get a minute.

Ideally CODAL would expose a single function like neopixel_write(pin, len, buf) that would output the given buf of data on the pin using neopixel encoding.

NeoPixel support has been added to CODAL: https://github.com/lancaster-university/codal-nrf52/blob/master/inc/neopixel.h
Thanks @finneyj !

no worries @dpgeorge @microbit-carlos.
Please note this is a blocking call (it will block the calling fiber until playout is complete), but doesn't disable interrupts or anything like that, so shouldn't have any of the side effects associated with the old bit banging implementations.
Under the hood it all runs async, so see here if you prefer it to just run in the background. You just need to flip play() to playAsync():
https://github.com/lancaster-university/codal-nrf52/blob/master/source/neopixel.cpp#L25
https://github.com/lancaster-university/codal-nrf52/blob/master/source/WS2812B.cpp#L129

Thanks @finneyj !  A blocking call is good enough for now.
Implemented on the Python side in 915b6aa746ce35d40862b973c4dd519ff11109f2, should be compatible with micro:bit v1, and also includes support for RGBW LEDs.

@microbit-mark @martinwork this should be ready for testing on the next MicroPython release.

@dpgeorge the ws2812_write function is also exposed via microbit module (microbit.ws2812_write), is that expected?

the ws2812_write function is also exposed via microbit module (microbit.ws2812_write), is that expected?

Yes that's expected, although at this stage I wouldn't document the function as it may change and/or be removed in the future.
We could move it out of the microbit module into something like _microbit intended for hidden functions if that is a better approach.


allow GIT_SECRET_SERVER to be url and use its scheme in git credential

In order to support http urls for git servers, credentialhelper needs to know the scheme of the url
This change implements support to allow GIT_SECRET_SERVER to be passed through as it is literaly defined in jx-requirements.
It is backwards compatible to support domain names only.
Tests verify both backwards compatibility and the new functionality

Hi @cameronbraid. Thanks for your PR.
I'm waiting for a jenkins-x or todo member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.
Once the patch is verified, the new status will be reflected by the ok-to-test label.
I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the jenkins-x/lighthouse repository.

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To complete the pull request process, please assign jstrachan
You can assign the PR to them by writing /assign @jstrachan in a comment when ready.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

/ok-to-test
/lgtm

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: jstrachan
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [jstrachan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


[VS Code] Joining collaboration session: The specified value isn’t a valid Live Share URL. Please check the link provided by the host and try again.

Error:
Joining collaboration session: The specified value isn’t a valid Live Share URL. Please check the link provided by the host and try again.
Steps to Reproduce:

Version Data

extensionName
VSLS

extensionVersion
1.0.18

protocolVersion
2.2

applicationName
VSCode

applicationVersion
1.32.3

platformName
MacOS

platformVersion
18.2.0

Closing as dupe of 1894. Please reopen if your repro steps are different than the ones described there.


Wrong unit of measure for Binford's Anthropometry > Heights B039 B040

The values like "1347.0 - 1410.6 cm" don't make sense. Maybe mm is correct(?) ;)

Hi Hans,
Yes, thanks. It should be mm. I thought this was corrected in the latest codebook. I will check again, and send a corrected version.
Kate
On 15 Apr 2016 12:24 pm, Hans-Jörg Bibiko<EMAIL_ADDRESS>wrote:

The values like "1347.0 - 1410.6 cm" don't make sense. Maybe mm is correct(?) ;)
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub

Revised Codebooks and "VariableList" files sent. Should allow issue to be closed. Thanks for catching.

They're just really tall!


Implement magic_antenna_check tool

Add magic_antenna_check as a tool

Performs antenna violation checks using magic
Returns antenna_violations, the number of violations that have occured

Example datasheet entry:
  magic_antenna_check:
    description: Run antenna violation check using magic
    display: Antenna Checks
    spec:
      antenna_violations:
        maximum:
          value: 0
    tool:
        magic_antenna_check

Hi @RTimothyEdwards, I have one question regarding querying the number of antenna violations.
Before I run antennacheck, I clear all feedback entries with feedback clear, afterwards I run feedback count to get the number of antenna violations that occurred. Interestingly, for just one antenna check I get a high number of violations. Do you perhaps know why this is the case? Is it counting all the rectangles for that one violation?
magic stdout:
Running antenna checks.
antennacheck finished.
violations = 69

magic stderr:
Cell: ota-5t
Antenna violation detected at plane metal3
Effective antenna ratio 2645.66 > limit 400
Gate rect (780 -1720) to (880 -720)
Antenna rect (-35559 -194046) to (-29926 -193986)

@mole99 :  Yes, if you use the feedback entries from antennacheck, they are intended to be used in magic to mark the entire net (or the portion of the net) that is in violation.  If you want to report individual violations, it works better to use antennacheck debug and then capture and parse the resulting output.

Thanks Tim! I'm going to match stderr then.


Feature Request: negation for the file kind condition

Is your feature request related to a problem?
I'd like to use a flag/action for supporting negation for finding a string within a condition, then it will be possible to declare those condition within the yml rather than using a shell script or similar.
Solution you'd like
if notfind look for the given value in the given file, then something like the below snippet could work:
...
conditions:
  goDefaultVersion-check:
    kind: file
    sourceid: latestGoVersion
    spec:
      file: .go-version
      notfind: '{{ source "latestGoVersion" }}'

Alternatives you've considered
As today I can get a similar behaviour by using the shell with a script call, something like:
#!/usr/bin/env bash
if grep -q "$1" .go-version ; then
  echo "found $1. No need to do nothing"
  # return false
  exit 1
fi
echo "not found $1. Need to update the file"
# return true
exit 0

...
conditions:
  goDefaultVersion-check:
    name: Check if defined golang version differs
    kind: shell
    sourceid: latestGoVersion
    spec:
      command: bash test-golang-version.sh

or use a trick too like the below one:
...
conditions:
  goDefaultVersion-check:
    name: Check if defined golang version differs
    kind: shell
    sourceid: latestGoVersion
    spec:
      command: 'grep -v -q {{ source "latestGoVersion" }} .go-version #'

# helps to use the grep with the right arguments
----------
The shell 🐚 command "grep -v -q 1.19.5 .go-version # 1.19.5" ran successfully with the following output:
----

Anything else?
I tried with matchpattern but there is no support for negation in https://pkg.go.dev/regexp/syntax

The condition accepts the boolean failwhen which allows to revert the expected result as in the following examples
conditions:
  jenkinsexist:
    kind: jenkins
    spec:
      version: '2.235'
      release: weekly
    failwhen: false
  jenkindonotexist:
    kind: jenkins
    spec:
      version: '2.235'
      release: weekly
    failwhen: true
  wrongjenkinsexist:
    kind: jenkins
    spec:
      version: '2.999'
      release: weekly
    failwhen: false
  wrongjenkindonotexist:
    kind: jenkins
    spec:
      version: '2.999'
      release: weekly
    failwhen: true

I opened this issue to track that I need to document this https://github.com/updatecli/website/issues/649

I am closing this pullrquest as I think the problem is resolved using failwhen.
Feel free to reopen if I am missing something

Thanks @olblak!! 🙇


feat(gwyneth): initial draft of the issue desc.

Draft of the described, simplified protocol here: https://github.com/taikoxyz/taiko-reth/issues/22

fix PR comments
make it buildable /tests
make deployments + proposeBlock script
maybe make install (install kurtosis if not installed on mac or linux + runnig a node + deploy L1 smart contracts, see if i can config static PORT... (?) )
integrate into CI (?)

@Brechtpd Makefile is added to packages/protocol.
It currently only up your local image (taiko_reth) based network. If the local taiko_reth named image not found, it builds it with that name. If found, uses that. (So in case you change something in the codebase but you have the image from a prev. version, you gotta rebuild it with same tag yourself before using the setup script.)
Also currently the deployment (DeployL1Locally.s.sol, but you can deploy yourself with the 1-liner command in local_deployments.md) is not available because default Kurtosis is running with random, ephemeral ports, so i either need a custom script to detect the RPC or dive deeper into Kurtosis if possible to use static port binding.

...currently the deployment (DeployL1Locally.s.sol, but you can deploy yourself with the 1-liner command in local_deployments.md) is not available because default Kurtosis is running with random, ephemeral ports, so i either need a custom script to detect the RPC or dive deeper into Kurtosis if possible to use static port binding.

Ok, make install now works, and deploys the contracts. (Port is still dynamic, but parsed out from the Kurtosis logs to support deployment.).
Also tested on ubuntu 20.04, and setup works.
Firing away the first proposeBlock() transaction can be done via the script:
forge script --rpc-url http://<IP_ADDRESS>:YOUR_PORT scripts/L2_txn_simulation/ProposeBlock.s.sol -vvvv --broadcast --private-key <PK_FROM_SAMPLE_ENV_IS_FINE> --legacy


state trust lands on reservations updates

These updates contain all the changes for the story about state trust lands on reservations – both updated data layers and code.

@mariaparazorose this PR is re: the parcels-on-reservations piece, right? Can we do in a new repo instead?

Yes it is! And a new repo makes sense -- should we do this after Annie reviews?

Migrating this PR to a new fork of this project


Update README.md

Hey Pro Dev!
Can you please fork the repo before editing it and send pull requests?
You are editing original repo. Please Pro fork the repo and edit it

Yours
Vc fight chordi hai bro wala bro

@NotReallyShikhar Op

wtf


-o option doesn't seem to work on CLI

It isn't too big a deal, since you can just use a pipe to save the output from the entab function, but I can't get the -o option to work. For example, if I run:
entab -i ~/entab/entab/tests/data/carotenoid_extract.d/dad1.uv -o test.txt
I get back:
No such file or directory (os error 2)

#####

I'm also sometimes getting:
Bad file descriptor (os error 9)
(I also get the same results if I write the a full path after the -o).
However, it works fine if I pipe it:
entab -i ~/entab/entab/tests/data/carotenoid_extract.d/dad1.uv > test.txt

It was calling File::open and since the file didn't exist to open, it errored. Fixed in 854d9235c5ea28641288f002277f955b284b8f03 by changing to File::create; this will error if the file already exists, but without an explicit override flag that's probably the desired behavior. I also cleaned up the "unused enum" warnings.

I was looking at this again & while it works now it seems like using the "-o" command is much slower than just using the pipe. For example, extracting the carotenoid_extract.d file takes about 3 seconds on my computer with the pipe but t takes about 17 seconds if I use the -o flag. Any idea why this might be?

Nice catch. They're handled almost identically in the code, but I think the underlying Stdout is buffered while File isn't and that can lead to a decent performance differential.


wget returning 404 on valid requests in python3

For whatever reason the much more common "wget" module for python is returning 404s when attempting to download the cef binaries even though the URLs are definitely correct.
It may be worth noting in the README, as per this Stack Overflow Question, using "python3-wget" resolves the issue without having to change anything in the build.py

@murt Thank to you I just realized that I forgot to copy prerequisite section from the original project:
https://github.com/stigmee/install#step-2-install-python3-packages

@murt Ok I fixed the doc. Please re-open the ticket if this documentation is still unclear.

Installed the recommended package "wget" instead of "python3-wget" as stated in the README, still getting this error.
urllib.error.HTTPError: HTTP Error 404: Not Found
File link downloads fine through the webbrowser.

Turns out that URL percent encoding is causing the issue. I'm going to open a PR with the fix, and while I am at it, I'll update the CEF version, too.

@raphipod I tried quickly: I uninstalled python3-wget and installed back wget and the + seems working but not the %2B (by the way, when you copy the URL from your browser you have %2B). At the opposite, from my memory, python3-wget does not like + but prefer %2B, and both uses import wget and requirements.txt generates wget for python3-wget :( :(

Concerning the URL encoding issues, I really don't know why it fails at my end

I think this is a mix up between python3_wget and wget while using import wget

Yeah, we should use python3-wget, already corrected that in PR#2

@raphipod @murt I have radically changed of package for downloading :) no more wget. I added the requierement file and updated README. I also updated CEF version. I hardly checked. Tell me if this is ok for you. If ok @raphipod you can cancel your PR

I just noticed that you've commented out the colon in line 346, it exits.

All right, it downloads now without errors! If @murt can confirm that too.


WIP: Generate docs from macros.

Which issue does this PR close?

Closes #.
Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

@Omega359 @alamb I tried to play with custom attributes to wrap up the documentation on top of the what @Omega359 already built. I'm experimenting with just 2 fields(description and examples) and the attribute has to be placed on top of the struct.

WDYT should we move in this direction?

I find this intriguing and technically very interesting! I do have a few questions:

What are you goals with this? I am assuming cleaner looking/easier to create docs. If so I would wonder how it would help in cases were there is significant amounts of docs vs boilerplate such as in the to_date udf.
Can it be made to handle n arguments?
Is it possible to generate rustdoc with this approach?
I noticed that this approach creates a Documentation object on every call. It's obviously pretty minor but it would be nice if it could be static.

To me in general while I find this quite interesting I wonder if the benefit outweighs the added complexity of the code generation. To be clear I am absolutely not against this approach - I just have a few concerns.

@alamb @Omega359 please have a look on real example to_date (I still need to include argements to be called with the builder). As you can see it is the same approach as before, the only difference the documentation is set by attributes rather than code, but it generates documentation builders as before

Thanks @alamb I'll consider the changes, @Omega359 is it good to have a documentation method on struct level instead of impl?

@Omega359 is it good to have a documentation method on struct level instead of impl?

I can't think of a reason why not off the top of my head. I'm hoping to have time to look over your latest updates in this PR later tonight

Thanks @Omega359 there is bunch of things needed to be added, its not a final PR. I'll add some minor missing pieces and the PR will be ready for the review, most likely next week. Because I need to make a cleanup, remove test structures, polish crates and everything.

I'm closer to this

We should also take care on handing things like https://github.com/apache/datafusion/pull/13367

@alamb @Omega359 the PR is ready for review

Done,

Maybe we could call the function something like macro_doc or derived_documentation to make it easier to find (aka so someone who runs across it can easily grep and find the macro definition)

Good idea, maybe we can name it as user_documentation. I'd prefer to leave it to the following PR though, because it will affect dozens of files

Another thing comes to my mind, we can automate transforming current API documentation approach to attribute based one wherever possible. There are more complicated options with macros, we can leave them with API approach for now

Let's wait to see what @Omega359 has to say too, but I would be inclined to merge it in (as it is an optional feature) once we have a bit more documentation

I'll hopefully be able to take a good look at this tonight 🤞

Sorry, I've been busy lately. I'm hoping to get to reviewing this PR this week

Thanks @Omega359 I will be adding the udfs and alternate syntax in follow up PR. I'll find the UDFs supporting both related_udfs and alternative_syntax and make these changes simultaneously along with some other things. Preferring the follow up PR as this one is already too large to review

I'm fine with that @comphead - we can file followup issues to add the alternative syntax, etc


Update perf module as Tachyon-0.6.0 released

Update perf module with Tachyon-0.6.0 (actually the version is 0.7.0-SNAPSHOT)
A few change in pom.xml, and the remaining is in perf/

@swatDong  Thanks for doing this. I just merged the master branch into the perf branch. Do you mind updating the branch based on that?

@haoyuan I've tested this on the latest master branch and this PR is based on it.
The only text changed in origin tachyon is the 3 lines in pom.xml, so it has no conflict.

We can’t automatically merge this pull request.
Use the command line to resolve conflicts before continuing.

I'll check it.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Tachyon-Pull-Request-Builder/2758/
Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Tachyon-Pull-Request-Builder/2759/
Test PASSed.


[generic] None: Unable to download webpage: HTTP Error 404: Not Found (caused by <HTTPError 404: 'Not Found'>)

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

[X] I'm reporting that a supported site is broken
[X] I've verified that I'm running yt-dlp version 2023.03.04 (update instructions) or later (specify commit)
[X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue
[X] I've read about sharing account credentials and I'm willing to share it if required

Region
France
Provide a description that is worded well enough to be understood
My other laptop has no issues downloading YouTube videos using the yt-dlp version (2023.3.4). Only difference I can deduce is that I recently installed oh-my-zsh, p10k, and some zsh plugins such as autocomplete and iterm2.
I tried installing/uninstalling using pip and brew but with no luck.
URL: https://www.youtube.com/watch?v=WV-XLYoIWA4
Provide verbose output that clearly demonstrates the problem

[X] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[ ] If using API, add 'verbose': True to YoutubeDL params instead
[X] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output
[debug] Command-line config: ['-vU', 'https://www.youtube.com/watch\\?v\\=WV-XLYoIWA4']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version<EMAIL_ADDRESS>[392389b7d] (pip)
[debug] Python 3.11.3 (CPython x86_64 64bit) - macOS-11.7.1-x86_64-i386-64bit (OpenSSL 1.1.1t  7 Feb 2023)
[debug] exe versions: ffmpeg 5.1.2 (setts), ffprobe 5.1.2
[debug] Optional libraries: Cryptodome-3.17, brotli-1.0.9, certifi-2022.12.07, mutagen-1.46.0, sqlite3-2.6.0, websockets-10.4
[debug] Proxy map: {}
[debug] Loaded 1786 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Available version<EMAIL_ADDRESS>Current version<EMAIL_ADDRESS>yt-dlp is up to date<EMAIL_ADDRESS>[generic] Extracting URL: https://www.youtube.com/watch\?v\=WV-XLYoIWA4
[generic] watch\?v\=WV-XLYoIWA4: Downloading webpage
ERROR: [generic] None: Unable to download webpage: HTTP Error 404: Not Found (caused by <HTTPError 404: 'Not Found'>); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "/usr/local/Cellar/yt-dlp/2023.3.4/libexec/lib/python3.11/site-packages/yt_dlp/extractor/common.py", line 694, in extract
    ie_result = self._real_extract(url)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/yt-dlp/2023.3.4/libexec/lib/python3.11/site-packages/yt_dlp/extractor/generic.py", line 2371, in _real_extract
    full_response = self._request_webpage(url, video_id, headers={
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/yt-dlp/2023.3.4/libexec/lib/python3.11/site-packages/yt_dlp/extractor/common.py", line 839, in _request_webpage
    raise ExtractorError(errmsg, cause=err)

  File "/usr/local/Cellar/yt-dlp/2023.3.4/libexec/lib/python3.11/site-packages/yt_dlp/extractor/common.py", line 821, in _request_webpage
    return self._downloader.urlopen(self._create_request(url_or_request, data, headers, query))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/yt-dlp/2023.3.4/libexec/lib/python3.11/site-packages/yt_dlp/YoutubeDL.py", line 3742, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File<EMAIL_ADDRESS>line 525, in open
    response = meth(req, response)
               ^^^^^^^^^^^^^^^^^^^
  File<EMAIL_ADDRESS>line 634, in http_response
    response = self.parent.error(
               ^^^^^^^^^^^^^^^^^^
  File<EMAIL_ADDRESS>line 563, in error
    return self._call_chain(*args)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File<EMAIL_ADDRESS>line 496, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File<EMAIL_ADDRESS>line 643, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

[debug] Command-line config: ['-vU', 'https://www.youtube.com/watch\?v\=WV-XLYoIWA4']

The URL is not valid. Remove the \ es


support function name as component name property

Clear and concise description of the problem
export const IconSimple = defineComponent(function IconSimple(props: IconSimpleProps) {
  return () =>
icon simple1111 {props.count}

})

console.log(IconSimple.name)

it is better set a function name as component name
Suggested solution
it is better set a function name as component name
Alternative
No response
Additional context
No response
Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guide.
[X] Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

Unable to set name automatically at runtime, but you can do it manually.
const Comp = defineFunctionalComponent(
  (props: Props) => {},
  {
    name: 'Comp',
    // other options, e.g. name, inheritAttrs, etc.
  },
)
export const Comp = defineSimpleComponent<Props>({
  name: 'Comp',
  setup() {},
})


Can you confirm the sample code on this page is correct?

Hi,
I am seeing different exceptions being thrown than the ones indicated on this page.  Is it possible to confirm the instructions on this page are correct and up to date with the latest botbuilder 4?
thanks,
Tom

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: a57254c9-963d-ea1d-ecaf-dd0e7913526c
Version Independent ID: 610c8514-bd72-46cc-75f1-991e2242019b
Content: Rate limiting - Teams
Content Source: msteams-platform/bots/how-to/rate-limit.md
Product: msteams
GitHub Login: @laujan
Microsoft Alias: anclear

I am checking with the engineering team. I will let you know what they say.

Hello @tfischer. The article has been updated. Thank you!


Implement new fee service

[ ] Support paying fee with non-native tokens
[ ] Showing fee in both $ and token amount
[ ] Support paying fee with multiple tokens

Phase 1: allow users to choose fee rate, other improvements will follow in the subsequent updates


How setup connection to TL using Proxy?

How setup connection to TL using Proxy?

need proxy seetings

need proxy settings


Implement an ntci::Proactor using io_uring

This PR adds initial support for an ntci::Proactor implemented using io_uring on Linux.
io_uring is an alternative system call interface that allows user space to initiate certain system calls in batches , to execute asynchronously, and learn about completed system calls in batches, attempting to reduce the system call overhead required for, e.g. one system call per sendmsg, recvmsg, etc. Userspace may learn about completed system calls without a system call itself, or it can make a system call to block until a minimum number of previously submitted system calls complete. A more thorough description of io_uring can be found in its Introduction.
In the design of NTC, an ntci::Proactor is an abstraction of an interface to an operating system whereby a user "proactively" initiates socket operations (e.g. connect, accept, send, receive) and is asynchronously notified when those operations complete. This PR implements the ntci::Proactor abstract class using io_uring.
The io_uring interface to the kernel consists of a set of three system calls, one to create a ring, one to configure/control a ring, and one to submit operations/reap completions. Userspace writes system call submissions to a ring buffer and then performs a system call to notify the kernel to drain the submission queue. The kernel writes the status of completed system calls to a separate ring buffer, which is drained by userspace by a system call. Userspace may submit and reap completions simultaneously. The submission queue ring buffer and completion queue ring buffer is memory mapped into userspace according to the ring buffer configuration specified by the kernel when the ring is created. Given this design, it is possible for userspace to utilize io_uring functionality with nothing by support for syscall and mmap by libc. This implementation leverages these characteristics so that it may be built on any Linux machine, even those machines that do not support io_uring (such as older kernels.) To achieve this, this implementation eschews both the linux/io_uring.h header and any support library, such as liburing.
io_uring is conceptually simple but has many aspects that make it challanging to figure out how to use correctly and efficiently:

io_uring has evolved significantly since its first release, gaining many features over time. Not all kernel versions support all features. Some features can be detected with io_uring's capabilities system, but not all (e.g. support for certain flags when submitting operations.) This implementation will probably work on Linux kernel versions >= 5.11 (it has only been tested on 6.2) but future work is necessary to support and test all functionality on older kernels. That is left for a future PR.

The principle argument for io_uring seems to be the efficiency gains for submitting recvmsg and sendmsg in batches, avoiding the system call overhead per-operation. While batching recvmsg may make some sense, it is unclear why batching sendmsg calls makes sense: this is trading for greater system call efficiency by increasing latency and reducing throughput. Moreover, an ntci::Proactor is designed to be safely used by multiple threads concurrently calling ntci::Proactor::send, ntci::Proactor::receive, and ntci::Proactor::wait. If a thread is blocked in wait for at least one event to complete, another thread calling send or receive  must submit and enter the ring immediately, otherwise the kernel never learns about the submission, never initiates the operation, which then never completes, and the thread blocked in wait remains blocked forever. This implementation chooses to always submit sendmsg operations and immediately enter the ring, and only defers entering the ring for other operations when it can be safely determined that only one thread will be calling ntci::Proactor::wait and that thread is the one calling ntci::Proactor::send, ntci::Proactor::receive, etc.

io_uring has evolved support for a number of ways of timing out when waiting for operation to complete. Each are a bit clumsy to use and inconsistent in what they support: userspace may submit a "timeout" operation with a relative timeout or (on some kernels) an absolute timeout, with (on some kernels) a specified clock type. Such "timers" need to constantly removed or at least re-submitted. New kernels support a direct specification of a timeout when entering the ring, but it is not clear if absolute timeouts are or will be supported, or any clock type other than CLOCK_MONOTONIC.

Very recent kernels support canceling all pending operations for a file descriptor, but older kernels do not. On older kernels, this implementation must maintain a data structure of all pending operations for a socket, so they can be cancelled when the user wishes to close the socket.

It is unclear if two threads can enter the ring simultaneously, where both are instructing the kernel to drain the submission queue. This implementation relies on that being safe, which should be reviewed or revisited in later improvements. If two threads cannot enter a ring simultaneously to submit, it is unclear how to use io_uring from multiple threads at all: the obvious (but perhaps naive) solution would be to maintain a thread safe copy of the submission queue, that sits in front of the ring's submission queue,  where threads initiating operations perform a thread-safe push onto this queue, then "kick" the ring with a "no-op" operation, waking up a thread blocked on entering the queue, so that thread can go right back into entering the queue again to actually submit the operations to kernel. This seems contradictory to the design and goals of io_uring.

The implementation of ntco_ioring is lightly tested in its component test driver, but is folded into the NTC proactor framework and more thoroughly tested in the ntcf_system test driver.
For now, I consider support for io_uring in NTC as experimental. As such, it is disabled by default in the build configuration. Builders may enable it by specifying configure --with-ioring during the configuration step of the build process. Further support is detected at runtime; if a machine does not support the io_uring system calls the "ioring" proactor will be automatically disabled.
Initial performance tests are underwhelming. The existing epoll backend out-performs this io_uring backend for most workloads, though a small few do show io_uring winning on latency and throughput. Future work should be planned to optimize this implementation.

There are multiple structures which are defined manually so that this code can be built on machines with old Linux kernel. But there are no tests which validate that these structures are defined properly. I think it can be done in compile time (maybe in the test driver) if the library is built on a system with modern kernel.

A comment probably mainly for myself: I am reading https://kernel.dk/io_uring.pdf and there is such statement:
However, since the sqe lifetime is only that of the actual submission of it, it's possible for the application to
drive a higher pending request count than the SQ ring size would indicate. The application must take care not to do so,
or it could risk overflowing the CQ ring.

I haven't yet found if there is some protection implemented here


IDEA 2020.2 EAP incompatible

Short description
The plug-in is shown as incompatible
Expected behavior
Is there a specific expected behavior?
Version information

IDEA Version: 2020.2 EAP

Caused because plugin build contains
<idea-version since-build="201.5985" until-build="201.*"/>

I have tried the workaround:

💥 The latest IntelliJ IDEA version (2020.1) contains a BUG 🐞 for several plugins (including plugin for Lombok) were shown as incompatible with the updated version of the IDE. 💥
You can find the issue with a detailed explanation here: https://youtrack.jetbrains.com/issue/IDEA-237113.

But then the version 0.9.* is installed. If the  until-build="201.* is removed it works.
❓ Is is possible for your needs to use updateSinceUntilBuild false or patchPluginXml.untilBuild(null) to disable the until generation for latest IDEA version (2020.*)

Short description
The plug-in is shown as incompatible
Expected behavior
Is there a specific expected behavior?
Version information

IDEA Version: 2020.2 EAP

Caused because plugin build contains
<idea-version since-build="201.5985" until-build="201.*"/>

I have tried the workaround:

💥 The latest IntelliJ IDEA version (2020.1) contains a BUG 🐞 for several plugins (including plugin for Lombok) were shown as incompatible with the updated version of the IDE. 💥
You can find the issue with a detailed explanation here: https://youtrack.jetbrains.com/issue/IDEA-237113.

But then the version 0.9.* is installed. If the  until-build="201.* is removed it works.
❓ Is is possible for your needs to use updateSinceUntilBuild false or patchPluginXml.untilBuild(null) to disable the until generation for latest IDEA version (2020.*)

Yes you are correct.
Using updateSinceUntilBuild false can solve this problem and I see nothing to pay for it.
I learned this trick from alibaba/p3c
Seems it is a "common secret" or something.
I really do not understand why jetbrains not make this in the doc for beginners.

It is documents for Gradle setup in SDK Documentation
https://www.jetbrains.org/intellij/sdk/docs/tutorials/build_system/gradle_guide.html#patching-the-plugin-configuration-file

It is documents for Gradle setup in SDK Documentation
https://www.jetbrains.org/intellij/sdk/docs/tutorials/build_system/gradle_guide.html#patching-the-plugin-configuration-file

thanks. my bad then.

No Problem.
so much doc to overlook it 😉

Thank you, @MCMicS, for providing the workaround. Works perfectly 👍

But notice it can only run at the latest version, but not compile at this version.
If you compile it using the latest snapshot, you will get some error like this:
9:38:40 PM: Executing task 'buildPlugin'...

Starting Gradle Daemon...
Gradle Daemon started in 7 s 102 ms

> Configure project :
IdeaVersion is : LATEST-EAP-SNAPSHOT
ArtifactVersion is : 0.30-EAP
Building for IntelliJ version: LATEST-EAP-SNAPSHOT
Generating Parser for GrammerKit: 2019.3
Generating Lexer for JFlexRelease: 1.7.0-1

> Task :generateLombokConfigParser FAILED

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.3/userguide/command_line_interface.html#sec:command_line_warnings
1 actionable task: 1 executed
Exception in thread "main" java.lang.NoClassDefFoundError: it/unimi/dsi/fastutil/objects/Object2ObjectOpenHashMap
	at com.intellij.openapi.extensions.PluginId.<clinit>(PluginId.java:18)
	at com.intellij.openapi.extensions.impl.ExtensionsAreaImpl.doRegisterExtensionPoint(ExtensionsAreaImpl.java:176)
	at com.intellij.openapi.extensions.impl.ExtensionsAreaImpl.registerExtensionPoint(ExtensionsAreaImpl.java:161)
	at com.intellij.core.CoreApplicationEnvironment.registerExtensionPoint(CoreApplicationEnvironment.java:257)
	at com.intellij.core.CoreApplicationEnvironment.registerExtensionPoint(CoreApplicationEnvironment.java:244)
	at com.intellij.core.CoreApplicationEnvironment.registerApplicationExtensionPoint(CoreApplicationEnvironment.java:262)
	at com.intellij.core.CoreApplicationEnvironment.<init>(CoreApplicationEnvironment.java:97)
	at com.intellij.core.CoreApplicationEnvironment.<init>(CoreApplicationEnvironment.java:74)
	at org.intellij.grammar.LightPsi$MyParsing.<init>(LightPsi.java:166)
	at org.intellij.grammar.LightPsi.<clinit>(LightPsi.java:65)
	at org.intellij.grammar.Main.main(Main.java:49)
Caused by: java.lang.ClassNotFoundException: it.unimi.dsi.fastutil.objects.Object2ObjectOpenHashMap
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 11 more

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':generateLombokConfigParser'.
> Process 'command 'C:\jdk8u252-b09\bin\java.exe'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 15s
9:38:56 PM: Task execution finished 'buildPlugin'.

yes the grammar should update too (like in PR #808)

I guess the indention was to build only for oneversion.
so a new Build with 2020.2 or LAtest-EAP  should be bumped.

@XenoAmess ,
Thanks a lot for #810
I was thinking of reverting to old idea version because of lombok plugin was not available.
But, I just gave this change a go,
I was able to build it with the above mentioned change for the plugin.
However since I was using openjdk11 I had to remove the flag CMSPermGenSweepingEnabled from the gradle.properties to get it create the jar successfully.
and finally I placed the new Jar in my plugin folder /my-path/IDEA-U/ch-0/202.4357.23.plugins/lombok-plugin/lib That seems to be working, I can see the compiler is not showing the errors now.

Just added new Build for IntelliJ 2020.2 EAP: https://plugins.jetbrains.com/plugin/6317-lombok/update/87712

Hi there, I'm still having the incompatibility issue:
19:49	Plugin Error
			Plugin "Lombok" is incompatible (until build 201.SNAPSHOT < IU-202.6397.59).

Using IDEA 2020.2, build:
IntelliJ IDEA 2020.2 (Ultimate Edition) Build #IU-202.6397.59, built on July 22, 2020
With Lombok plugin version 0.30: 0.30-2020.1

@gerardbosch Please use https://plugins.jetbrains.com/plugin/6317-lombok/versions/stable/87712
0.30-EAP

Thanks, it seems after, uninstalling Lombok plugin, restarting IDEA, and installing Lombok back, it started to work again :)


Have to avoid certain field name when using pc.Model

Describe the bug
A clear and concise description of what the bug is.
When converting [Model].name to BaseVar for rendering, it will show the random variable name (e.g. ykiyfsfo) instead of the name field define in the model. Can this be fixed? If not, what reserved keywords should I be careful of when defining the model?
To Reproduce
Steps to reproduce the behavior:

Code/Link to Repo:

class Environment(pc.Model, table=True):
    id: Optional[int] = Field(default=None, primary_key=True)
    name: str

class State(BaseState):
    environments: List[Environment] = []

pc.foreach(State.environments, lambda env: pc.option(env.name,value=env.id)),

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
** Specifics (please complete the following information):**

Python Version: 3.11.2
Pynecone Version:  0.1.20
OS: Linux
Browser (Optional):

Additional context
Add any other context about the problem here.

REF-167

Fixed in 0.3.0


Add configure_opts

Closes #50
This PR adds a configure_opts arg that gets passed through to the ./configure call unmodified.
This uses a TrailingVarArg to collect options in a multiple arg. It supports the following two forms:
frum install 3.0.1 --with-jemalloc # only when no flags are specified
frum install 3.0.1 -- --with-jemalloc # always

I'm opening this as a draft PR for now to get feedback on two questions:

Are you ok with this approach @TaKO8Ki? I originally just added a with-jemalloc arg but this has the potential for becoming unwieldy so It may make more sense to allow for arbitrary arguments to be passed through.
Should we remove the dedicated with-openssl-dir option and handle it through the same mechanism?

Once we decided on this I'll finish off this PR and fix the tests (they will fail for the completions alone).
changelog: Add configure_opts to pass through to ./configure.

Thank you for your contribution! I'm going to review this PR within today or tomorrow.

No rush @TaKO8Ki! It's not done yet anyway, as soon as you have time to answer the two questions I posted above I will fix the specs and do any other outstanding work.

@citizen428
Ok. I'm going to answer the two questions within today or tomorrow!

@citizen428
Sorry for the late reply.

Are you ok with this approach @TaKO8Ki? I originally just added a with-jemalloc arg but this has the potential for becoming unwieldy so It may make more sense to allow for arbitrary arguments to be passed through.

Yes. It would be ideal to implement all configure options and completions for them, but it seems difficult because configure has many options and different versions probably have different options. So, at this stage, I think your approach is the best!

Should we remove the dedicated with-openssl-dir option and handle it through the same mechanism?
Yes. Please do so.

@TaKO8Ki This should be ready for review now.

@TaKO8Ki I added the default value for the OpenSSL directory back in.

@citizen428
Thank you for working on this. I left some suggestions. Then could you add e2e tests for configure_opts to https://github.com/TaKO8Ki/frum/blob/main/tests/e2e/mod.rs? I'm sure you can guess how to add e2e tests from other tests, but if you have any questions, please ask me.

Right, I seem to have some issues with the e2e test, if you could take a look that would be helpful 😒

@citizen428
I'm sorry for the merge conflict.

@TaKO8Ki No problem at all, it's sort of my own fault. I wanted to finish this up yesterday, than had a busy day and forgot.

@TaKO8Ki Finally ready, sorry this took so long.

@citizen428

Finally ready, sorry this took so long.

Don't worry about it :).

Given this branch was quite messy (sorry for that, I had resolved the merge conflicts in the browser when I wasn't on my main computer, clearly that went wrong) it's probably best to squash this one on merge.


Sync eng/common directory with azure-sdk-tools for PR 881

Sync eng/common directory with azure-sdk-tools for PR 881

Can one of the admins verify this patch?