ERROR: [facebook]<PHONE_NUMBER>336513: Cannot parse data

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

[X] I'm asking a question and not reporting a bug or requesting a feature
[X] I've looked through the README
[X] I've verified that I'm running yt-dlp version 2023.07.06 (update instructions) or later (specify commit)
[X] I've searched known issues and the bugtracker for similar questions including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue

Please make sure the question is worded well enough to be understood
yt-dlp -Fv --ignore-config https://www.facebook.com/khimsokheng.ksh/videos/1963379427336513
[debug] Command-line config: ['-Fv', '--ignore-config', 'https://www.facebook.com/khimsokheng.ksh/videos/1963379427336513']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version<EMAIL_ADDRESS>[7cccab79e] (win_exe)
[debug] Python 3.8.10 (CPython AMD64 64bit) - Windows-10-10.0.22621-SP0 (OpenSSL 1.1.1k  25 Mar 2021)
[debug] exe versions: ffmpeg N-111199-g2cac3fd744-20230620 (setts), ffprobe N-111199-g2cac3fd744-20230620
[debug] Optional libraries: Cryptodome-3.18.0, brotli-1.0.9, certifi-2023.07.22, mutagen-1.46.0, sqlite3-2.6.0, websockets-11.0.3
[debug] Proxy map: {}
[debug] Loaded 1864 extractors
[facebook] Extracting URL: https://www.facebook.com/khimsokheng.ksh/videos/1963379427336513
[facebook]<PHONE_NUMBER>336513: Downloading webpage
[facebook]<PHONE_NUMBER>336513: Downloading webpage
ERROR: [facebook]<PHONE_NUMBER>336513: Cannot parse data; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
File "yt_dlp\extractor\common.py", line 715, in extract
File "yt_dlp\extractor\facebook.py", line 733, in _real_extract
File "yt_dlp\extractor\facebook.py", line 680, in _extract_from_url
How to fix it?. I updated to<EMAIL_ADDRESS>Provide verbose output that clearly demonstrates the problem

[X] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[ ] If using API, add 'verbose': True to YoutubeDL params instead
[ ] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output
No response

Duplicate of #7901


Added defaultText property for text type validations

Description
Added defaultText property for text type validations
Fixes #3384
Type of change

Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?
Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes

You should remove text and defaultValue from the validation map


[IOTDB-5176] Change ratis snapshot dir with datadir

Currently, when user assigns a customized data_dir in iotdb-datanode.properties, the ratis snapshot dir is not updated with this customized value, so that there will be inconsistent snapshot dirs.
In this pr I propose to update ratis snapshot dir togather with data_dir.

Why not cherry-pick to rel/1.0?


Dependency trees get merged when they should not

What happened:
When collecting multiple BOMs with the sbom-cataloger, the dependency trees of the BOMs might get mixed up.
Example:
My Rust app App1 has a direct dependency proc-macro2 1.0.89. This dependency has a transitive dependency unicode-ident 1.0.13.
App2 depends on proc-macro2 1.0.89 as well, plus also directly on unicode-ident 1.0.9 (a lower version than the transitive dependency in App1).
I generate SBOMs for both apps and package the apps alongside their SBOMs in a container. When I scan the container with the latest version of Syft and enable the sbom-cataloger, the dependency tree in the resulting SBOM looks like this:

This is because the same package ID is generated for proc-macro2 1.0.89 and both packages are merged into one, which is understandable. But I think since the transitive dependencies can differ, even if the package itself is the same, the package ID should depend on some identifier of the cataloged SBOM as well. This effectively means the dependency trees for all cataloged SBOMs will be treated separately instead of being merged into one tree.
I think that this problem is not specific to Rust and affects other languages as well, at least I can think of similar examples in Java.
The change in this PR is just to demonstrate a very hacky solution specific to CycloneDX SBOMs with a serialnumber, it's definitely not a proper solution, hence I marked this PR as draft: I just pass the bom object into collectPackages to be able to access the BOM's serialnumber. I then temporarily prepend that serialnumber to the package name to make the package ID dependant on the BOM it was found in (two equal packages in different BOMs will now have different IDs). If you think the change in general makes sense, we could discuss a proper solution.
I generated the SBOM for my example app with the modified version of Syft and it looks as expected, the dependency trees are separate (proc-macro2 1.0.89 is in there twice, with two different package IDs).
What you expected to happen:
An SBOM with a dependecy tree like this:

Steps to reproduce the issue:
See https://github.com/dervoeti/syft-reproduce-dependency-overlap
Anything else we need to know?:
SPDX would need this as well, maybe all catalogers that obtain information about dependency relationships would need adjustments.
Environment:

Output of syft version: 1.14
OS (e.g: cat /etc/os-release or similar): NixOS 24.05 (Uakari)

Thanks for the quick response!
Yeah, makes sense that only sbom-cataloger is affected if that's the only one merging packages.
I tried to reproduce the behavior with SPDX, I added two Java apps to my repo and generated SBOMs for them with mvn spdx:createSPDX.
They both depend on the same version of Gson and a different version of error_prone_annotations (transitive dependency through Gson). The resulting SPDX SBOM produced by Syft contains only one Gson component. However it's not really problematic in that case, since the sbom-cataloger does not pick up all the dependency information from the SPDX SBOMs generated by Maven anyway. The resulting SBOM just contains a flat list of components for each cataloged SBOM.
Maybe I did something wrong, I don't know too much about SPDX, but I see the dependency relation between Gson and error_prone_annotations in the Maven generated SBOMs: https://github.com/dervoeti/syft-reproduce-dependency-overlap/blob/05ce5a52c4621d116a3a48b1cf1828644f3a200f/java/app1/target/site/com.example_java-app-1.0.0.spdx.json#L95-L99
They are missing in the final SBOM for the container.
But that's probably out of scope for this issue. So yes, it looks like this is currently only a problem for CycloneDX SBOMs.


fix(server): fix comment and post likes

Description
Fixed the issue of increasing the number of likes and comments
Checklist

[ ] A description of the changes in this PR is mentioned above.
[ ] All the new and existing tests pass.
[ ] The code follows the code style and conventions of the project.
[ ] No plagiarized, duplicated, or repetitive code that has been directly copied from another source.
[ ] I have read the whole Contributing Guidelines of the project and its resources/related pages.

Screenshots (if any)

Note to reviewers

Thanks for your contribution.


21350: Ghost Admin Profile Panel requesting 'null' URL for cover image

Got some code for us? Awesome 🎊!
When we do not have a cover image, cover_image is being null.
Added a validation to show image if it null then show none.
Please include a description of your change & check your PR against this list, thanks!

[x] There's a clear use-case for this code change, explained below
[x] Commit message has a short title & references relevant issues
[x] The build will pass (run yarn test:all and yarn lint)

We appreciate your contribution!

@mithelan Can you please fix the linting errors and then we can merge this
  /home/runner/work/Ghost/Ghost/apps/admin-x-settings/src/components/settings/general/UserDetailModal.tsx
    365:1  error  Expected indentation of 20 spaces but found 16  indent
    366:1  error  Expected indentation of 24 spaces but found 20  indent
    367:1  error  Expected indentation of 28 spaces but found 24  indent
    368:1  error  Expected indentation of 24 spaces but found 20  indent

@allouis sure let me fix it

@allouis it has been fixed , please check


Current active tab

I need to programatically close all the tabs depending on X
Is there a way I can get the current tab without diving down the DOM and finding the current tab?
Could we possibly make _getCurrentTabRef a public method?

I'm not planning on adding this, sorry!


Fix Vulnerability in System.Drawing.Common

Describe the bug
A vulnerability has been found in System.Drawing.Common.  This library is used in Microsoft.Data.SqlClient.
Considered High.  See link below.
Additional context
Although not directly imported from NuGet, it is referenced from another library.
https://app.snyk.io/org/daeer73/project/c94eb35e-9d5a-4a38-8225-aa77119aef68#issue-SNYK-DOTNET-SYSTEMDRAWINGCOMMON-3063427

The simplest thing to do is directly import the fixed (updated version) library into the project affected.


Remove all usages of the deprecated usage package

Fixes https://github.com/flutter/flutter/issues/150575.
Also fixes https://github.com/flutter/flutter/issues/150665.
Also fixes https://github.com/flutter/flutter/issues/150663.
Pre-launch Checklist

[x] I read the Contributor Guide and followed the process outlined there for submitting PRs.
[x] I read the Tree Hygiene wiki page, which explains my responsibilities.
[x] I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
[x] I signed the CLA.
[x] I listed at least one issue that this PR fixes in the description above.
[x] I updated/added relevant documentation (doc comments with ///).
[x] I added new tests to check the change I am making, or this PR is test-exempt.
[x] I followed the breaking change policy and added Data Driven Fixes where supported.
[x] All existing and new tests are passing.

This will be succeeded by smaller PRs to make this change more reviewable.


WOIN character sheet improvements

The W.O.I.N. character sheet could be improved in a number of ways. There are some semi-obsolescent sections of the sheet (as the 1.2 version of N.E.W. changed how status tracks work, and the other games have been given errata to match). There are also some sections of the sheet that are very confusing, as the terms used do not match the terms used in the game rules.
I have a separate issue where I've reported a formatting problem that should not require any game knowledge, so I won't include that here.
I'll try to break down the issues as I see them here:

On the "Basic" tab, the "movement" section at the bottom of the page is mostly useless. You have to compute everything by hand, and after you do, there's no easy way share use the information while playing. I think it's fine for the sheet to just collect values that must be calculated manually, but if that's the case it should offer buttons to broadcast them.
Some of the items in the second should be removed or relocated.

Actions is always 2 except for characters with a few rare exploits, so it can probably just be removed. For the characters some other number applies, the exploit granting the extra actions is a better place to document the situation.
Natural damage can just be entered as the damage of an unarmed attack on the Combat/Equipment tab, so it should be removed here.
Initiative and perception can be removed, as they're both just skill tests (though initiative can use either tactics or reactions). Maybe the intention is to precalculate results for these tests to speed up the game, but that's not something the rules actually describe (only defense scores are precalculated rolls).

If you want to automatically calculate the remaining movement related values, here's how:

Base movement is the size of the STR die pool plus the size of the AGI die pool, plus the size of the running skill pool if you have that skill. That's not a roll of those pools, just the number of dice. The unit is (5-foot) squares.
Secondary movement types are mostly the same as running movement, but use a skill other than running (swimming, climbing, zero-g, low-g, high-g), and the result is halved at the end. Some exploits make other movement types "primary" (meaning the speed is not halved), so a checkbox for each one might be need to get the computation right.
Jumping has separate calculations for vertical and horizontal jump distances. Horizontal is 2 * AGI while vertical is min(2 * AGI, STR) (the attribute ratings, not pool sizes). The units for the jumping distances are feet.
Carry amount is (STR + END + carry skill) * 10. This might be better located on the Combat/Equipment tab, since that's where weights of gear is listed. The unit is pounds.

If you do automatically calculate the values, a modifier field for the base move speed is probably necessary, as there are a lot of exploits that adjust it. Some other exploits increase jumping distances as well, but those are proportional increases, not flat ones (e.g. adding 50 or 100% rather than 2 squares). Maybe you could have a row with a base SPEED modifier, checkboxes to make any of the secondary movement types primary, and a multiplier for the jump distances. Then the computed values could be on a second row.
If calculating things automatically is not considered a reasonable option, skip the modifier fields and just give fields and broadcast buttons for "Move", "Climb", "Swim", "Jump (vertical)", and "Jump (horizontal)". Optional extras would be "Zero-G", "Low-G", and "High-G".

The boxes for Attacks on the Combat/Equipment tab are very confusing.

There doesn't seem to be any difference between Damage and Damage Mod, so it's not clear what they mean.
On the other hand, it would be nice if the skill being used for the attack could be selected, rather than needing to add its dice to the Attack Mod box (combined with other modifiers, like for weapon quality).
My preferred layout would be Weapon Name, a Melee/Ranged checkbox, Skill (a dropdown of skills entered on the "Basic" tab, adding their dice automatically to the attack roll), Attack Modifier (d6), Damage (d6), Flat Damage, Range increment. If the checkbox indicated a melee attack, the better of the STR and AGI die pools would be used. If it was ranged, the better of AGI and INT. For melee attacks, the size of the STR pool would be added as flat damage.
If a skill dropdown is not practical, I'm not really sure what the best solution is. I think either the Attack Mod box should be split in two (one for permanent modifiers, like skill dice and weapon quality), or the Damage Mod box should be removed (combining it with the existing Damage box). The former would be ugly as there are already too many boxes and too little to explain the differences. But the latter is awkward to use, as you have to tweak the values to take account of temporary modifiers.
There are a lot of other things that might be nice to attach to attack listings, such as damage types (e.g. heat, ballistic, piercing). Weapon quality is another thing that would be nice to have. The question is how much you can fit in each row.

The condition tracks at the bottom of the Combat/Equipment page are out of date, and may have been confused even with the older editions of the rules (they seem to be mixing another table in with the statuses). The newer editions define this list of conditions:

Afraid
Angry
Bleeding
Blind
Burning
Charmed
Confused
Dazed
Deaf
Disarmed
Disarmored
Downed
Drunk
Fatigued
Forgetful
Manic, Pain
Poisoned
Restrained
Sick
Sleeping
Slowed
Each status track should probably have three levels. I'd label them Temporary, Severe and Persistent. Technically Severe is a kind of temporary condition, but there's no name for the basic level of the conditions, so I think this is best.

The Advancement tab is mostly good. But it would be really nice if it let you record of what XP was earned for, not only what it was spent on. Maybe add a second list (like the one for the XP expenditures) where you add each XP award, and it totals them up for you in the Total Earned box.

Greeting @BlckKnght
You may want to also make a post over on Character Sheet forum. The WOIN sheet does seem to get a lot of updates on GitHub. I pinged the last person to contribute to it to see if they may be interested in fixing the other issue you posted. Posting these suggestions on Character Sheet forum as well may be a chance to find additional sheet authors interested in making updates.
I can also point you to some resources if you'd like to try your hand at editing the code. :)

Thanks, I've posted a thread for discussion here.
I'd love some pointers to references. I looked at the code for the sheet in Github, but I really don't know what it's doing. I know a fair amount of programming but I don't have much experience with web stuff, nor any experience at all with Roll20 macros.

@BlckKnght
I am happy to see your thread is gaining traction. Looking at another developer's code, or sometimes even your own, and not knowing what its doing is pretty standard for Web Development. ;) There are a lot of references out there which I'm working to gather into one place. I'll list some below:

Web Developemnt Code Academy has a Web Development course which can be really helpful if you need a quick how to for HTML, CSS, and Javascript. You mentioned having some programming knowledge so you may not need this. The three languages mentioned here are primarily what sheet authors use.
Building Character Sheets this wiki entry has the basics on common elements found in Character Sheets.
Roll20 Macros & Dice Reference this is the general wiki article for Macros and the Dice roller. What I will do is make a game for development with the character sheet I am looking to improve. I will fill out a character. Then I will use the Abilities & Attributes tab on the character handout. The Abilities column can be used to make Macros just for that character which can reference the character's attributes for example /r @{strength}d6 will roll the character's strength attributed in six sided dice. I play around with macros here until they roll the way I want them to. Then its easy enough to copy the macro into my HTML button in my text editor.

So I've been discussing the WOIN sheet with other players in games with me, and a few other issues have been pointed out:
When you update your attribute scores or max die pool, the buttons to roll skills and attacks do not get updated with the new values immediately. I'm not exactly sure what triggers an update, but you need to change around some things on each skill or attack row for them to sync up.
The Luck pool, which is represented by radio buttons near the top of the sheet, always resets to full when you reopen the sheet. (Or something like that, I'm not exactly sure how it works.) Refreshing your luck pool is something that happens only a limited number of times in play (once a day usually), so having it reset between sessions that don't correspond to an in-game rest makes the value wrong.
The weight fields in the equipment section expect all items to have integer weights, but some common items like communicators are half a pound in the rules. This is not a big deal, but it's annoying to need to round the weight up or have an angry looking red outline on the field when you put the correct value in.
A nice-to-have improvement: The grades on the life path tab could be added up, and the max die pool (currently shown at the top of the page in big text) could be automatically calculated from that total. I'm not sure there's a nice formula for all grades. The grades used by most player characters (starting at 5) are pretty straight forward, and similar to die pool sizes from skills and attributes (based on triangular numbers, just with an offset this time). A table is at the top of this page in the rules reference that covers grades 5+: http://www.woinrpg.com/difficulty-benchmarks/ For lower grades which are usually not played, 1-3=3d6, 4=4d6.
I've also had an alternative WOIN character sheet pointed out to me: https://github.com/EtoileLion/Roll20-WOIN My understanding is that it is not possible for that sheet to become an official one because it uses API scripting to make its die rolls work. I haven't yet looked at its license too closely, but maybe some of its code can either be used, or can inspire improvements on the current sheet.

Is this fixed or is it still a problem?

Most of the issues I mentioned above are still problematic. After having many more months of play experience (including introducing a variety of new players to the sheets), I think these are the most important issues:

There's nowhere to enter die pool modifiers for skill tests. There should be two kinds, equipment quality modifiers (which are subject to the maximum die pool cap after combining with the attribute and skill pool) and situational/exploit modifiers which get applied after the cap. It might be nice to prompt for additional situational/exploit pool modifiers, since there may be some that only apply rarely, not all the time.
The same kinds of modifiers (quality+situational/exploit) should be used for attacks. The current set of fields for attacks and their damage are only minimally functional, they don't make a whole lot of sense. A more logical split would have two fields for attack pool modifiers: Skill+Quality (capped after adding to the attribute pool) and Situational/Exploit (not capped). The current damage fields are also confusing. It might be nicer to write it as "___ d6 + " (and "+ d6" for situational bonus damage dice, though we might leave this prompt only). Prompting for situational attack and damage modifiers would be nice, and is even more important than prompting for skill test modifiers.
The state of the luck pool is not tracked effectively. There are radio buttons that work OK to count it off during a session, but the state is forgotten if you exit the game and return.
A less important, but hopefully easy issue: The backgrounds of the character sheets disappeared some short time after the last update.

This issue is now 3 years old and it appears some of these issues have been resolved. If there are outstanding features or bugs, please feel free to create fresh issues for them.


[Task]: Implement clear button for filters dirty state

Is there an existing issue for this?

[X] I have searched the existing issues

SubTasks
When the state of filters is dirty, we should show a clear button.
Refer: https://github.com/appsmithorg/appsmith/issues/15373

This is done by commit 32.


Support using the version id for secrets manager secrets

Describe the solution you'd like
Setting the data.remoteRef.version key in an ExternalSecret resource, which refers to a Secrets Manager secrets store, will only use the version as the VersionStage in the GetSecretValue api call. If you needed to get a version of the secret that is not AWSPREVIOUS or AWSCURRENT, there is no way to retrieve the secret through the ESO.
What is the added value?
Secrets that are more than one version old will be retrievable from the external secrets operator. Also, users will be able to specify an explicit secret version id, and that secret version will be immutable.
Give us examples of the outcome
On the ExternalSecret resource, I should be able to specify the secret version as either the version stage, or the version id. For instance, I could have the secret data spec look like both
data:
  - secretKey: secret-key-to-be-managed
    remoteRef:
      key: my-aws-secret
      version: "123456-abcd-efgh-abcd-654321"

or
data:
  - secretKey: secret-key-to-be-managed
    remoteRef:
      key: my-aws-secret
      version: "AWSPREVIOUS"

hi @matt-demers ! It seems to me that it is possible to create some custom "VersionStages" to your secret during update and/or create as per here This would allow the user to have more control about the secret Version itself, and possibly be a little bit more versatile than just the version of the secret.
For your use case, do you believe customizing VersionStages could be a valid approach?

In any case I think we can do some logic to get something like this as a way to be compatible with possible edge cases (like if someone decides to create a VersionStage called 123456-abcd-efgh-abcd-654321 :smile: ):
data:
  - secretKey: secret-key-to-be-managed
    remoteRef:
      key: my-aws-secret
      version: "AWSPREVIOUS"
  - secretKey: secret-key-to-be-managed
    remoteRef:
      key: my-aws-secret
      version: "uuid/123456-abcd-efgh-abcd-654321"

Interesting I did not see that documentation around custom version stages. I still think we would need to be able to specify the actual version id because the way we modify secrets by using the aws console, which does not give that option to set the version stage.


pressure test. after 10000 times request, out of memory.

Riot version (or commit ref):
Go version:  v.0.10.0

Operating system and bit:
Can you reproduce the bug at Examples:

[ ] Yes (provide example code)
[ ] No
[ ] Not relevant

Provide example code:
this is the code fo search:
output := searcher.SearchDoc(types.SearchReq{
Text: query,
//Tokens: text1,
RankOpts: &types.RankOpts{
ScoringCriteria: &ArticleScoringCriteria{},
OutputOffset:    (page-1)*count,
MaxOutputs:      count,
},
})

Log gist:

Description
...

I used engine.UsedMem() to get memoery used data. It is increasing continuously. I did presss test to request same url.
It would be a problem.
Thank you very much.


Feature request : Tag filtering field

Hi ttscoff,
First of all I like to thanks you for NV which is a great software that I now use every day (with the simple note sync).
I have a sugestion related to the use of tags. As far as I understand, Tags are only used for spotlight search and are searched as all the note contents in NV.
Sugestion : Why add a "tag filtering" filed in NV ?
I could be below or on the right of the main search field.
The interest would be to restrict the search to only the notes that include the tags in the "filter" field.
Personnally it would be of great interest. Let me explain why. All my life (personnal and work) is linked to "music". I'm a researcher on brain & music, I'm teaching in this field and in my personnal life I play music, I listen to music etc..
Thus I have hundreds of notes where the word music appears in. When tagging my note I often have the "Music" tag but it is completed with other tags such as "Course", "Research", "Guitar", "score"...
If I want to see list of all my notes that I have used to prepare my "courses", I can search with Music and Course but if these words also appears in other notes I will have to much results.
Thus I propose to introduce this "tag filtering" possibility. Tags would be considered as means of classifying information. Of course, the normal functionning of NV is preserved if nothing is introduced in the filtering field.
What do you think of that proposition ? Is there a place where I can submit it ?
Sylvain

Really looking forward to a feature like this. For those of us who don't use spotlight it will finally make tags worth using. I'd like to suggest using a hash (or other single character) as the tag specifier in the search bar to maximize search brevity.


migrate to pytorch or keras

caffe is outdated dinosaur, nobody use it.
Also your windows build does not work.
I suggest you to migrate this and future projects to pytorch or keras , which are user friendly.

@iperov Good suggestion! We should do that.


Update release notes for 0.1

Here:  https://fifewiki.fnal.gov/wiki/Release_notes

First stab at that completed.  Will review/clean up tomorrow and close this out.


where I can delete other peds

where I can delete other peds ,only choose   "mp_m_freemode_01",  "mp_f_freemode_01",

ok  I got this ,sry


Add tx fee activation account modal

fill in account activation cost in onboarding modal flow.

@Mergifyio backport dev


TypeError: Promise.all is not a function

I am getting error while minifying image.
TypeError: Promise.all is not a function
    at module.exports (/home/ajit/Desktop/ubuntu backup/demo-project/node_modules/globby/index.js:50:17)
    at module.exports (/home/ajit/Desktop/ubuntu backup/demo-project/node_modules/imagemin/index.js:53:9)
    at EventEmitter.minify (/home/ajit/Desktop/ubuntu backup/demo-project/application/controller-service-layer/controllers/ImgMinController.js:112:7)

code in my controller:

const imagemin = require("imagemin");

const imageminMozjpeg = require('imagemin-mozjpeg');
const imageminPngquant = require('imagemin-pngquant');

imagemin(['/home/ajit/Pictures/Screenshot from 2016-11-22 11:22:58.png'], '/home/ajit/Desktop/min-img', {
    plugins: [
        imageminMozjpeg(),
        imageminPngquant({
            quality: '65-80'
        })
    ]
}).then(files => {
    console.log(files);
});

@ajit-soman

Show me the output of node -v && npm -v.
Are you using some JavaScript compilers (e.g. Babel) to run your code?

ajit@ajit-Inspiron-N5050:~/git$ node -v
v6.9.1
ajit@ajit-Inspiron-N5050:~/git$ npm -v
3.10.8

I just use NODE_ENV=development node app.js or some time NODE_ENV=development nodemon app.js to run code

@ajit-soman Can you add console.log(global.Promise) just before const imagemin = require("imagemin"); and show me the output?

I got [Function: defer] as output

@ajit-soman OK, That is the cause. The native Promise is overridden by another broken Promise implementation somewhere in your application or in one of the libraries you use. Usually console.log(global.Promise) outputs [Function: Promise].
imagemin has nothing to do with this issue.


[firebase_crashlytics] Expected encodable string, but found NSNull instead. crashes not showing in iOS

errorMessage
Expected encodable string, but found NSNull instead.
Android is good, but iOS Crashlytics not showing in 2.9.0
i added "${PODS_ROOT}/FirebaseCrashlytics/run" this script is in the end of Build Phases.
and run Release mode
How to fixed this issue?
testCode
`   WidgetsFlutterBinding.ensureInitialized();
await Firebase.initializeApp();
await FirebaseCrashlytics.instance.setCrashlyticsCollectionEnabled(true);
FlutterError.onError = _onError;

try {
  throw Exception("Crash test");
} catch (error, stack) {
  FirebaseCrashlytics.instance.recordError(
    error,
    stack,
    reason: "Test",
    fatal: true
  );
}`

result
flutter: ----------------FIREBASE CRASHLYTICS----------------
flutter: The following exception was thrown Test:
flutter: error

10.7.0 - [FirebaseCrashlytics][I-CLS000000] Expected encodable string, but found NSNull instead. Set a symbolic breakpoint at FIRCLSEncodeString to debug.
[✓] Flutter (Channel stable, 3.3.10, on macOS 13.3.1 22E772610a darwin-x64, locale ko-KR)
• Flutter version 3.3.10 on channel stable at /Users/duke/development/flutter
• Upstream repository https://github.com/flutter/flutter.git
• Framework revision 135454af32 (5 months ago), 2022-12-15 07:36:55 -0800
• Engine revision 3316dd8728
• Dart version 2.18.6
• DevTools version 2.15.0
[✓] Android toolchain - develop for Android devices (Android SDK version 33.0.0)
• Android SDK at /Users/duke/Library/Android/sdk
• Platform android-33, build-tools 33.0.0
• Java binary at: /Applications/Android Studio.app/Contents/jre/Contents/Home/bin/java
• Java version OpenJDK Runtime Environment (build 11.0.13+0-b1751.21-8125866)
• All Android licenses accepted.
[✓] Xcode - develop for iOS and macOS (Xcode 14.3)
• Xcode at /Applications/Xcode.app/Contents/Developer
• Build 14E222b
• CocoaPods version 1.12.1
[✓] Chrome - develop for the web
• Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
[✓] Android Studio (version 2021.3)
• Android Studio at /Applications/Android Studio.app/Contents
• Flutter plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/9212-flutter
• Dart plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/6351-dart
• Java version OpenJDK Runtime Environment (build 11.0.13+0-b1751.21-8125866)
[✓] IntelliJ IDEA Community Edition (version 2022.2.2)
• IntelliJ at /Applications/IntelliJ IDEA CE.app
• Flutter plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/9212-flutter
• Dart plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/6351-dart
[✓] VS Code (version 1.78.0)
• VS Code at /Applications/Visual Studio Code.app/Contents
• Flutter extension version 3.62.0
[✓] Connected device (3 available)
• iPhone (mobile) • 00008030-000964380A92802E • ios            • iOS 16.3.1 20D67
• macOS (desktop) • macos                     • darwin-x64     • macOS 13.3.1 22E772610a darwin-x64
• Chrome (web)    • chrome                    • web-javascript • Google Chrome 112.0.5615.137
[✓] HTTP Host Availability
• All required HTTP hosts are available

I'm still got this Error when app starts in iOS Real Device (IPhone 11 pro)
 10.7.0 - [FirebaseCrashlytics][I-CLS000000] Expected encodable string, but found NSNull instead. Set a symbolic breakpoint at FIRCLSEncodeString to debug.
TestCode
    await FirebaseCrashlytics.instance.recordError(       "Record error to Crashlytics",       StackTrace.fromString("No StackTrace."),       reason: "Init Crashlytics !",     );

@darshankawar
I'm still got this Error when app starts in iOS Real Device (IPhone 11 pro)
 10.7.0 - [FirebaseCrashlytics][I-CLS000000] Expected encodable string, but found NSNull instead. Set a symbolic breakpoint at FIRCLSEncodeString to debug.
TestCode
    await FirebaseCrashlytics.instance.recordError(       "Record error to Crashlytics",       StackTrace.fromString("No StackTrace."),       reason: "Init Crashlytics !",     );

okay
I solved this problem.


[DEV] - Console Warning: Function components cannot be given refs

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: Dev
Reproducible in staging?: dev
Reproducible in production?: dev
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Expensify/Expensify Issue URL:
Issue reported by: @rojiphil
Slack conversation: https://expensify.slack.com/archives/C049HHMV9SM/p1728537899975699
Action Performed:

Load/Reload the iOS App
Open a chat report with messages
Long press on a message to display the popover menu.

Expected Result:
The warning Function components cannot be given refs. should not come-up.
Actual Result:
The following warning is displayed:
Warning: Function components cannot be given refs. Attempts to access this ref will fail. Did you mean to use React.forwardRef()?
Workaround:
unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android: Native
[ ] Android: mWeb Chrome
[x] iOS: Native
[ ] iOS: mWeb Safari
[ ] MacOS: Chrome / Safari
[ ] MacOS: Desktop

Screenshots/Videos
https://github.com/user-attachments/assets/8965b291-e0c1-4a22-b117-ae1188f3edc5

Add any screenshot/video evidence

View all open jobs on GitHub

This has been labelled "Needs Reproduction". Follow the steps here: https://stackoverflowteams.com/c/expensify/questions/16989

Proposal
Please re-state the problem that we are trying to solve in this issue.
We are facing a warning message in our iOS app that says, "Function components cannot be given refs." This warning appears when users try to interact with chat messages
What is the root cause of that problem?
The root cause of this problem is that some functional components in our app are being assigned refs, but they are not set up to handle them. In React, functional components do not accept refs by default, which leads to this warning.
What changes do you think we should make in order to solve the problem?

To solve this problem, we should update the functional component by wrapping it with React.forwardRef(). This will allow these components to accept and properly use refs without generating warnings.
What alternative solutions did you explore? (Optional)

Proposal
Please re-state the problem that we are trying to solve in this issue.
Encountering a console error: "Function components cannot be given refs" when long-pressing the message.
What is the root cause of that problem?
This specific issue is originating from the usage of withOnyx and wrapping it in memo on the BaseReportActionContextMenu file. We're also seeing the same error in the ReportScreen and Sidebar screens too, though those errors are coming from different files.
What changes do you think we should make in order to solve the problem?
I suggest a similar approach as with this issue: we should replace withOnyx with useOnyx.
For this issue, we can focus on the BaseReportActionContextMenu and BaseKycWall files.
What alternative solutions did you explore? (Optional)
If right now we dont want to replace the usage of useOnyx instead of withOnyx then we can do use the forwardRef for BaseReportActionContextMenu like this:
https://github.com/Expensify/App/blob/0510a66ecf8d2003db3c9b295d90c7b10610d2c4/src/pages/home/report/ContextMenu/BaseReportActionContextMenu.tsx#L93

const BaseReportActionContextMenu = React.forwardRef(function BaseReportActionContextMenu({

I think GH Actions will fail if we touch those files and don't replace the withOnyx call, so go with that in both places you mentioned

@rojiphil offer out to you

@rojiphil did you work on this one at all or just report it?

Payment Summary:

Contributor: @jayeshmangwani $250 - submit once you've completed the checklist
Contributor+: @hoangzinh $250 paid via upwork

I don't think a checklist is needed here since these are console warning and should have been caught when we initially passed the ref to the functional component during creation or migration of the page.
Let me know if you think otherwise, @garrettmknight @hoangzinh

did you work on this one at all or just report it?

@garrettmknight I just reported it. No payment is due for me here. Thanks.

Thanks, I'll undo the payment and we can close.

@hoangzinh I accidentally requested a refund on this one in Upwork. Please decline.

I confirm that this payment summary is accurate.

$250 approved for @jayeshmangwani


Improve manage.html page

Is there an existing issue for this?

[X] I have searched the existing issues

Feature Description
I had designed manage.html page of this website few days back but the changes were not reflected in the website. I want to fix this again
Current UI:

Final output:

Use Case
This will enhance the look of the website. I have used colors according to the logo of ambuflow
Benefits
No response
Priority
Medium
Record

[X] I agree to follow this project's Code of Conduct
[X] I'm a GSSOC contributor
[x] I want to work on this issue
[X] I'm willing to provide further clarification or assistance if needed.

@mansiruhil13 plz assign this issue to me


Disable ways to switch into Wallet Only mode

No issue.
It's a temporary solution to turn off this feature since our Infra is not ready yet.
It disabled buttons that switch the wallet into Wallet Only mode in two places:

in Wallet creation/recovering workflow
in the Settings page (button "switch to local node" still works in case the wallet is in a wallet-only mode for some reason)

If I miss something — let me know 🙏

in general ok.
But there's one scenario that gets me stuck: when I "open an existing wallet" from a file, and this wallet was "Wallet only".
But I'm not sure if this will be actually possible in the Mainnet? the stests wallets will not be supported/usable and if there's no way to create a wallet only, then the users should not be able to follow this scenario of opening such wallet only file. (unless there’s something I missed)

opening such wallet only file
It is still doable, but if there will be no public RPC — then it won't connect somewhere.
But it should show "No available public APIs" and show the "Skip" button then.
Afterward, it is possible to switch to local node mode (in Settings, as usual) or just do smth with the wallet "offline" (check addresses, past transactions/rewards stored on fs, etc).

p.s.
I decided not to make "force-switch" for wallet-only wallets, because it should be a rare case for Genesis and User still able to switch to local mode.


feat(Mathlib.Algebra.Polynomial.RingDivision): add Polynomial.nmem_nonZeroDivisors_iff

We add Polynomial.nmem_nonZeroDivisors_iff, known as McCoy's theorem: a polynomial P : R[X] is a zerodivisor if and only if there is a : R such that a ≠ 0 and a • P = 0.

[x] depends on: #12958

Co-authored-by<EMAIL_ADDRESS>

maintainer merge

I've been hacking on this proof a bit more. I suggest first proving
theorem McCoy (f : R[X]) (h : ∀ r : R, r • f = 0 → r = 0) :
    ∀ (g : R[X]), f * g = 0 → g = 0 := by
  intro g hg
  by_contra hg₀
  have key : ∃ i, g.leadingCoeff * f.coeff i ≠ 0 := by
    contrapose! hg₀
    rw [← leadingCoeff_eq_zero]
    apply h
    simp [Polynomial.ext_iff, hg₀]
  have hnonempty : {i | f.coeff i • g ≠ 0}.Nonempty := key.imp fun i hi ↦ by
    rw [Set.mem_setOf_eq, ← leadingCoeff_ne_zero, smul_eq_C_mul, leadingCoeff_mul']
    all_goals rwa [leadingCoeff_C, mul_comm]
  have hbdd : BddAbove {i | f.coeff i • g ≠ 0} := by
    use f.natDegree
    intro i (hi : f.coeff i • g ≠ 0)
    contrapose! hi
    rw [coeff_eq_zero_of_natDegree_lt hi, zero_smul]
  let m := g.natDegree
  let l := sSup {i | f.coeff i • g ≠ 0}
  have hl : f.coeff l • g ≠ 0 := Nat.sSup_mem hnonempty hbdd
  suffices (f.coeff l • g).natDegree < g.natDegree by
    apply hl
    apply McCoy f h (f.coeff l • g)
    rw [smul_eq_C_mul, mul_left_comm, hg, mul_zero]
  suffices f.coeff l * g.leadingCoeff = 0 by
    apply (natDegree_smul_le (f.coeff l) g).lt_of_ne
    contrapose! hl
    rwa [← leadingCoeff_eq_zero, ← coeff_natDegree, coeff_smul, hl, coeff_natDegree, smul_eq_mul]
  suffices (f * g).coeff (l + m) = f.coeff l * g.leadingCoeff by rw [← this, hg, coeff_zero]
  rw [coeff_mul]
  apply Finset.sum_eq_single (l, m) _ (by simp)
  simp only [Finset.mem_antidiagonal, ne_eq, Prod.forall, Prod.mk.injEq, not_and]
  intro i j hij H
  obtain hi|rfl|hi := lt_trichotomy i l
  · have hj : m < j := by omega
    rw [coeff_eq_zero_of_natDegree_lt hj, mul_zero]
  · omega
  · suffices f.coeff i • g = 0 by rw [← coeff_C_mul, ← smul_eq_C_mul, this, coeff_zero]
    contrapose! hi
    refine le_csSup hbdd hi
termination_by g => g.natDegree

using recursion.

@jcommelin I've modified the PR using your proof, thanks for the review!

@Ruben-VandeVelde I've reverted your last commit. termination_by does not like variables before the colon.

It feels like the proof of eq_zero_of_mul_eq_zero_of_smul is slightly too complicated, but I can't figure out a way to simplify it (reordering proofs, trying calc blocks, etc.) So looks good to me, thanks!
bors r+

It feels like the proof of eq_zero_of_mul_eq_zero_of_smul is slightly too complicated, but I can't figure out a way to simplify it (reordering proofs, trying calc blocks, etc.) So looks good to me, thanks!
bors r+

I think the proof is indeed surprisingly tricky! There is a better proof using localization, but it requires a much bigger machinery (and I am not sure it works for semirings...).


Fix glob patterns with mixing relative and absolute paths.

See https://groups.google.com/forum/#!topic/closure-compiler-discuss/vIkk0XImMpg.
Also, skip a file or directory if visiting it failed.

It's kind of hard to write a unit test for this that works internally :(

If you write one that only works externally then we can do some MOE trickery so that it doesn't run internally. Or if it makes more sense, maybe the test can just start with essentially if (!WINDOWS) {return}.

@MatrixFrog It's not Windows specific. I added two tests with // NOTE(moz): This might not work on Google's internal build system..

Okay, well I'll give it a try.

@MatrixFrog IIRC @blickly said creating files in the current directory worked internally, but not creating directories. I modified the tests to not create directories so they might actually work internally.

Oops, I was wrong. testGlobJs7 is failing but I don't see why. Anyway we'll figure it out.

@MatrixFrog Might be a permission issue, but in that case testGlobJs6() should also fail.

It is a permissions issue, and if you wrap all your renameTo calls in an assertTrue then the other test fails too. https://docs.oracle.com/javase/7/docs/api/java/io/File.html#renameTo(java.io.File) recommends this.

@MatrixFrog Alright.. I wrapped all of the renameTo calls with assertTrue. We'd have to do some MOE trickery with testGlobJs6 and testGlobJs7 then :(

Ran into another problem: This breaks the use of whitelist files. The whitelist file will generally use a path that is relative to the root of the project, but the error is reported using the absolute path, so the whitelist fails to suppress the error.

@MatrixFrog OK I changed to check the normalized absolute paths for exclusion, but return the original paths. Hopefully it should work now.


Encoder readings are slightly inaccurate

When moving the robot forwards 100cm, it will register ~75cm.
I think this is an issue in out PPR calculation

Another possible solution is to use the configSelectedSensor method with one of these options on the sensor talon.

Here is the data sheet for the current encoders (again, these will be replaced soon)
Our model number is E4T 1000-250-S-H-D-B

Switching the PPR count to 1024 may fix this (according to ctre)

Solved


Scrobbles are not being sent (only now playing)

debug_info.zip
OS: Windows 10 (latest)
Issue Descriptions: Tracks are not scrobbled after they're finished. Only Now Playing notifications are sent to Last.FM.
Steps to Reproduce: Play any track.

Duplicate. Fixed in master.

Not sure if this is fixed, I'm on the latest build and it's still happening.

It's been fixed but nothing has been released yet.


fix multiple loadproblem! calls

I first tried the finalize_env mechanism used in Gurobi.jl but it gave me segfault because it happened that the model that is thrown away gets gc'd second which cause it to be freed after the environment has already been freed by the new model that was gc'd first.

It seems suspicious that we would need to manually implement a reference counter. Is there a simpler fix?

Since each problem has a reference to the environment we might just trust Julia's GC to detect when to delete Env

@blegat, see discussion at https://github.com/JuliaLang/julia/issues/3067

@mlubin Thanks for the link.
So we know that when the Env finalizer is called, it is no longer referenced any where so the user will not create new models or use any models and the finalizer of all models have either already been called or are going to be called soon. Is see two solutions

Have a counter in Env and a Bool saying whether the GC has already called finalized.

In the Env finalizer, if the counter is 0, free it otherwise, just set the Bool to true.
In the Model finalizer, free the model and ask the env to decrease the counter. If the counter goes to zero and the Bool is true, free the env.

Have a list of models in Env and a Bool in Model to say whether the Model has already been freed

In the Env finalizer, for every problem in the list, free it then free the Env.
In the Model finalizer, free it then remove it from Env's list.

Which one do you prefer ?

I'm a bit confused here. Would the issue be resolved by the approach in Gurobi.jl where you can optionally provide an Env to the Model object, in which case the model's finalizer won't free the Env? We can tell people to do this if they need to call loadproblem! more than once.

Who is going to free the Env then ?

The finalizer for the Env, as it should.

But then what prevents Env to be finalized before the models ?
In gurobi, the Env does not call the finalizer, it is commented
# finalizer(env, free_env)  ## temporary disable: which tends to sometimes caused warnings

Good question, I didn't know that was disabled. Ok, let's go ahead with the manual reference counting. Whichever approach you prefer.

I have implemented the first one :)


Add Quantum Random Access Code(QRAC) feature map

What is the expected enhancement?
Implement n*(3,1,0.79) Quantum Random Access Code(QRAC) introduced by Chuang, as a new kind of feature map, which can encode 3n bit feature string within n qubits, with the possibility 0.79.
Reference: https://arxiv.org/abs/quant-ph/0604061

The FeatureMap class in qiskit.aqua will be deprecated, but there is no FeatureMap Class in qiskit.ml now, should us first implement it?

The feature map circuits that presently exist are here https://github.com/Qiskit/qiskit-terra/tree/main/qiskit/circuit/library/data_preparation

This has been open for sometime and not addressed, as such I am going to close it. I will note that QRA Optimzation PR that is pretty much complete, at the time I am writing this, in Qiskit Optimization see qiskit-community/qiskit-optimization#487


Search & Replace omits first occurence when selected beforehand but the cursor being behind

Godot version:

3.2.3.rc3 (verified also on 3.2.2 stable)
OS/device including version:

Windows 10
Issue description:

When marking a word via double-click to use it for step-wise Search & Replace, the selected word isn't taken into account, but the succeeding occurrence. Apparently it's solely cursor dependant and in most cases the cursor is located after the selected word when double-clicking in the middle area of the text.
Steps to reproduce:
Create new_script.gd and paste a code with several occurences:
extends Node

func _ready() -> void:
	pass
	pass
	pass
	pass
	pass

With the mouse, double-click the first pass.
Press CTRL+ R.
Type "replaced" into the replace field.
Hit Replace.
Watch the second pass be replaced, effectively omitting the first (selected) one.

Current workaround:
Double-click the word carefully at the left edge, so that the cursor remains in front of the word and thus takes it into account for Search & Replace.

This is a duplicate of:
"Replace" doesn't take effect on current text if selected from left to right #33315
That issue was closed as completed by the PR:
Fix some bugs about search in code editor #61776
The PR author explained the fix as:

Well, now when there is a selection, searching and replacing is performed from the beginning of the selection.

The issue is not reproducible in: v4.0.beta5.official [89a33d28f]
But the issue is reproducible in: v3.5.1.stable.official [6fed1ffa3]


Increasing the memory capacity for Qualifying Test

The Why
There was an issues activating test for exercise DDJ 066 which had 2764 candidates for each situational Judgement  and critical analysis test. The platform couldn't activate all the 2764 tests.
After investigation we realised the default setting for processing tests was set at 2000. We would need to allocate more memory for 2000+ tests.
The Action
We need to permanently increase the limit of the default setting to process 5000 tests per user for each test. The platform should be able to successfully activate 5000 candidates for each test.
We need to allocate more memory to the cloud functions so that this action can be process successfully.  Can we test and ensure that the platform can successfully activate 10,000 test

Writing a note on this ticket - we currently have an 8GB limit on function invocation on google cloud which is a hard limit, and unable to be increased further. I'm looking at a way to chunk these to only process 1000 at a time, so that this function should be able to handle it.

As discussed this task is more about generating test data we can use to test QTs (and other functions) so we gain confidence that the platform will handle large amounts of data.
We are looking to ensure the platform handles an exercise with 10,000 candidates, applications and tests.
Our cloud functions have recently been manually allocated more memory than they need (8GB). In reality they need a lot less than this. Originally they were allocated 128MB which was sufficient to handle 2000 candidates but no more. So would expect ~1GB to be more than enough.
An example of allocating memory is here:
https://github.com/jac-uk/digital-platform/blob/develop/functions/callableFunctions/exportExerciseData.js#L9
Functions docs are here:
https://firebase.google.com/docs/functions/manage-functions
Firebase pricing info is here:
https://cloud.google.com/functions/pricing

Re-assigned to @warrensearle (that's me that is!)

Closing as this has been completed on the new QT platform


DMW1000 and NodeMCU Lolin

Does anyone had already tried to use NodeMCU ESP8266 connected to DMW1000?
NodeMCU ESP8266 (http://www.nodemcu.com/index_en.html#fr_54747361d775ef1a3600000f)

I am just buying the goods for the test

I hope to connecting the devices on friday 2017-03-10

Any results from the NodeMCU connection?

have a look...
https://github.com/vortex314/esp_tcp
He was able to communicate with the DWM1000, but not very reliable.

@CornevanDijk Did you got some results with the nodemcu?

Seems working with esp #156 so nodemcu should work too.


Dependency 'Result', '0.6-beta.1' causing newer frameworks to break

Your podspec includes:
  s.dependency 'Result', '0.6-beta.1'

Unfortunately this is causing newer frameworks (such as ReactiveCocoa, Swift 2 branch), Moya, and others to break:
dyld: Library not loaded: @rpath/Result.framework/Result
  Referenced from: /Users/zbeckman/Library/Developer/CoreSimulator/Devices/FCFAAFBF-1AB4-4223-9BCC-CA91CBDE96DC/data/Containers/Bundle/Application/9E7272D7-5C66-48F9-9E37-892E41662655/Glimpulse.app/Frameworks/ReactiveCocoa.framework/ReactiveCocoa
  Reason: Incompatible library version: ReactiveCocoa requires version 1.0.0 or later, but Result provides version 0.6.0
(lldb)

I'm sorry you're experiencing problems!
The error message states that ReactiveCocoa requires Result version 1.0.0, but that version does not exist. 0.6-beta.1 is the newest. Also, I've read the ReactiveCocoa issue, but I'm unfortunately not following why the conclusion there is that BrightFutures is the problem.
Can you explain what you think I should change to make it work? I hope I can help!

You have to manually define what version of Result you use in your Podfile until there's an official tag + release.

Thanks @orta for chiming in. If I understand you correctly, there's not much I can do from the BrightFutures side and @zbeckman should update the Podfile to point to a specific version of Result.
Let me know if that works @zbeckman!

I'm afraid I have no control of ReactiveCocoa...

If you're still having problems, it would help if you'd post your Podfile here.

I've updated/fixed the BrightFutures podspec, which does solve some problems with the Result dependency. I'm going to close the issue for now, but feel free to reply if you have new information.


Fix Oracle tests running locally

Q
A

Is bugfix?
❌

New feature?
❌

Breaks BC?
❌

Fixed issues
-

Why GitHub Actions work without this changes, but don't work locally?

I have locally Oracle Database 21c XE (Express Edition)
This does not work if not specify DB service name XE
'oci:dbname=localhost:1521/XE;charset=AL32UTF8',

In some cases during development there were problems with dependent CONSTRAINTS when dropping tables.
Now it works without changes, but in the future may encounter the same problem.


Warning: Missing atom for atom ID

I am receiving warnings like the following during some of my Alphafold runs on Omegafold output:
3ZW9
Missing atom for atom ID OAP
The offending lines from my cif look like:
HETATM 5006 O  OAP B COA BV 3    . ?   7.041  15.430   9.293 0.30 154.79 ? 1   COA BV OAP 1
Alphafill is completing successfully but some inputs throw warnings like this for multiple atoms and I'm curious if the output will likely be affected.
Pdb-repo downloaded with 'rsync -av rsync://rsync.pdb-redo.eu/db/ pdb-redo/'

Sometimes an atom is missing in the experimental model that can trigger such a warning. If there are more experimental models with the same compound the most complete one is taken. In your case the COA seems to have been transplanted from a different model. The occupancy is 0.30 which none of the parent molecules in 3zw9 have.


correctness of ContentTypeJson value

Looks like the following:
	eventData := esdb.EventData{
		ContentType: esdb.ContentTypeJson,
		EventType:   "TestEvent",
		Data:        data,
	}

in  https://github.com/EventStore/EventStore-Client-Go/blob/c29aa7d3ce9c0aadc7d2bfa3dd060f2865efb227/samples/quickstart.go#L42C21-L42C36 is incorrect ,  does it have to be int as in below?
https://github.com/EventStore/EventStore-Client-Go/blob/c29aa7d3ce9c0aadc7d2bfa3dd060f2865efb227/esdb/event_data.go#L12
the following code works for me:
	eventData := esdb.EventData {
		ContentType: 1,
		EventType: "TestEvent",
		Data: data,
	}

So, the ContentType: esdb.ContentTypeJson need to be changed to ContentType: 1 ?

Hey @karthiksedoc,
I'm a bit confused by your question. See esdb.ContentTypeJson is a regular integer as shown at its declaration site.
https://github.com/EventStore/EventStore-Client-Go/blob/v4.2.0/esdb/event_data.go
So it is normal to be able to use either 1 or esdb.ContentTypeJson.

Hey @karthiksedoc,
I'm a bit confused by your question. See esdb.ContentTypeJson is a regular integer as shown at its declaration site.
https://github.com/EventStore/EventStore-Client-Go/blob/v4.2.0/esdb/event_data.go
So it is normal to be able to use either 1 or esdb.ContentTypeJson.

Sorry. I might be not setting it properly:
When I set
        eventData := esdb.EventData {
                ContentType: esdb.ContentTypeJson,
                EventType: "TestEvent",
                Data: data,
        }

I get the follow:
➜  gorpc_test git:(main) go build main.go
# command-line-arguments
./main.go:42:21: undefined: esdb.ContentTypeJson

the build works without error when setting:
	eventData := esdb.EventData {
		ContentType: 1,
		EventType: "TestEvent",
		Data: data,
	}

I was following the example and encountered this 🙇


MobileNetV4 support in Model Maker

MediaPipe Solution (you are using)
Model Maker
Programming language
No response
Are you willing to contribute it
No
Describe the feature and the current behaviour/state
MobileNetV4 is the latest generation of MobileNets, featuring numerous improvements. It'd be nice to add the possibility of using this new generation instead, since Model Maker is currently stuck at MobileNetV2
Will this change the current API? How?
No response
Who will benefit with this feature?
All users of Model Maker for image classification and object detection
Please specify the use cases for this feature
Improvement on an existing feature
Any Other info
No response

HI @Tachi107,
Thanks for the suggestion. Unfortunately the team currently doesn't have bandwidth to update the MobileNet architecture in the near future.
If there is a tfhub or kaggle checkpoint of MobileNetV4, then you can try to add it in to the uri of the ModelSpec for your use cases.

This isn't a bug, but a feature request. Therefore it should be kept open to track the feature request, or closed as wontfix. There's no reason to mark this as "stale".


Filter airports based on runway length

Any chance for this? Getting jets into grass strips is a bit odd.
Thanks!

Well seems like you can!


Layout images alt text

The background is: Remove alt text from images that serve only a layout purpose and add nothing to the content.
Changes are approved, thanks @ibexad !


Unzip failure with v1.0.15

Hi,
We've been running v1.0.14 with no problems. We just updated to v1.0.15 and now its failing on invocation with:
Error: invalid signature: 0x88b1f\n at /var/task/node_modules/unzip/lib/parse.js:59:13\n at runCallback (timers.js:794:20)\n at tryOnImmediate (timers.js:752:5)\n at processImmediate [as _immediateCallback] (timers.js:729:5)

Any ideas why? Seems like its related to unzipping the browser zip file maybe? Its the same file we've been using since v1.0.13

@AdamAH   there is no changes , can you double check the exactly version of  unzip ?
here is the version in package.json "unzip": "^0.1.11" , the mini version maybe different
also please have a look https://github.com/EvanOxfeld/node-unzip/issues/89

use unzipper instead of unzip to fix this problem


Provide an archive mode to work on past forecast data

Maybe the best is to be able to extract data from archived large GRIB files, like download process does, in order to work as in real-time forecast data:

https://www.unidata.ucar.edu/software/thredds/current/tds/
http://suite.opengeo.org/docs/latest/geoserver/services/wcs/index.html

Should be better to directly work on raw GRIB/Tiff data. In this context streamed probing does not make sense as well. Not sure if it is worth it to include this in Weacast, might be a separated project.

Archiving could be managed in loaders only. Archiving probes might make sense in order to make statistics or find specific conditions for eg training.
Linked to https://github.com/weacast/weacast-loader/issues/16 & https://github.com/weacast/weacast-probe/issues/16.


Remove debug logging

This change will remove debug logging from curl that was left behind.

Can we get the above pull request accepted?


get.sh: use standard syntax for locating javac

Fixes https://github.com/adoptium/infrastructure/issues/2695
I think this should be equivalent to the old code and work properly on a find command that does not have the -path parameter so will be more portable.
Original failing job listed in the PR: See https://ci.adoptopenjdk.net/job/Grinder/5391
Re-run against this PR branch: See https://ci.adoptopenjdk.net/job/Grinder/5393
If reviews can verify that it looks like it's done the right thing in that job it would be appreciated :-)
Signed-off-by: Stewart X Addison<EMAIL_ADDRESS>

This needs grinders to be run for primary platforms.

This needs grinders to be run for primary platforms.

Validation runs with TARGET=jdk_io

sparcv9_solaris: Grinder#5526 (NOTE: Grinder#5524 is a new re-run of the failing condition)
x86-64_solaris: Grinder#5528
x86-64_linux: Grinder#5529
x86-64_windows: Grinder#5530
x86-64_mac: Grinder#5531

https://github.com/adoptium/aqa-tests/pull/3918#issuecomment-1232772527

The tap files formats ( directory information)  looks different. @sxa is that related with this PR?

@sophia-guo Could that have come from having the upstream test job name of Test_openjdk8_hs_extended.openjdk_sparcv9_solaris from the original grinders I based the other platform runs on?

Suspiciously similar to https://github.com/adoptium/aqa-tests/issues/3579

Rebuilding with the UPSTREAM_TEST_JOB_NAME and UPSTREAM_TEST_JOB_NUMBER blanked out:
sparcv9_solaris: Grinder#5572
x86-64_solaris: Grinder#5568
x86-64_linux: Grinder#5570
x86-64_windows: Grinder#5571
x86-64_mac: Grinder#5574

Above all LGTM

zos: Grinder


Implement new snapshotting system with support for elastic memory growth via mmap()

This PR replaces the existing snapshot system in pkg/noun/events.c with a conceptually similar but new implementation. This new implementation removes the need to mmap() a large chunk (i.e. 2GB) of memory when a ship launches. Instead, it creates a file-backed memory mapping for the snapshot and then lazily maps new pages that lie outside of the snapshot when necessary. When a new snapshot is captured, all anonymous mappings are removed and the snapshot files are remapped, leading to a minimization of the Urbit runtime's memory footprint. This works draws inspiration from @joemfb's work in https://github.com/urbit/urbit/pull/6063 and https://github.com/urbit/urbit/pull/6152.
In addition to the functionality described above, this PR also makes the snapshotting system a largely orthogonal component relative to the rest of the runtime. As much as possible, the implementation tries to be simple to read and understand. It's also unit tested. Finally, as an added benefit, SIGSEGV raised as a result of non-loom address accesses now generate a segfault as expected rather than complaining of "address out of loom" (thanks to sigsegv_init() and sigsegv_dispatch()).
Remaining tasks:

[x] Benchmark to determine performance differences compared to old snapshot system.
[x] Document the interfaces introduced in pkg/pma.
[x] Remove pkg/noun/events.c and pkg/noun/events.h.

Resolves #188.
Testing
Functionality
All unit tests pass, including the new [pkg/pma/pma_tests]:
$ bazel test --config=test --build_tests_only ...

Also booted comets on macos-aarch64 and linux-x86_64 using the 1.20, 1.21, and 1.22 pills and successfully sent a DM to my planet. For example, testing the 1.21 pill:
$ bazel build :urbit
$ ./urbit -c some-comet -u https://bootstrap.urbit.org/urbit-v1.21.pill

Also booted a comet using vere-v1.18, exited, and then successfully ran that comet and sent a DM using the binary built from the tip of this branch:
$ git checkout vere-v1.18
$ bazel build :urbit
$ ./urbit -c some-comet -u https://bootstrap.urbit.org/urbit-v1.18.pill
$ git checkout i/188
$ bazel build :urbit
$ ./urbit some-comet

Performance
Ran pkg/vere/benchmarks.c 20 times on both i/188 (commit SHA 67d596c7fa) and develop (commit SHA dee0cef167) and computed the following averages:

Benchmark
i/188
develop

jam og
24.9ms
24.55ms

jam xeno
15.55ms
14.4ms

jam cons
188.1 ms
183.6

jam cons with
5.15ms
7.35ms

cue og
145.4ms
149.35ms

cue atom
30.4ms
31.15ms

cue xeno
31.1ms
30.35ms

cue xeno with
14.8ms
15.0 ms

cue test
21.5ms
21.4ms

cue test with
6.6ms
6.85ms

cue cons
352.5ms
345.6ms

cue re-cons
542.45ms
557.25ms

cue virtual og
130.4ms
129.75ms

cue virtual atom
20.15ms
19.5ms

The linux-aarch64 is failing because pma_tests are exhausting the disk space on the (small) linux-aarch64 VM. I'll debug further first thing Monday.

@mcevoypeter I don't think this is related to your changes. #255 is now hitting this too, even though it hasn't touched anything that ought to cause such errors. I tired adding a commit that removes pre-installed libs from the GitHub runner image, but it didn't help. I also tried using this GitHub workflow action, but it's not on the approved list and so caused the workflow to fail.
The failures might have something to do with the outage GitHub experienced yesterday?

@ashelkovnykov  The linux-aarch64 runner is self-hosted, and it turns out the VM it's running on has a default tmpfs size of 1GB. The PMA tests create a 1GB backing file in /tmp, which exhausts tmpfs, causing the test to fail without cleaning up the file, leaving tmpfs completely full. This then causes other jobs run on the runner to fail because parts of the build process write to tmpfs as well. The solution was to increase the size of tmpfs to 2GB (see https://unix.stackexchange.com/questions/442944/tmpfs-usage-and-resizing).

Hi Peter, this is neat! I've been working on fixing the bugs we have in our current snapshot system (namely the strange page issue [we merged a PR for this] and when we crash after a guard page re-centering error [still WIP]).
If you see #234 you can see that a reliable reproduction for the guard page crash is to boot a ship with 1.10 (%zuse 418) and then try and upgrade it to %zuse 415 with the latest binary.
I went ahead and ran the same process with a fakezod using a binary built with this branch i/188, and it crashed unfortunately. Igot the following trace:
...
>   arvo: +load next
lull: ~lasbex-sonseg
zuse: ~ligfep-radfyn
vane: %ames: ~hatmeg-lavpun
ames: larva reload
vane: %behn: ~sogmut-tonmus
vane: %clay: ~fildux-polhes
           allocate: reclaim: half of 2139 entries
allocate: reclaim: half of 1069 entries
allocate: reclaim: half of 534 entries
allocate: reclaim: half of 267 entries
allocate: reclaim: half of 133 entries
allocate: reclaim: half of 66 entries
allocate: reclaim: half of 33 entries
allocate: reclaim: half of 16 entries
allocate: reclaim: half of 8 entries
allocate: reclaim: half of 4 entries
allocate: reclaim: half of 2 entries
allocate: reclaim: half of 1 entries
allocate: reclaim: memo cache: empty

bail: meme
allocate: reclaim: half of 2064 entries
pier: serf unexpectedly shut down

matthew@domus:~/ships/dev$

Is this PR meant to fix this specific issue? Perhaps the issue isn't related to the snapshot system at all? I haven't yet determined the root cause of this crash in particular, but I'm also still deep in my inspection of backtraces. What do you think?

@matthew-levan this PR does not address #234, but instead aims to reduce memory pressure on the host system by the runtime by mmap()ing a page at a time on demand rather than mmap()ing the entire loom up front. I'm not surprised to see it fails on memory exhaustion.

The PR description previously mentioned that the guard page had been removed temporarily. That's no longer true; I reimplemented the guard page today.

@mcevoypeter, I just tried running the tests on my M1 and got this:
//pkg/pma:pma_tests                                                      FAILED in 1.2s
  /private/var/tmp/_bazel_matt/657ca343b546ec28817b7a65bf3b8da7/execroot/__main__/bazel-out/darwin_arm64-fastbuild/testlogs/pkg/pma/pma_tests/test.log

test.log

exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //pkg/pma:pma_tests
-----------------------------------------------------------------------------
pma: failed to create<PHONE_NUMBER>-byte mapping for /tmp/definitely-exists-heap.bin at 0x200000000: Permission denied
Assertion failed: (pma_), function test_pma_, file pma_tests.c, line 204.

I've made the following changes after a live code review with @ashelkovnykov, @joemfb, and @philipcmonk:

Just-mapped pages are marked as dirty, not clean.
A redundant wal_apply() call in map_file_() has been removed.
The unmapping logic at the end of pma_sync() has been removed.
The write-ahead log is validated by recording the checksum of each page in the write-ahead log.

@matthew-levan I suspect the issue you're seeing is related to the fixed address pma_tests is using as the base address for the arena. I'll take a closer look.

First, a question on style: How does this code fit in with the rest of the codebase? The PMA is written in idiomatic C, and vere is obviously far from that. Does having both idiomatic C and c3-style C in the same codebase introduce confusion? Genuinely curious about these thoughts from those with more experience with this codebase.

The PMA is in a separate directory than the other directories which use c3-style C, so the styles aren't mixed within files, libraries, or binaries. If the PMA introduced a similarly unusual C style alongside the existing unusual c3 style, then I agree it might cause confusion, but idiomatic C is easy to read for any C programmer, making it a safe choice.

On design, will you please summarize your approach vs. the demand paging implementation in 1.14? It'd be nice to understand the differences and corresponding justifications/arguments.

This implementation creates file-backed mappings for north.bin and south.bin, whereas the 1.14 implementation only creates a file-backed mapping for north.bin.
This implementation lazily maps new pages, whereas the 1.14 implementation maps the entire loom up front.
This implementation uses a local SIGSEGV handler to only handle SIGSEGV generated by addresses within the loom, whereas the 1.14 implementation uses a global SIGSEGV handler that handles all SIGSEGV generated by the runtime.
This implementation defines the abstraction of a persistent memory arena, whereas the 1.14 implementation does not.

Lastly, it's clear that this (more or less) works as intended: a few minutes after boot, a fresh fakezod will reduce its memory footprint to a mere ~75MB, which is quite an improvement! Neat.

Nice! On a similar note, I'm running my planet and star with the changes, and I saw the memory usage of both drop from ~450MB to ~150MB upon switching when running top -pid <king_pid> -pid <serf_pid>.

Awesome thanks for elaborating @mcevoypeter. I think this work has a lot of merits. What's left?

After speaking with @philipcmonk, we've decided to pass on #293 in favor of a u3e_ward()-style function to manually recenter the guard page since we're still unsure of the implications of shrinking the pad as described in #292. He also suggested adding a global checksum to the write-ahead log validation process to make our corruption detection more robust. I'll make both changes by the end of the day today.

As another data point, I tried running this on my planet.  On develop, the serf uses almost exactly 2GB.  On this branch on start it went to 800MB, then after a couple minutes dropped briefly to 200MB and quickly climbed up to 524MB.  When I navigated to grid and groups, it climbed again, peaking just under 1GB.  When I leave it alone for a couple minutes, it drops to about 300MB.  This all sounds basically correct to me.
It feels like much of that 1GB is probably ephemeral memory anyway.  @mcevoypeter mentioned that a later improvement could be to map the free memory to another file with MAP_SHARED (like a bespoke swap file), and that would allow even that memory to be written to disk when under memory pressure.

Playing around with this a bit I ran
=(~ =|(i=@ |-(?:(=(i ^~((bex 23))) ~ [i $(i +(i))]))))
on a fresh comet and the serf's memory shot up to 672MB (as expected).
I waited a bit for it to go down and no such luck. Restarting the process reduced the memory back to 100MB.

Playing around with this a bit I ran
=(~ =|(i=@ |-(?:(=(i ^~((bex 23))) ~ [i $(i +(i))]))))
on a fresh comet and the serf's memory shot up to 672MB (as expected).
I waited a bit for it to go down and no such luck. Restarting the process reduced the memory back to 100MB.

I'm seeing the same thing, at least per RES in htop.  I guess this is because the contiguous free space is not file-backed, so we would somehow need to tell the OS to throw it away periodically?  Not sure if there's an madvise call for this.

to tell the OS to throw it away periodically? Not sure if there's an madvise call for this.

MADV_DONTNEED

Does that get automatically unset once we start using that memory again?

I mainly have questions about the general utility of a timer-based RSS reducer that this essentially functions as -- mapping the home road to a file-backed mapping, collecting ephemeral state in page-sized anonymous mappings, on a u3e_sync (or whatever it's called now ... pma_sync), expanding/contracting the persistent memory as necessary and unmapping all the anon mappings - thus reducing the RSS size of the process.
But this doesn't enable you to run vere on devices with less memory - we haven't reduced the overall memory footprint of vere across time - just every two minutes or so we try to shrink the RSS of the process. Since the ephemeral state is composed of anon mappings, this may be of some use since they can't be paged out. But whenever your urbit isn't idle, it's going to fill up that RSS again. And overall, there will still be much pain running urbit on a system where memory is in contention.
I think the same general behavior could be accomplished with a simpler snapshotting scheme that mmaps the maximum size of a loom and additional logic that madvises MADV_DONTNEED when we fall back to a parent road. That way RSS is reduced ASAP rather than every X minutes. Additionally, with the ephemeral data being written to file-backed memory, it could be swapped out in low memory situations. Such changes may be more easily implemented with a single consolidated loom image (moving the global u3H to before the rut of the home road (rather than mat where it current lies)).

I mainly have questions about the general utility of a timer-based RSS reducer that this essentially functions as ....
But this doesn't enable you to run vere on devices with less memory - we haven't reduced the overall memory footprint of vere across time - just every two minutes or so we try to shrink the RSS of the process.

I'm not sure what this means, and whether your comment is referring to the entire PR or the proposal to MADV_DONTNEED the free space?
Even if you occasionally need more memory, it is useful for the process to take less memory most of the time.  This is especially the case anywhere you're trying to pack in many ships on the same machine.

But whenever your urbit isn't idle, it's going to fill up that RSS again.

Only whatever gets read, and quite a bit hardly ever gets read.

I think the same general behavior could be accomplished with a simpler snapshotting scheme that mmaps the maximum size of a loom and additional logic that madvises MADV_DONTNEED when we fall back to a parent road. That way RSS is reduced ASAP rather than every X minutes. Additionally, with the ephemeral data being written to file-backed memory, it could be swapped out in low memory situations.

If you mean as a replacement for the whole PR, then I think this doesn't work because we can't let the OS write to the snapshot on its own.  The snapshot has to be updated atomically, which is what the WAL gives us.

I'm not sure what this means, and whether your comment is referring to the entire PR or the proposal to MADV_DONTNEED the free space?

It's in reference to the entire PR.

Even if you occasionally need more memory, it is useful for the process to take less memory most of the time. This is especially the case anywhere you're trying to pack in many ships on the same machine.

I agree that minimizing the RSS of vere is useful. But I'm not sure that a per page anon mapping like this PR performs is the only (certainly not simplest) way to accomplish that. That said, there are no other competing implementations ;).

Only whatever gets read, and quite a bit hardly ever gets read.

@philipcmonk can you elaborate? I'm not sure I understand. When urbit isn't idle, we're some number of child roads deep. All of that ephemeral space is allocated with anon mappings and naturally resident in the process. It's not until we take a snapshot that we can evict that ephemeral space from memory and reduce RSS. So the nature of the process' resident memory usage is that it now fluctuates a lot more (with admittedly a lower floor), but it can still expand just as high as before.

if you mean as a replacement for the whole PR, then I think this doesn't work because we can't let the OS write to the snapshot on its own. The snapshot has to be updated atomically, which is what the WAL gives us.

Agreed. We would still need an atomic update mechanism. Dirty page tracking and redundant sparse snapshots (.bhk) likely previously could work, so could a WAL. Maybe there are other mechanisms.

The "entire loom" part of that vague design isn't necessary either. It could be accomplished like the changes in 1.14 where the home road is file-backed and the ephemeral space is one contiguous anon mapping. But I think it should be possible to evict from resident memory that ephemeral data we don't need when falling back to a parent road rather than as a function of time with judicious use of madvise.

If we don't read from some data on the home road (eg ancient clay files, chat backlog, or cold jet state, which are the three largest things on my ship), it doesn't have to be resident.  And even if it's read in the middle of an event, it can be evicted before the event finishes.  It can expand as high as before (and in particular, |mass and |pack will likely make everything resident), but it rarely will.
Suppose we alter this PR to MADV_DONTNEED after every event -- then how does this differ from your proposal?  As I understand it the main difference is that the free space is also file-backed, which @mcevoypeter considered as a later optimization, but probably a good idea.  Although you seem to propose doing it in the same file, which doesn't seem like it has much of an impact either way.  In particular, I believe to make your proposal work, the home road needs to be MAP_PRIVATE while the free space should be MAP_SHARED, right?

If we don't read from some data on the home road (eg ancient clay files, chat backlog, or cold jet state, which are the three largest things on my ship), it doesn't have to be resident.

Yes absolutely. Regarding memory usage, I was mainly comparing behavior of the 1.14 implementation (to which this also applies) and this PR.

Suppose we alter this PR to MADV_DONTNEED after every event -- then how does this differ from your proposal?

I think it would be a bit more difficult to do this with the current "mmap on a page fault" that this PR does. It would be simpler to madvise all ephemeral data when we fall back to a parent road. But we can't do that if we haven't mmaped that data. If you track the high page of the child road -- perhaps? As a side note though, MADV_FREE as opposed to MADV_DONTNEED is probably correct since it's lazy and falling and quickly leaping (thus using that same region of memory again) is a fairly common pattern.

In particular, I believe to make your proposal work, the home road needs to be MAP_PRIVATE while the free space should be MAP_SHARED, right?

Yes home road needs to remain CoW. There may be other problems. If we can't rely on the backing file to be sparse.

AIUI he changed it so that we don't munmap anymore, and I believe PS_UNMAPPED is completely unused now.
This PR registers a SIGSEGV handler which marks touched pages as dirty, so that when we sync, we know which pages need to written to the WAL and then writing to the backing file.  You're not proposing changing that necessarily, right?
Can you more explicitly point out what your proposal would do differently during SIGSEGV handling, at the end of each event, and on sync?

I've avoided discussing style, but I have to ask: why the underscore suffix as a prefix to demarcate static functions and variables? The style also seems to have been applied to a number of local variables -- extensively in tests, e.g. pma_tests.c -- but also in a few locations in non-test code. grep pma.c for "fd_", "len_", "i_", more?

As a basic benchmark, I would like to know how a brass pill boot compares with/without this PR -- not to test the speed of snapshotting (since it won't be invoked), but to compare the speed of the new SIGSEGV handling logic.
Also, interested in how a basic benchmark of snapshot latency compares with/without these changes. I don't think the performance tests in pkg/vere/benchmarks.c mentioned capture this -- do they?
I have a branch doing a basic bench with struct timespec here: https://github.com/urbit/vere/tree/barter-simsum/freelist-sane-bench.
Overall snapshot time is obviously proportional to number of dirty pages. Curious of differences with/without changes for:

no urbit interaction
after allocating a large noun

I understood that you changed this to mmap the whole loom up front, is that correct?

We're still lazily mapping a page at a time. The reason for the lazy mapping is because it's unclear how mmap() the whole thing up front impacts memory usage in the context of a pod running in a Kubernetes cluster, which is of concern for hosting providers. I can easily switch it to mmap() the whole loom as MAP_PRIVATE | MAP_ANON and then madvise(MADV_DONTNEED) the entire loom to ensure nothing is made resident prematurely.

RE brass pill boot:
Looks good. Appears negligible difference between the two (when comparing 16K).
Makes me wonder if we should go ahead with 64K pages though

Here are the snapshot latency measurements using the code in barter-simsum/freelist-sane-bench. Based on these measurements pma_sync() runs in in ~50% to ~85% of the time compared to u3e_save() depending on the scenario.

Commit
Description
Snapshot latency on boot
Snapshot latency on exit (no interaction)
Snapshot latency on exit (after running |mass)

847a899f27
Tip of barter-simsum/freelist-sane-bench
0.186716s
0.045495s
0.125963s

f116320e60
16KB pages on i/188
0.091422s
0.038471s
0.093622s

847a899f27
64KB pages on i/188
0.090197s
0.038478s
0.091637s

do we see any additional gain by going higher? e.g. 128K

Stand by. I'll test 128KB pages.

why the underscore suffix as opposed to a prefix to demarcate static functions and variables?

I was operating under the assumption that _[a-z]+ is a reserved identifier in C, but it turns out that regex is only reserved in the global scope, not the static scope. See https://devblogs.microsoft.com/oldnewthing/20230109-00/?p=107685.

The style also seems to have been applied to a number of local variables -- extensively in tests, e.g. pma_tests.c -- but also in a few locations in non-test code. grep pma.c for "fd_", "len_", "i_", more?

The usage in tests was a mistake, and I'm happy to drop the use of the trailing underscore in variable names there. fd_ and len_ are used when fd and len, respectively, are already in scope. The use of i_ is a mistake.

Most recent changes incorporate style feedback from @barter-simsum and add file-level documentation.

I've addressed what I believe is all of the major points of feedback. These are:

Global checksum and per-entry checksum in WAL to detect corruption.
pma_adjust() to account for non-standard memory accesses (such as leaping to an inner road).
madvise(MADV_DONTNEED) when the heap and/or stack has shrunk to inform the kernel the truncated part of the heap and/or stack is no longer needed.
More thorough performance testing.

I've also tried as much as possible to address the smaller points of feedback that were raised. The big picture is that using 64KB pages, this PR demonstrates a noticeable increase in performance and decrease in memory usage compared to the existing snapshot implementation. It's also documented and unit tested, and I've been running it on both my planet and star for over a week and a half now without issue. Given this, what blockers remain for merging this? cc @barter-simsum @jalehman @philipcmonk

Current blockers: tests are running.
Then we'll merge.

I'm not sure MADV_DONTNEED is really what you want as opposed to MADV_FREE since the other is lazy and falling into a parent road and quickly leaping back into a child is a common pattern
@mcevoypeter

Using 64KB pages trades-off write volume for fault frequency. Characterizing the performance improvements from that requires testing different scenarios -- boot is biased towards it since all the persistent pages are new and will be dirty regardless.

Some combination of memory protections, advice, and remapping should be sufficient to keep the resident set small on any OS we support. If this turns out to be wrong, and Kubernetes (for instance) requires more aggressive intervention, we'll have to explicitly, lazily/incrementally request memory from the OS (in the allocator, on road transitions, via "guard" pages to grow, &c). But that will be quite a bit more work, and I don't believe it will be necessary.

I really don't think there's anything more aggressive than MADV_DONTNEED - at least Linux's interpretation of it - which is explicitly non lazy and advises the kernel that the process doesn't expect access in the near future. An immediate read on a region madvised with DONTNEED will read in zeroes as if the same region had been freshly mmaped anonymously (or the contents of the backing file, but that isn't relevant to our usage). This has the effect of immediately reducing RSS.
It's also possible that this is too aggressive and madvising with MADV_FREE (lazy) is better. I have doubts there will be a significant performance difference in vere. Additionally, since it's lazy, advising with FREE won't necessarily immediately drop rss. I have doubts though that that would actually be a problem for kubernetes. I'd imagine on memory pressure, it would try clawing back whatever resident memory it can from each linux container.

After our group review, I've made a few fixes:

removed the MADV_DONTNEED immediately after mmap in pma_load.  mmap doesn't make anything resident, so it's unnecessary.
removed the redundant PS_MAPPED_INACCESSIBLE state and instead check the guard page address, which we store anyway.
removed PS_UNMAPPED now that everything is always mapped.
This reduces the number of page states to two: clean or dirty.  The default state is dirty, so that if we unexpectedly fault on a page (perhaps due to bad initialization), we will crash in _handle_page_fault.
At the end of pma_sync, we no longer loop to mark those pages as clean.  It doesn't matter what state they're in, since _append_dirty_pages is bounded by the actual size of the heap/stack.
Instead, we MADV_DONTNEED the ephemeral space.  This allows us to reclaim ephemeral memory on sync.  As noted in the comment, this is likely fast enough to do after every event, at least if we use MADV_FREE, which is lazy.  For now, we use the strict MADV_DONTNEED to make it easier to observe its behavior.

This last change greatly reduces the average memory use of the process by fixing the issue @mopfel-winrux reported here: https://github.com/urbit/vere/pull/277#issuecomment-1478404107
I believe these are all the blocking changes we identified, but we did not finish walking through the code.  @joemfb and everyone else, let's continue that soon.

We used to write the snapshots to .bhk every time we u3e_saved (by calling u3e_backup). We're no longer doing that in pma_sync. u3m_backup replaced the functionality  of the former, but that only gets called on pier startup and on chop. Is that intentional? I didn't see it explicitly called out anywhere. @mcevoypeter  @joemfb

When did we start doing that? I remember bhk being used exclusively for when you upgraded to v1.8 (or thereabouts), and then it got repurposed to be used every chop, which seems proper. I don't remember it happening more often than that, and I feel like you don't want it to -- if you're constantly backing up, then unless corruption is caught immediately (in which case you don't need the backup), your backup will be corrupted.

@philipcmonk
misread the u3e_backup implementation.
Looks like 3c13a6d271a6670699a157c2d4ed80c6c969b183 added a call to _ce_backup which only backed up the pier if a bhk didn't already exist (fine).
origin/develop has some changes from Matt that changed _ce_backup to u3m_backup.  A conditional is passed in the call that if c3n (it is in the u3e_save call) will exit early and not take the snapshot. That conditional is c3y (forces the backup) in the _cw_chop.

Note to future self: @joemfb and @philipcmonk are working together on this offline.


support for /api/mysql-relay-log-index-file

/api/mysql-relay-log-index-file heuristically returns the name of the relay log index file.
There is a hidden assumption that this file lies under the MySQL datadir. This assumption may be lifted in the future.

Also of interest, sorry that I incorporated it withing this PR, is TokenHttpHeader. IT is now possible to pass the token as HTTP header, instead of query param. To do so, specify header name in TokenHttpHeader.

support for /api/mysql-relay-log-files: list of relay logs. There is still the assumption that they are located under the datadir.


Add support for login field

Proposed fix for https://github.com/google/go-github/issues/759

I signed it!


proguard NoSuchFieldError

seems like adding only:
-keep class com.commonsware.cwac.** { *; }
to the proguard.pro is not enough. perhaps i misunderstood your meaning in the readme. but i had to add the eventbus proguard rules as well (from what you wrote, i got the feeling like the cwac keep should also keep all the eventbus classes).
also, the link you supplied for the rules is broken. it should be updated to:
https://github.com/krschultz/android-proguard-snippets/blob/c636c6c8448565e91e25a66f11fab3fca035c29c/libraries/proguard-eventbus-3.pro

I have made some adjustments to the README, including fixing the broken link. Thanks!


GET a file while it's in processing doesn't return 425 code (async uploads)

Steps to reproduce

Start oCIS with these envvars:

PROXY_ENABLE_BASIC_AUTH=true \
STORAGE_USERS_OCIS_ASYNC_UPLOADS=true \
OCIS_EVENTS_ENABLE_TLS=false \
POSTPROCESSING_DELAY=30s \
./bin/ocis server

Upload a file

curl -XPUT "https://localhost:9200/remote.php/dav/files/admin/lorem.txt" \
-d"uploaded" -uadmin:admin -vk

Try to GET the file using old, new and spaces endpoints

Old
curl -XGET "https://localhost:9200/remote.php/webdav/lorem.txt" -uadmin:admin -vk

New
curl -XGET "https://localhost:9200/remote.php/dav/files/admin/lorem.txt" -uadmin:admin -vk

Spaces
curl -XGET "https://localhost:9200/dav/spaces/5c3aacd7-e23d-43b0-89ec-ca4c67ae02ef%24cae6d2cf-a0dc-4e1a-8354-20b74ef12f34/lorem.txt" \
-uadmin:admin -vk

Expected behavior
to return status code 425 as mentioned in https://github.com/owncloud/ocis/issues/4095#issuecomment-1359241819
Actual behavior
returns 500 internal server error
< HTTP/1.1 500 Internal Server Error
< Access-Control-Allow-Origin: *
< Content-Length: 0
< Content-Security-Policy: default-src 'none';
< Date: Wed, 04 Jan 2023 11:25:41 GMT
< Vary: Origin
< X-Content-Type-Options: nosniff
< X-Download-Options: noopen
< X-Frame-Options: SAMEORIGIN
< X-Permitted-Cross-Domain-Policies: none
< X-Robots-Tag: none
< X-Xss-Protection: 1; mode=block
<
* Connection #0 to host localhost left intact

Setup
oCIS: built from master branch

fix landed

Confirmed:
old dav: curl -XGET "https://localhost:9200/remote.php/webdav/lorem.txt" -uadmin:admin -vk
new dav: curl -XGET "https://localhost:9200/remote.php/dav/files/admin/lorem.txt" -uadmin:admin -vk
spaces dav: curl -XGET "https://localhost:9200/dav/spaces/17f05810-47dc-4442-84f7-6e4fc4ad59c5\$0dec08c6-65a4-4d29-b2e3-d63e1a613a4e/lorem.txt" -uadmin:admin -vk
Response:
< HTTP/1.1 425 Too Early
< Access-Control-Allow-Origin: *
< Content-Length: 0
< Content-Security-Policy: default-src 'none';
< Date: Fri, 10 Mar 2023 04:02:15 GMT
< X-Content-Type-Options: nosniff
< X-Download-Options: noopen
< X-Frame-Options: SAMEORIGIN
< X-Permitted-Cross-Domain-Policies: none
< X-Robots-Tag: none
< X-Xss-Protection: 1; mode=block


feat: Register more @[bound] lemmas for floor, ceil, and log

Int.ceil_lt_add_one and Real.log_pos came up in https://github.com/girving/ray, and I've registered a variety of nearby lemmas.

@girving I'm tempted to trust you, and just merge this, but I'm waiting because I have a larger question, which I believe needs answering:
What constitutes a bad @[bound] lemma?
My concern is that without this kind of guidance, we'll end up tagging lemmas with @[bound] that cause extremely serious performance degradation. Moreover, it may not show up immediately with !bench. Is there somewhere this is addressed that I have missed?

The main performance constraint is that expressions on either side of inequality hypotheses should be strict subexpressions of the conclusion (unless they are very simple expressions like 0 or 1), to (1) avoid loops and (2) adhere to the general goal of bound. However, I am not sure what the precise formalization of this should be, or whether there is a precise formalization that definitely avoids loops.
Note that this requirement holds vacuously if there are no hypotheses, which is true of a lot of the non-gcongr-like lemmas tagged here. Real.log_pos shows why you need "unless they are very simple", as it turns 0 < log x into 1 < x where 1 doesn't occur in the goal.

If we can find a satisfactory version of this which is programmatic I could make a test enforcing it, or enforce it at attribute time.

!bench

if you are able to write some tests to prevent some bad uses of the attribute, I encourage you to do so; that would be well-received! But that seems like it's material for a follow up.
maintainer merge
(@j-loreaux)

bors merge

@joneugster Thank you! I will eventually write the subexpression test, but there may be a delay involved.


Rewrite documentation to account for API changes

We're going to need to change our API blueprint documentation to account for the API changes that have been incurred during this migration to Phoenix.

This is related to #26.


cant create tests because of bad captcha handling?

everytime I try to add\edit tests cloudflare take me to the captcha page - all good the problem is after completing successfully the captcha the form get reset and I need to refill the details(and then again redirected to captcha page after submitting)
I don't know what side the bug is but please keep the form post after the captcha success

Sorry you're running into issues with Cloudflare. Unfortunately, that's entirely separate from the application code here in this repository. Your best course of action may be to work with Cloudflare directly.

Yes, it's completely cloudfare's fault, not jsperf's.. lol what fine user support


Assistant hears themselves when speaking - Any tips?

Not sure if this is something I need to implement in my code, or an audio equipment issue.
This happens during server_vad mode.
When the assistant is chatting with me, it will hear it's own voice through my speakers and then automatically pipe that into the next input.
Obviously, we could implement client side VAD (push to talk). Or when in server_vad mode, I could automatically pause my microphone while the assitant is responding, however this makes it hard for people to interrupt the assistant when they are talking.
Or should I just try to separate my microphone and speakers with different equipment.
Has anyone else run into a similar issue?

yeah, same issue i am facing


Use of policies.json

Right now there's a default policies.json which can't be changed as it gets overwritten/restored every time "phyrox-esr-portable.exe" is started. I have a customized policies.json file I'd like to use but it's not possible at the moment. Would it possible to add a argument for "phyrox-esr-portable.exe" on startup  to not touch the policies.json if it has been modified or perhaps some other workaround ?

You can write  /data/policies.json
It will be applied
but
{
"Disablefirefoxstudies": True,
"DisableTeleTry": True
}
Unable to change


On Windows

"pip install pianoplayer" gives
ERROR: Cannot install pianoplayer==1.0.0, pianoplayer==1.1.0, pianoplayer==1.1.1, pianoplayer==2.0.0, pianoplayer==2.0.1, pianoplayer==2.0.4, pianoplayer==2.0.5, pianoplayer==2.0.6, pianoplayer==2.0.7 and pianoplayer==2.1.0 because these package versions have conflicting dependencies.

please post the complete error message.
you may also try
pip install pianoplayer --no-deps