archive/zip: very large zip file cannot be read by unzip or OS X archive utility

I have an approximately 8 GB ZIP file created using archive/zip.  zipdetails -v on the file ends with:
1FD0EB303 000000004 50 4B 01 02 CENTRAL HEADER #127F  02014B50
1FD0EB307 000000001 14          Created Zip Spec      14 '2.0'
1FD0EB308 000000001 03          Created OS            03 'Unix'
1FD0EB309 000000001 14          Extract Zip Spec      14 '2.0'
1FD0EB30A 000000001 00          Extract OS            00 'MS-DOS'
1FD0EB30B 000000002 08 00       General Purpose Flag  0008
                                [Bit  3]              1 'Streamed'
1FD0EB30D 000000002 00 00       Compression Method    0000 'Stored'
1FD0EB30F 000000004 16 A5 4B 46 Last Mod Time         464BA516 'Wed Feb 11 20:40:44 2015'
1FD0EB313 000000004 B0 92 2C B7 CRC                   B72C92B0
1FD0EB317 000000004 FF FF FF FF Compressed Length     FFFFFFFF
1FD0EB31B 000000004 FF FF FF FF Uncompressed Length   FFFFFFFF
1FD0EB31F 000000002 24 00       Filename Length       0024
1FD0EB321 000000002 1C 00       Extra Length          001C
1FD0EB323 000000002 00 00       Comment Length        0000
1FD0EB325 000000002 00 00       Disk Start            0000
1FD0EB327 000000002 00 00       Int File Attributes   0000
                                [Bit 0]               0 'Binary Data'
1FD0EB329 000000004 00 00 A4 81 Ext File Attributes   81A40000
1FD0EB32D 000000004 FF FF FF FF Local Header Offset   FFFFFFFF
1FD0EB331 000000024 38 36 34 20 Filename              '864 Pics
                    50 69 63 73                       1.68gbs/20140807_053444.jpg'
                    20 31 2E 36
                    38 67 62 73
                    2F 32 30 31
                    34 30 38 30
                    37 5F 30 35
                    33 34 34 34
                    2E 6A 70 67
1FD0EB355 000000002 01 00       Extra ID #0001        0001 'ZIP64'
1FD0EB357 000000002 18 00         Length              0018
1FD0EB359 000000008 D4 23 2A 00   Uncompressed Size<PHONE_NUMBER>2A23D4
                    00 00 00 00
1FD0EB361 000000008 D4 23 2A 00   Compressed Size<PHONE_NUMBER>2A23D4
                    00 00 00 00
1FD0EB369 000000008 B8 DA DD FC   Offset to Central   00000001FCDDDAB8
                    01 00 00 00 Dir

1FD0EB371 000000004 50 4B 06 06 ZIP64 END CENTRAL DIR 06064B50
                                RECORD
1FD0EB375 000000008 38 00 00 00 Size of record<PHONE_NUMBER>000038
                    00 00 00 00
1FD0EB37D 000000001 2D          Created Zip Spec      2D '4.5'
1FD0EB37E 000000001 00          Created OS            00 'MS-DOS'
1FD0EB37F 000000001 2D          Extract Zip Spec      2D '4.5'
1FD0EB380 000000001 00          Extract OS            00 'MS-DOS'
1FD0EB381 000000004 00 00 00 00 Number of this disk   00000000
1FD0EB385 000000004 00 00 00 00 Central Dir Disk no   00000000
1FD0EB389 000000008 7F 12 00 00 Entries in this disk<PHONE_NUMBER>00127F
                    00 00 00 00
1FD0EB391 000000008 7F 12 00 00 Total Entries<PHONE_NUMBER>00127F
                    00 00 00 00
1FD0EB399 000000008 93 B4 06 00 Size of Central Dir<PHONE_NUMBER>06B493
                    00 00 00 00
1FD0EB3A1 000000008 DE FE 07 FD Offset to Central dir 00000001FD07FEDE
                    01 00 00 00
Unsupported Size (56) in Zip64EndCentralHeader

The zip was built using Go 1.4 or 1.4.1 on a 64-bit Linux system.
A Go reader on OS X can reread the file just fine.  This is my first foray into the internals of zip, so I am not sure how or why 56 is an unsupported size.
Smaller files (in the 3.7 GB range) are opened fine by unzip and the built-in OS X zip support.

Hmm, one possibility: section <IP_ADDRESS> of https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT says:

The value stored into the "size of zip64 end of central directory record" should be the size of the remaining record and should not include the leading 12 bytes.

By my count the record's total size is 56 bytes, so the "Size of record" field should be 44 (0x2c) rather than 56 (0x38).

Ok, opening a hex editor and changing that 0x38 to a 0x2c makes zipdetails happy, but it might be a red herring.  zip -FF fails in the same place regardless:
 Local ( 1<PHONE_NUMBER>): copying: 864 Pics 1.68gbs/20150114_202208.jpg   (1875410 bytes)
 Local ( 1<PHONE_NUMBER>): copying: 864 Pics 1.68gbs/20150114_202212.jpg
zip I/O error: Bad address
zip error: Output file write failure (write error on zip file)

as does unzip -l:
Archive:  RFM-rgriffin.zip
error [RFM-rgriffin.zip]:  missing 49368129 bytes in zipfile
  (attempting to process anyway)
error [RFM-rgriffin.zip]:  start of central directory not found;
  zipfile corrupt.
  (please check that you have transferred or created the zipfile in the
  appropriate BINARY mode and that you have compiled UnZip properly)

zipdetails output around the point where it dies:
1B4058E4A LOCAL HEADER #1029    04034B50
1B4058E4E Extract Zip Spec      14 '2.0'
1B4058E4F Extract OS            00 'MS-DOS'
1B4058E50 General Purpose Flag  0008
          [Bit  3]              1 'Streamed'
1B4058E52 Compression Method    0000 'Stored'
1B4058E54 Last Mod Time         464BA516 'Wed Feb 11 20:40:44 2015'
1B4058E58 CRC                   00000000
1B4058E5C Compressed Length     00000000
1B4058E60 Uncompressed Length   00000000
1B4058E64 Filename Length       0024
1B4058E66 Extra Length          0000
1B4058E68 Filename              '864 Pics 1.68gbs/20150114_202212.jpg'
1B4058E8C PAYLOAD

1B422CE8B STREAMING DATA HEADER 08074B50
1B422CE8F CRC                   F41015AF
1B422CE93 Compressed Length     001D3FFF
1B422CE97 Uncompressed Length   001D3FFF

I was able to decompress the whole zip without modification on a Linux box.  I wonder if it's a Mac OS-only issue.

Ok, what all of this has taught me is: large ZIP file support in OS X is garbage, and the failure modes are unkind.
The built-in archive utility doesn't support them, the version of unzip shipped with Yosemite doesn't include ZIP64 support, and the newer version that does support ZIP64 still blows up as soon as it hits 4 GB.  Meanwhile, Linux works through the files happily.
The only thing that might make sense is to adjust the "size of zip64 end of central directory record" to make zipdetails happy, but it doesn't seem to affect other tools (unzip, zipinfo) on Linux.

After your investigation I can't tell if there is anything for us to do here.  Should I go ahead and close out this issue.

@ianlancetaylor: I think we should fix the size field in the ZIP64 end-of-central-directory record to match the spec.  I'll write up a patch for that.

https://go-review.googlesource.com/4760 PTAL

On 13 February 2015 at 09:09, Joe Shaw<EMAIL_ADDRESS>wrote:

@ianlancetaylor https://github.com/ianlancetaylor: I think we should
fix the size field in the ZIP64 end-of-central-directory record to match
the spec. I'll write up a patch for that.

Sounds good. Please "git codereview mail
https://golang.org/doc/contribute.html" it to<EMAIL_ADDRESS>when it's
ready.


LoRA support

Are there any plans to support LoRA?
If so, I assume the .safetensors file will need to be converted with the model?

apple team, please add it!!

This repo has some models converted to Core ML after a LoRA was merged into a base model. Not the real thing, but a good bit of it . . .
https://huggingface.co/jrrjrr/LoRA-Merged-CoreML-Models

We need Lora

I have a feeling that SD-XL is capturing everyone's attention right now.  LoRA probably won't happen now until SD-XL is all figured out, but that seems to be happening quickly.  Hopefully that is out of the way before Sonoma and full ml-s-d 1.0.0 grab the spotlight and LoRA gets bumped again.

Hey @jrittvo thanks so much for the link!
Didn't know we could do that with LoRAs, gonna test a couple of your models now

The conversion from .safetensors (or .ckpt) to CoreML is pretty straightforward, once you get the environment to do it all set up.  Getting it set up is not that straightforward, unfortunately.  There is a good guide here:
https://github.com/godly-devotion/MochiDiffusion/wiki/How-to-convert-Stable-Diffusion-models-to-Core-ML
If you give it a shot and get stuck, someone at the Mochi Diffusion Discord can help you:
https://discord.gg/x2kartzxGv
You can also drop a specific request (or requests) at my LoRA-Merged-CoreML-Models repo and I'll run it (or them) for you, usually within a day or two.

Hello everyone! I just added the option to merge LoRAs before conversion on Guernika Model Converter, basically it takes the LoRAs and merges them using this script by Kohya.

Is there any progress now? The XL model size has increased, and the demand for Lora has become more urgent. 🥶

Hi.
I am trying to convert LCM-LoRA applied model, but failing.
Could someone advice me?
What I did are,

added the code below in get_pipeline function in torch2coreml.py. (You also need adding import LCMScheduler)

    pipe.load_lora_weights("latent-consistency/lcm-lora-sdv1-5", adapter_name="lcm")
    pipe.fuse_lora()
    pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)

converted a  model by the script below.

mkdir -p mlmodels
pipenv run python -m python_coreml_stable_diffusion.torch2coreml \
    --model-version runwayml/stable-diffusion-v1-5 \
    --attention-implementation ORIGINAL \
    --convert-unet \
    --convert-text-encoder \
    --convert-vae-decoder \
    --convert-vae-encoder \
    --convert-safety-checker \
    --quantize-nbits 6 \
    --bundle-resources-for-swift-cli \
    -o mlmodels \

adding codes for LCMScheduler options in pipeline.py.

generate an image by the script below

#!/bin/zsh

prompt="rabbit on moon, high resolution"

pipenv run python -m python_coreml_stable_diffusion.pipeline \
    --model-version runwayml/stable-diffusion-v1-5 \
    --scheduler LCM \
    --prompt "${prompt}" \
    -i mlmodels \
    -o . \
    --compute-unit ALL \
    --seed 42 \
    --num-inference-steps 8

The image which I got is something strange.

The image which I expect, which is actually generated on Diffusers at the same condition,  is below.

I wonder that I am missing something, but I have no idea since I am a newbie of Generative AI.
Give me advices, please!
Thanks.

So, what's the progress in these field? Is there any approach to make Lora models "hot-swapable" rather then pre-baking it?

Has anyone figured out how to use the newly introduced Multifunction Models? Apple even demonstrated it with a fine-tuned version of SDXL at WWDC 2024.
Any guidance or support would be greatly appreciated!


How can I save the last watch time of the video?

I have created the video player using chewie controller.  I have a video list screen.  Touching any item opens the video player page.
I need to get the time where the user stopped the video watching either he had quit the app or came back or paused. How can I achieve this?

you can get user's current timer where user has stopped/left playing by chewieController?.videoPlayerController.value.position;. Before disposing the video controllers save this duration value somewhere locally and when user plays same video again use startAt: inside chewieControlleer initializer.

Thank you.  If the application crashes or is closed by the system while the user is watching the video, the last value.  How can I save  position value?

In 99.99% of the cases it wont crash unless you do some serious mistake but if you are still scared then you can run a timer inside initState and change/store the current progress value. If something wrong happens you will have last stored progress value which you an use next time video is played again.

Thank you very much for your help.


fix: OPTIC-1013: Adding fast_first_or_create util function

PR fulfills these requirements

[ ] Commit message(s) and PR title follows the format [fix|feat|ci|chore|doc]: TICKET-ID: Short description of change made ex. fix: DEV-XXXX: Removed inconsistent code usage causing intermittent errors
[ ] Tests for the changes have been added/updated (for bug fixes/features)
[ ] Docs have been added/updated (for bug fixes/features)
[ ] Best efforts were made to ensure docs/code are concise and coherent (checked for spelling/grammatical errors, commented out code, debug logs etc.)
[ ] Self-reviewed and ran all changes on a local instance (for bug fixes/features)

Change has impacts in these area(s)
(check all that apply)

[ ] Product design
[ ] Backend (Database)
[ ] Backend (API)
[ ] Frontend

Describe the reason for change
(link to issue, supportive screenshots etc.)
What does this fix?
(if this is a bug fix)
What is the new behavior?
(if this is a breaking or feature change)
What is the current behavior?
(if this is a breaking or feature change)
What libraries were added/updated?
(list all with version changes)
Does this change affect performance?
(if so describe the impacts positive or negative)
Does this change affect security?
(if so describe the impacts positive or negative)
What alternative approaches were there?
(briefly list any if applicable)
What feature flags were used to cover this change?
(briefly list any if applicable)
Does this PR introduce a breaking change?
(check only one)

[ ] Yes, and covered entirely by feature flag(s)
[ ] Yes, and covered partially by feature flag(s)
[ ] No
[ ] Not sure (briefly explain the situation below)

What level of testing was included in the change?
(check all that apply)

[ ] e2e
[ ] integration
[ ] unit

Which logical domain(s) does this change affect?
(for bug fixes/features, be as precise as possible. ex. Authentication, Annotation History, Review Stream etc.)

Shoutout to @jombooth for the fast_first_or_create idea!


Calling root_validator on pydantic body input yields dictionary type

First check

[ x ] I added a very descriptive title to this issue.
[ x ] I used the GitHub search to find a similar issue and didn't find it.
[ x ] I searched the FastAPI documentation, with the integrated search.
[ x ] I already searched in Google "How to X in FastAPI" and didn't find any information.
[ x ] I already read and followed all the tutorial in the docs and didn't find an answer.
[ x ] I already checked if it is not related to FastAPI but to Pydantic.
[ x ] I already checked if it is not related to FastAPI but to Swagger UI.
[ x ] I already checked if it is not related to FastAPI but to ReDoc.
[ x ] After submitting this, I commit to one of:

Read open issues with questions until I find 2 issues where I can help someone and add a comment to help there.
I already hit the "watch" button in this repository to receive notifications and I commit to help at least 2 people that ask questions in the future.
Implement a Pull Request for a confirmed bug.

Example
Here's a self-contained, minimal, reproducible, example with my use case:
from fastapi import FastAPI
from pydantic import BaseModel, root_validator
from typing import Optional, Dict, Any

app = FastAPI()

class UserUpdate(BaseModel):
    password: Optional[str]
    name: Optional[str]

    @root_validator
    def validate(cls, values: Dict[str, Any]) -> Dict[str, Any]:
        return values

@app.patch("/me", response_model=schemas.User)
def update_user_me(
    user_in: UserUpdate,
) -> Any:
    assert isinstance(user_in, dict)
    return user_in

Description

Using root_validator in the pydantic model changes the behavior of the dependency injection, instead of return an instance of UserUpdate, a dict is returned
Commenting out the root_validator in the pydantic model leads to the expected behavior, in this case user_in is an instance of UserUpdate
I would expect user_in to be of type UserUpdate

Environment

OS: [e.g. Linux / Windows / macOS]: Linux

FastAPI Version [e.g. 0.3.0]: 0.61.1

Python version: 3.8.5

You should rename your "def validate" to something else (like "def _validate"). "validate" is an existing method for pydantic's BaseModel, so you are overriding it with yours, causing that unexpected behaviour. I tried it and after renaming the root_validator function name, it works as expected.

Thanks for the help @David-Lor! ☕
And thanks for closing the issue @moldhouse 🍰

Sorry for the long delay! 🙈  I wanted to personally address each issue/PR and they piled up through time, but now I'm checking each one in order.


Dedicated metrics examples

Adding some examples of collecting metrics for authzed dedicated

@kbrwn I took your commit from #17 and fixed up the linter errors here. Thanks for the contribution!


feat: Minimal circleci

Trying to alleviate spot instance issues
Later we can remove the commented out code, but thought it was good to have for reference for now in case people end up redoing any of the parts

Merging through aztec-packages


Upgrade Unidecode, add text-unidecode as extra option

Allow install of text-unidecode as extra. "pip install python-slugify[text-unidecode]"
Upgrade Unidecode

Coverage increased (+2.0%) to 95.05% when pulling 7addc366c4187d558bce3c719333936a6974fcc1 on sandbox into 8aea5c49b960b66e5c81bf20f17ec23e41c8d157 on development.


Flutter test not working trying to run the application entrypoint, instead of those of test fixtures

I followed instructions from https://codelabs.developers.google.com/codelabs/flutter-app-testing#0 to introduce unit tests into my application. In short, the test always fails due to a generic Connection closed before test suite loaded. error.
I noticed that running the tests was failing on unhandled exceptions due to all sorts of (self-written) states/providers that were trying to initialize. Providers that application (normally) relies on, but definitely should have been out of the scope of the unit test. The call stack gave me the first impression that it was initializing my application (instead of the test).
The build pipeline appears to inject a file called generated_main.dart. This file, seems to import an entry-point of the application package, and then runs that from its own main function. I get the impression that the main function in this file end's up being the actual entry point used by the executable.
//
// Generated file. Do not edit.
// This file is generated from template in file `flutter_tools/lib/src/flutter_plugins.dart`.
//

// @dart = 2.6

import 'package:liquid/main.dart' as entrypoint;
import 'dart:io'; // flutter_ignore: dart_io_import.
import 'package:path_provider_linux/path_provider_linux.dart';
import 'package:path_provider_windows/path_provider_windows.dart';

... //stripped for brievity

typedef _UnaryFunction = dynamic Function(List<String> args);
typedef _NullaryFunction = dynamic Function();

void main(List<String> args) {
  if (entrypoint.main is _UnaryFunction) {
    (entrypoint.main as _UnaryFunction)(args);
  } else {
    (entrypoint.main as _NullaryFunction)();
  }
}

For me, line 8 of this file, always contains import 'package:liquid/main.dart' as entrypoint;.  This is the entry point of the application, and not the unit-test. I asserted this idea with a simple validation, if I compile unit tests while I uncomment the main function in main.dart. it gave me a compile error.
I couldn't verify if anything that I observed above was done intentionally by design, it just seems very unlikely. According to https://github.com/flutter/flutter/issues/8516 flutter test should have been supported under windows.

flutter doctor -v
[√] Flutter (Channel dev, 2.5.0-6.0.pre, on Microsoft Windows [Version 10.0.19043.1165], locale en-US)
    • Flutter version 2.5.0-6.0.pre at E:\SDK\flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 225a43d941 (2 weeks ago), 2021-08-05 11:32:48 -0700
    • Engine revision 07ec4b82c7
    • Dart version 2.14.0 (build 2.14.0-383.0.dev)

[√] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
    • Android SDK at E:\SDK\Android\SDK
    • Platform android-30, build-tools 30.0.3
    • Java binary at: E:\SDK\Android\Android Studio\jre\bin\java
    • Java version OpenJDK Runtime Environment (build 11.0.8+10-b944.6842174)
    • All Android licenses accepted.

[√] Chrome - develop for the web
    • Chrome at C:\Program Files (x86)\Google\Chrome\Application\chrome.exe

[√] Visual Studio - develop for Windows (Visual Studio Community 2019 16.9.4)
    • Visual Studio at C:\Program Files (x86)\Microsoft Visual Studio\2019\Community
    • Visual Studio Community 2019 version 16.9.31205.134
    • Windows 10 SDK version 10.0.19041.0

[√] Android Studio (version 4.2)
    • Android Studio at E:\SDK\Android\Android Studio
    • Flutter plugin can be installed from:
       https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
       https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.8+10-b944.6842174)

[√] VS Code (version 1.59.1)
    • VS Code at C:\Users\phr34k\AppData\Local\Programs\Microsoft VS Code
    • Flutter extension version 3.25.0

[√] Connected device (3 available)
    • Windows (desktop) • windows • windows-x64    • Microsoft Windows [Version 10.0.19043.1165]
    • Chrome (web)      • chrome  • web-javascript • Google Chrome 92.0.4515.131
    • Edge (web)        • edge    • web-javascript • Microsoft Edge 92.0.902.73

• No issues found!

Yes, this is a bug that wasn't noticed until recently; see also https://github.com/flutter/flutter/pull/87991#issuecomment-900412165
In the short term you can work around it by deleting that generated main before running flutter test.
/cc @blasten

This should be fixed by https://github.com/flutter/flutter/pull/90288


pkg.md, rejected_casks.md: allow_untrusted

https://github.com/caskroom/homebrew-cask/issues/45146

The main docs also need to be updated, to explain the option exists for users who want to do in their own taps, but to not be used in the main repos.

Updated the pkg docs.

Added to audit so it doesn't sneak back in. https://github.com/Homebrew/brew/pull/3974


Add Universal Intercom Keys repository

https://github.com/GlUTEN-BASH/Flipper-Starnew

Added in update 70.

Reopening so I can mark this as merged.


Fix: Maestro tests for Favorites

Task/Issue URL:
Description
Steps to test this PR
Feature 1

[ ]
[ ]

UI changes

Before
After

!(Upload before screenshot)
(Upload after screenshot)

Current dependencies on/for this PR:

develop

PR #2975   👈

This comment was auto-generated by Graphite.


chore(release): REVISED release notes for 2.28.1

My original PR got mangled a bit as some 2.28.0 stuff got pulled in -creating this one to replace it.

Resolves GitHub issue:

Resolves Jira: [JIRAPROJECT-123]

Still have one item to add, so not yet ready to merge.

Not going to end up using this one.


[MINOR][SQL] Remove regression-test-SPARK-8489/test-2.11.jar

What changes were proposed in this pull request?
This pr just remove a useless jar: sql/hive/src/test/resources/regression-test-SPARK-8489/test-2.11.jar
Why are the changes needed?
sql/hive/src/test/resources/regression-test-SPARK-8489/test-2.11.jar is used to assist with the test case SPARK-8489: MissingRequirementError during reflection, but as Spark 3.0 no longer supports Scala 2.11, it is no longer needed now.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Pass GitHub Actions
Was this patch authored or co-authored using generative AI tooling?
No

Merged to master.

Thanks @dongjoon-hyun @HyukjinKwon


In Video full screen mode user is able to record the screen

In ios when user start the recording and if the user play a video in full screen mode then in the recording video is showing. rather than showing the black background.
i am using
if (Platform.OS == 'ios') {
  enableSecureView(); //This function blocks the Screen share/Recording and taking screenshot for iOS devices.
}

in the App.js file

can you elaborate more or copy/paste the component code here, how it is implemented?

closing because of no response


Add type resolver to product strawberry types

Probably need to pin mypy to 1.9, 1.10 was released yesterday


[$500] Add payment method popover has wrong position

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Action Performed:

On A, request money from B
On B, go to IOU report
On report header, click pay button and then click arrow button
Observe popover positions

Expected Result:
Popover should show below button
Actual Result:
Popover overlaps with button
Workaround:
Can the user still use Expensify without this being fixed? Have you informed them of the workaround?
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android / native
[ ] Android / Chrome
[ ] iOS / native
[ ] iOS / Safari
[x] MacOS / Chrome / Safari
[ ] MacOS / Desktop

Version Number: 1.3.57-2
Reproducible in staging?: y
Reproducible in production?: y
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos: Any additional supporting documentation
https://github.com/Expensify/App/assets/43996225/acf45e6a-e4e6-4173-8157-66d9a23c0d3d
https://github.com/Expensify/App/assets/43996225/27685809-7172-493c-a8e9-49f52651a633
Expensify/Expensify Issue URL:
Issue reported by: @aimane-chnaif
Slack conversation: https://expensify.slack.com/archives/C049HHMV9SM/p1692363288571469
View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~01a94242021f694c9e
Upwork Job ID:<PHONE_NUMBER>553955328
Last Price Increase: 2023-08-31

Proposal
Please state again the problem we are trying to solve in this issue.
Add payment method popover has wrong position.
What is the cause of this issue?
We does not pass anchorAlignment to PopoverMenu in AddPaymentMethodMenu.
And after redesign money request report, calculating anchor position in getAnchorPosition does not updated.
What changes do you think we should make to solve the problem?
First, we should pass anchorAlignment to KYCWall, AddPaymentMethodMenu and PopoverMenu.

In SettlementButton.js

	<KYCWall
    ...
+       anchorAlignment={this.props.anchorAlignment}
	>

In BaseKYCWall.js

	<AddPaymentMethodMenu
    ...
+       anchorAlignment={this.props.anchorAlignment}
	>
* In `AddPaymentMethodMenu.js`
```js
	<PopoverMenu
    ...
+       anchorAlignment={this.props.anchorAlignment}
	>
Second, we should update anchor position calculating in `getAnchorPosition` function.
We can modify calculating like drop down menu as below.
```js
    getAnchorPosition(domRect) {
        return {
            anchorPositionVertical: this.props.anchorAlignment.vertical === CONST.MODAL.ANCHOR_ORIGIN_VERTICAL.TOP
                ? domRect.top + domRect.height + CONST.MODAL.POPOVER_MENU_PADDING
                : domRect.top - CONST.MODAL.POPOVER_MENU_PADDING,
            anchorPositionHorizontal: domRect.left + domRect.width,
        };
    }

What alternative solutions have you investigated? (Optional)
None.

I'm catching up from being OOO, I will test soon.

@aimane-chnaif - let's solve this bug! I've assigned the issue to you.

@aimane-chnaif - When you get a chance, can you look at the proposals to identify if one will fix this issue? Thanks!

@aimane-chnaif - any update here?

Proposal
Please re-state the problem that we are trying to solve in this issue.
Popover for the transfer balance opens on incorrect place
What is the root cause of that problem?
Default vertical anchorAlignment is bottom which means it won't open below the passed anchor. And we are not passing anchorAlignment values from BaseKYCWall component.
https://github.com/Expensify/App/blob/57b76067cd3d7b59e84c4a387d11ad33e6a97d05/src/components/KYCWall/BaseKYCWall.js#L137-L145
What changes do you think we should make in order to solve the problem?

Pass anchorAlignment in AddPaymentMethodMenu by setting the vertical field to TOP and also add props in AddPaymentMethodMenu to receive which will eventually pass it to the PopoverMenu.

 anchorAlignment={{
      horizontal: CONST.MODAL.ANCHOR_ORIGIN_HORIZONTAL.LEFT,
      vertical: CONST.MODAL.ANCHOR_ORIGIN_VERTICAL.TOP, // we assume that popover menu opens below the anchor, anchor is at TOP
 }}

https://github.com/Expensify/App/blob/57b76067cd3d7b59e84c4a387d11ad33e6a97d05/src/components/KYCWall/BaseKYCWall.js#L137-L145

Now we need to correct the anchorPosition for the anchorPositionVertical and that needs to be the bottom of the anchor ref + 6px for some space between menuitem and popver.
https://github.com/Expensify/App/blob/57b76067cd3d7b59e84c4a387d11ad33e6a97d05/src/components/KYCWall/BaseKYCWall.js#L67

anchorPositionVertical: domRect.bottom + 6,

Inside the MenuItem we have different elements and on click of those it won't open the popover below the MenuItem instead it opens below the clicked target(for eg. menuItem text, icons). To fix we need to pass the MenuItem ref for clickedElementLocation instead of clicked target ref.

const clickedElementLocation = getClickedTargetLocation(this.anchorRef.current);

https://github.com/Expensify/App/blob/57b76067cd3d7b59e84c4a387d11ad33e6a97d05/src/components/KYCWall/BaseKYCWall.js#L113
https://github.com/Expensify/App/assets/14358475/24d3c444-14d2-41ce-b3b3-1921c1f9f126

We need to make a dynamic prop called popoverPlacement inside SettlementButton and expect it from the parent as the same component we are using to show the popover either top(ReportPreview) or bottom(transfer balance and MoneyReportHeader).

@aimane-chnaif have you been able to review the recent proposals? Thanks!

@aimane-chnaif - any feedback here from your research?

@aimane-chnaif I have missed popoverPlacement.

@aimane-chnaif - what's the next step here? It looks like @StevenKKC replied to your question.

@aimane-chnaif - we need your feedback to continue forward. Thanks!

@aimane-chnaif - any update here?

Awesome, looks like the PR is ready for review.

I will work on the payment process today

Here is the payment summary:

External issue reporter @aimane-chnaif $50
Contributor that fixed the issue @Nikhil-Vats $1000
Contributor+ that helped on the issue and/or PR @aimane-chnaif $1000

Upwork Job: https://www.upwork.com/jobs/~01a94242021f694c9e
*If applicable, the bonuses will be applied on the final payment
Extra Notes regarding payment: https://github.com/Expensify/App/issues/26070#issuecomment-1749480156 there is an urgency bonus and no regression. I'm going to factor these into the final payments.

Alright, @aimane-chnaif you've been paid via Upwork for reporting and C+. Please complete the checklist, thanks! I want to close this GH asap.


Icon binding not working when retrieved from database

Icon binding not working when retrieved from database.
I have tried several approaches where I:

stored the icon names as strings in db.
stored their values as int from the Enum.
copied the entire Enum to the domain layer, stored them as Enum.
I also saved them as strings and wrote an extension method to convert them at runtime.
None of these methods have worked for me, what am I missing?

The IconSource property is the name of the icon stored in db. as string:
modelBuilder.Entity<Category>().HasData(
     Category.Create(
         name: "Food & Drinks",
         nature: CategoryNature.Need,
         **iconSource: "Food24",**
         color: "#ff3d00",
         isVisible: true,
         order: 1
         ),

Here is how i use it in XAML:
<dx:DXImage x:Name="ImageIcon"
            Source="{mi:Fluent BindingContext={x:Reference model:CreateTransactionVM}, Icon={Binding IconSource}}"
            HeightRequest="20" />

Icon binding not working when retrieved from database. I have tried several approaches where I:

stored the icon names as strings in db.
stored their values as int from the Enum.
copied the entire Enum to the domain layer, stored them as Enum.
I also saved them as strings and wrote an extension method to convert them at runtime.
None of these methods have worked for me, what am I missing?

The IconSource property is the name of the icon stored in db. as string:
modelBuilder.Entity<Category>().HasData(
     Category.Create(
         name: "Food & Drinks",
         nature: CategoryNature.Need,
         **iconSource: "Food24",**
         color: "#ff3d00",
         isVisible: true,
         order: 1
         ),

Here is how i use it in XAML:
<dx:DXImage x:Name="ImageIcon"
            Source="{mi:Fluent BindingContext={x:Reference model:CreateTransactionVM}, Icon={Binding IconSource}}"
            HeightRequest="20" />

Please Try v4 that is Released Few Minutes Ago, Let Us know whether it fixes your issue?
We have Reworked the Whole Data Binding in v4, Most Probably it would fix your issue


Fix ignore messages

Fixes #3600 part 2

This needs specs!

I like this change! Being able to disable all messages is useful. However, I'm not sure this actually fixes the problem with ignoring a single gem. When I ignored just one gem, it caused empty install messages to be printed for every other gem in the bundle. That's what I was talking about in #3600.

It does because before has_post_install_message evaluated to true when it was empty? which is obviously the opposite which meant that the ones that did print were empty.

Is there any way to rerun the tests without having to commit a trivial commit as its rubygems' error.

Unfortunately, RubyGems master is actually broken right now. :(


player.play() doesn't correctly handle sound files with non-default format

I'm trying to use the new Sounds ext, and I have noticed that playing mp3 files with only one channel produces weird result - I'd say the file is played at 2x speed, or the pitch is shifted up. Converting the file to stereo fixes the issue.
MediaInfo output for mono file:
General
Complete name                            : Minion General Speech<EMAIL_ADDRESS>Format                                   : MPEG Audio
File size                                : 25.3 KiB
Duration                                 : 1 s 619 ms
Overall bit rate mode                    : Constant
Overall bit rate                         : 128 kb/s
Genre                                    : Other

Audio
Format                                   : MPEG Audio
Format version                           : Version 1
Format profile                           : Layer 3
Duration                                 : 1 s 620 ms
Bit rate mode                            : Constant
Bit rate                                 : 128 kb/s
Channel(s)                               : 1 channel
Sampling rate                            : 44.1 kHz
Frame rate                               : 38.281 FPS (1152 SPF)
Compression mode                         : Lossy
Stream size                              : 25.3 KiB (100%)

MediaInfo output for stereo file:
General
Complete name                            : Minion General Speech<EMAIL_ADDRESS>Format                                   : MPEG Audio
File size                                : 26.0 KiB
Duration                                 : 1 s 645 ms
Overall bit rate mode                    : Constant
Overall bit rate                         : 128 kb/s
Genre                                    : Other
Writing library                          : LAME3.100

Audio
Format                                   : MPEG Audio
Format version                           : Version 1
Format profile                           : Layer 3
Format settings                          : Joint stereo / MS Stereo
Duration                                 : 1 s 646 ms
Bit rate mode                            : Constant
Bit rate                                 : 128 kb/s
Channel(s)                               : 2 channels
Sampling rate                            : 44.1 kHz
Frame rate                               : 38.281 FPS (1152 SPF)
Compression mode                         : Lossy
Stream size                              : 25.7 KiB (99%)
Writing library                          : LAME3.100

I don't have any 5.1 or 7.1 files, but I would guess they will also be handled incorrectly, maybe the pitch will shift down?

So, a small update: player.play() expects the sound file to be in one specific format: 44.1 kHz, 128 kb/s, 2 channels. If any of the parameters are off, the sound will not be played correctly.

can confirm - my mono .mp3s also play super fast

until I changed the lines as suggested above:
            self.proc = subprocess.Popen(
                [
                    ffmpeg_bin,
                    "-i",
                    source,
                    "-loglevel",
                    "panic",
                    "-vn",
                    "-f",
                    "s16le",
                    "-ab",
                    "128",
                    "-ar",
                    "44100",
                    "-ac",
                    "2",
                    "pipe:1",
                ],

This is where the sample rate is hard coded. It just appears like this just isnt finished, and local files havnt been fully covered yet. Idk if theres a better way, maybe try to read in sample rate from meta data on audio, but I made a PR to at least allow you to set the values of those things that way we dont have to change the library code anymore lol

The actual rate right now for the audio files must be converted to 48000 and not 44100. I don't think this is mentioned in the docs anywhere and is not trivial find this solution.
Is there a problem to merge the PR by @sockheadrps?

Is there a problem to merge the PR by @sockheadrps?

I forgot it existed :)
There's a couple meta issues with the pr, but once they've been fixed it can be merged

I think this issue can be closed since #454 addressed the problem in this issue by detecting sample rate and channels from the audio meta data, and exposing both properties with setters just in case the meta data isn’t accurate.

Thanks for looking into it. With this update however, I'm getting weird results - if I play two sounds in a row, the second one isn't played, and holds the file open for eternity. I will try to post a reproducer later.

And here is the reproducer:
import asyncio
import os
import time

import eyed3

from pathlib import Path

from twitchio.ext import commands, sounds

class Bot(commands.Bot):
    def play_sound(self, sound: str):
        soundfile = str(Path(__file__).parent / sound)
        sound = sounds.Sound(soundfile)

        print("play sound", soundfile)

        self.player.play(sound)

        duration = eyed3.load(soundfile).info.time_secs
        time.sleep(duration)
        print(f"slept for {duration}s")

    def __init__(self, initial_channels=None):
        super().__init__(
            token=os.getenv("TWITCH_CHAT_PASSWORD"),
            client_id=os.getenv("TWITCH_CHAT_CLIENT_ID"),
            nick="arachnobot",
            prefix="!",
            initial_channels=["#iarspider"],
        )

        self.player = sounds.AudioPlayer(callback=self.player_done)

    async def event_ready(self):
        print(f"Ready | {self.nick}")
        self.play_sound("ding-sound-effect_1.mp3")
        self.play_sound("ding-sound-effect_1.mp3")

    async def player_done(self):
        print("Player done")
        pass

async def main():
    global client, twitch_bot
    twitch_bot = Bot()
    await twitch_bot.start()

if __name__ == "__main__":
    asyncio.run(main())

(only extra dependency is eyed3 to get sound file duration, can replace with fixed-duration). File used: ding-sound-effect_1.mp3
This outputs:
Ready | arachnobot
play sound e:\Temp\4\ding-sound-effect_1.mp3
slept for 2.95s
play sound e:\Temp\4\ding-sound-effect_1.mp3
slept for 2.95s
Player done

and plays ding only once.

This worked fine before. The Bot.play_sound function in my "production" code is used also to play temporary Text-to-Speech mp3 files, and the waiting is mostly used to remove those temporary files once they are done playing.

This one works properly:
import asyncio
import os
import time

from pathlib import Path

# from dotenv import load_dotenv
from twitchio.ext import commands, sounds

class Bot(commands.Bot):
    async def play_sound(self, sound: str):
        soundfile = str(Path(__file__).parent / sound)
        sound = sounds.Sound(soundfile)

        print("wait for lock")
        await self.lock.acquire()
        print("play sound", soundfile)
        self.player.play(sound)

    def __init__(self, initial_channels=None):
        super().__init__(
            token=os.getenv("TWITCH_CHAT_PASSWORD"),
            client_id=os.getenv("TWITCH_CHAT_CLIENT_ID"),
            nick="arachnobot",
            prefix="!",
            initial_channels=["#iarspider"],
        )

        self.lock = asyncio.Lock()
        self.player = sounds.AudioPlayer(callback=self.player_done)

    async def event_ready(self):
        print(f"Ready | {self.nick}")
        await self.play_sound("ding-sound-effect_1.mp3")
        await self.play_sound("ding-sound-effect_1.mp3")
        await self.play_sound("voxworker-voice-file.mp3")

    async def player_done(self):
        print("player done")
        self.lock.release()
        pass

async def main():
    global client, twitch_bot
    twitch_bot = Bot()
    await twitch_bot.start()

if __name__ == "__main__":
    # load_dotenv()
    asyncio.run(main())

Thanks for looking into it. With this update however, I'm getting weird results - if I play two sounds in a row, the second one isn't played, and holds the file open for eternity. I will try to post a reproducer later.

Here is the commit for the work I added, it's pretty minimal.
https://github.com/PythonistaGuild/TwitchIO/pull/454/commits/0c46ba733addd75367384de536356526f75cbab9
Are you sure this isnt an issue with other parts of your code? I dont think I see necessarily where anything I added could have caused anything like this to happen.
If you look at the output of your code
Ready | arachnobot
play sound e:\Temp\4\ding-sound-effect_1.mp3
slept for 2.95s
play sound e:\Temp\4\ding-sound-effect_1.mp3
slept for 2.95s
Player done

You're only getting the callback once, which seems a bit odd, but looking further, at the code you're experiencing a bug in, you're using time.sleep, which is a blocking function call for one, but also in the event_ready() function definition, you arent awaiting the async callback function you passed into the sounds.AudioPlayer object, so it looks like what's causing the issue is that youre playing the first audio file, and then immediately loading another sound file into it, causing the reference the first file made to AudioPlayer.play() to be overwritten, thus the AudioPlayer never invokes your first callback because you're overwriting the _is_playing property.
Thats my guess anyways, I didnt test your code, but Im pretty certain it has nothing to do with the merge of this PR, I think it has to do with how youre handling the AudioPlayer itself. You should be awaiting the callback function before you load and play any other sounds to avoid this kind of issue.
Hope that's helpful, sorry for the late response, I hadn't noticed you had mentioned anything in this issue.
@iarspider

@sockheadrps Sorry for not replying sooner as well. From a quick look at your code, looks you are handling things in a similar manner (just using global variable instead of lock). I like your implementation of sound queue, btw - nice work!


Why you don't use official video.js ads plugin?

Why you don't use official video.js ads plugin?

Hi @gangstaJS
I imagine that you are referring to the videojs-contrib-ads plugin from videojs. We really tried to use it and we did use the contribs ads plugin for quite some time. We decided to stop using it about 2 months ago. Below I am going to try to explain you the reasons why we decided to stop using it.
The documentation was not as good as we would have liked
They have some documentation and it gets better with every version but up to version 2.0 we had to read the code of the plugin from top to bottom to be able to fully understand what had changed on every migration.
We felt that we were sacrificing too much flexibility
The plugin decided when to start the ad, how to start it, and how to resume the content after the ad has played. It also limited the type of ads you could support.
In the end it is a black box that tells you when to play the ad and the resulting code is not very easy to read/understand. You need to fully understand the contribs ads plugin to be able to understand the code.
We had some issues on mobile devices where the preroll ad started after the content video had started playing (some times up to 10 seconds of content before the ad started playing). This was not the fault of the plugin itself. The problem was that our page is very heavy and the media evt were not always triggered when they were supposed to be triggered. While not considering it the plugin's fault we struggled to fix it because we could not decide when to start the ad as this was the responsibility of the contrib ads.
Migration of major versions where painful
We started using the plugin with version 0.x and experienced the migration from 0.x to 1.x and from 1.x to 2.x
Migrating from 0.x to 1.x
At version 0.x the contrib-ads plugin would trigger an event every time their state machine changed states. On 1.x it no longer triggered those events. They also changed the names of other events. This changes were documented on the migration document and we were able to do the migration. Although the missing events made the migration a bit painful.
Migrating from 1.x to 2.x
This release contained huge changes. It added support for postrolls ads. Changed all the events of the player adding a prefix 'ad' to all the event.
The postroll ads added new states to the contribs state machine and we were forced to do a hack in order to be able to replay videos.
The new events were a huge change because they not only added a prefix to all the player events. They also stopped the propagation of all the original player events. This was a huge problem for us because it meant that if the player have the ads plugin it would behave differently than if the player didn't have the ads plugin.
At some point 2 months ago we decided that we wanted to fully control the development cycle of our ads plugin.
I hope this helps you understand why we decided not use the videojs-contrib-ads plugin any more.


update base infra image to latest (2.2.0)

Note: caos team may also be updating with the new nrjmx version, so we might need to another small version bump in the base image

Note: caos team may also be updating with the new nrjmx version, so we might need to another small version bump in the base image


Widget test

Added 2 test for the widgets on the main screen
First one validates the error message is being displayed and the second one checks weather a result is being showed

@viveeeeeek I resolved conflicts and updated my test. Everything is working okay now 👍


feat: add logic to self-signed JWTs if alwaysUseJWTAccess is true

Work for the self-signed JWT implementation
In conjunction with: https://github.com/googleapis/gapic-generator-typescript/pull/908

Still need to add tests

@bcoe @JustinBeckwith, comments  have been addressed & tests added!

Waiting for: https://github.com/googleapis/gapic-generator-typescript/pull/908

Same feedback here as in the gapic generator: I'd advocate for all of these to be renamed from useJWTAccessAlways to useJwtAccessWithScope for consistency (see AIP #4111), unless this was a conscious decision with justification.

As https://github.com/googleapis/google-auth-library-nodejs/pull/1204 is already released and uses the name useJWTAccessAlways, it's ok to keep that variable name here, and this should not block rolling the feature out!


Volume entity not working anymore

Hi,
Noticed that the sensor.beosound_volume entity does not work correctly anymore with my Beosound Theatre. Not sure whether this is related to the new firmware (<IP_ADDRESS>) on the Beosound, Home Assistant 2024.3.3 or the Bang & Olufsen integration.
You can still set the volume using the slider but the sensor.beosound_volume entity does not change.
I have scripts to change the volume up/down that does also not work anymore because they depend on the current volume level to work.
Kind regards
/Jan Lebeck

Example of a script to change the volume down by 5:
service: media_player.volume_set
target:
entity_id: media_player.beosound_theatre_36199013_3
data:
volume_level: >-
{{ states.media_player.beosound_theatre_36199013_3.attributes.volume_level - 0.05 }}

Never changes...

Have checked on a completely fresh installation of Home Assistant also - same problem.

I'm guessing that you have tried rebooting the Theatre.
What version of this integration are you using? If it is 2.1.1, can you try downgrading to 2.0.0?
And are any other WebSocket notifications working?, for example does changing source change artwork and metadata as expected?
Kind regards

After downloading everything works... tried to upgrade to 2.1.1 again - then NOT working again. Going back to 2.0.0. Repeated this a few times to verify. 1.8.1 also works.

Ok. I have not been able to replicate this issue with WebSocket notifications on 2.1+, but you are the second one reporting problems. The issue seems to be the new WebSocket backend for the API client. I'll try to revert to the old backend and release a new version tomorrow.

Can you try updating to 2.1.3?

Hi ... just upgraded and tested... and it now works as usual on 2.1.3.

Good. I'll stay with the old WebSocket backend.


remove 6 byte packet numbers

6 byte packet numbers were only used in gQUIC.

Codecov Report

Merging #1613 into master will decrease coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #1613      +/-   ##
==========================================
- Coverage    87.5%   87.49%   -<.01%
==========================================
  Files          99       99
  Lines        6247     6245       -2
==========================================
- Hits         5466     5464       -2
  Misses        558      558
  Partials      223      223

Impacted Files
Coverage Δ

internal/protocol/protocol.go
100% <ø> (ø)
:arrow_up:

internal/protocol/packet_number.go
100% <100%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bd6567a...7696bf5. Read the comment docs.


cmd/compile: Bad code generation for ARM (bounds test fails).

What version of Go are you using (go version)?

$ go version
go version go1.15.2 darwin/amd64

and also

$ go version
go version go1.15 linux/arm64

and also

$ go version
go version devel +869c02ce1f Sat Oct 3 17:02:58 2020 +0000 linux/arm64

Does this issue reproduce with the latest release?
Yes.

Compile this program https://play.golang.org/p/TfpedZeakHY for GOARCH=arm GOOS=linux
GOOS=linux GOARCH=arm go run arm.go

Code also appears below:
package main

type decimal struct {
	d  [8]byte // digits, big-endian representation
	dp int     // decimal point
}

var powtab = []int{1, 3, 6, 9, 13, 16, 19, 23, 26}

//go:noinline
func foo(d *decimal) int {
	exp := int(d.d[1])
	if d.dp < 0 || d.dp == 0 && d.d[0] < '5' {
		var n int
		if -d.dp >= len(powtab) {
			n = 27
		} else {
			n = powtab[-d.dp]
		}
		exp += n
	}
	return exp
}

func main() {
	var d decimal
	d.d[0] = '1'
	println("foo(&d)=", foo(&d))
}

What did you expect to see?
The playground output: foo(&d)= 1
What did you see instead?
On an ARM:
GOARCH=arm go run arm.go
panic: runtime error: index out of range [0] with length 9

goroutine 1 [running]:
main.foo(0x202c7ac, 0x0)
	/home/drchase/work/tmp/arm.go:18 +0x78
main.main()
	/home/drchase/work/tmp/arm.go:28 +0x30
exit status 2

The bug is very delicate, it depends on something going wrong in arm lowering/optimization to machine code, and it is in that dyslexia-inducing bit where comparisons are reversed, inverted, etc.  The signature of the bug (seen in ssa.html) is that the comparison -d.dp >= len(powtab) gets compiled into a CMN instruction both at the comparison and at the bounds check; if this happens, something goes wrong.  The pattern firing depends on the SSA variable numbering of the inputs to the originally generated CMP.
From here, a complete dump of the debugging work, just collecting, not analyzing, that can happen later.

Ultimately, the code looks like this:

According to gdb, this is the code that is generated:
=> 0x6c170 <main.foo>:	ldr	r1, [r10, #8]
   0x6c174 <main.foo+4>:	cmp	sp, r1
   0x6c178 <main.foo+8>:	bls	0x6c1ec <main.foo+124>
   0x6c17c <main.foo+12>:	str	lr, [sp, #-12]!
   0x6c180 <main.foo+16>:	ldr	r2, [sp, #16]
   0x6c184 <main.foo+20>:	ldrb	r3, [r2, #1]
   0x6c188 <main.foo+24>:	ldr	r4, [r2, #8]
   0x6c18c <main.foo+28>:	cmp	r4, #0
   0x6c190 <main.foo+32>:	bge	0x6c1cc <main.foo+92>
   0x6c194 <main.foo+36>:	ldr	r11, [pc, #96]	; 0x6c1fc <main.foo+140>
   0x6c198 <main.foo+40>:	ldr	r1, [r11]
   0x6c19c <main.foo+44>:	ldr	r11, [pc, #92]	; 0x6c200 <main.foo+144>
   0x6c1a0 <main.foo+48>:	ldr	r2, [r11]
   0x6c1a4 <main.foo+52>:	cmn	r1, r4
   0x6c1a8 <main.foo+56>:	bgt	0x6c1bc <main.foo+76>
   0x6c1ac <main.foo+60>:	mov	r0, #27
   0x6c1b0 <main.foo+64>:	add	r0, r3, r0
   0x6c1b4 <main.foo+68>:	str	r0, [sp, #20]
   0x6c1b8 <main.foo+72>:	ldr	pc, [sp], #12
   0x6c1bc <main.foo+76>:	rsb	r0, r4, #0
   0x6c1c0 <main.foo+80>:	bls	0x6c1e4 <main.foo+116>
   0x6c1c4 <main.foo+84>:	ldr	r0, [r2, r0, lsl #2]
   0x6c1c8 <main.foo+88>:	b	0x6c1b0 <main.foo+64>
   0x6c1cc <main.foo+92>:	bne	0x6c1dc <main.foo+108>
   0x6c1d0 <main.foo+96>:	ldrb	r2, [r2]
   0x6c1d4 <main.foo+100>:	cmp	r2, #53	; 0x35
   0x6c1d8 <main.foo+104>:	bcc	0x6c194 <main.foo+36>
   0x6c1dc <main.foo+108>:	mov	r0, r3
   0x6c1e0 <main.foo+112>:	b	0x6c1b4 <main.foo+68>
   0x6c1e4 <main.foo+116>:	bl	0x69d50 <runtime.panicIndex>
   0x6c1e8 <main.foo+120>:	andeq	r0, r0, r0
   0x6c1ec <main.foo+124>:	mov	r3, lr
   0x6c1f0 <main.foo+128>:	bl	0x680fc <runtime.morestack_noctxt>
   0x6c1f4 <main.foo+132>:	b	0x6c170 <main.foo>
   0x6c1f8 <main.foo+136>:	b	0x6c1f8 <main.foo+136>
   0x6c1fc <main.foo+140>:	andeq	r1, sp, r4, asr r0
   0x6c200 <main.foo+144>:	andeq	r1, sp, r0, asr r0
   0x6c204 <main.main>:	ldr	r1, [r10, #8]

and this is the trace where it goes wrong:
2: /x $cpsr = 0x80000010
4: /x $r0 = 0x0
5: /x $r1 = 0x9
6: /x $r2 = 0xd0100
7: /x $r3 = 0x0
8: /x $r4 = 0x0
9: x/3i $pc
=> 0x6c1a4 <main.foo+52>:	cmn	r1, r4
   0x6c1a8 <main.foo+56>:	bgt	0x6c1bc <main.foo+76>
   0x6c1ac <main.foo+60>:	mov	r0, #27
(gdb)
0x0006c1a8	15	in /Users/drchase/work/tmp/arm.go
2: /x $cpsr = 0x10
4: /x $r0 = 0x0
5: /x $r1 = 0x9
6: /x $r2 = 0xd0100
7: /x $r3 = 0x0
8: /x $r4 = 0x0
9: x/3i $pc
=> 0x6c1a8 <main.foo+56>:	bgt	0x6c1bc <main.foo+76>
   0x6c1ac <main.foo+60>:	mov	r0, #27
   0x6c1b0 <main.foo+64>:	add	r0, r3, r0
(gdb)
0x0006c1bc	22	in /Users/drchase/work/tmp/arm.go
2: /x $cpsr = 0x10
4: /x $r0 = 0x0
5: /x $r1 = 0x9
6: /x $r2 = 0xd0100
7: /x $r3 = 0x0
8: /x $r4 = 0x0
9: x/3i $pc
=> 0x6c1bc <main.foo+76>:	rsb	r0, r4, #0
   0x6c1c0 <main.foo+80>:	bls	0x6c1e4 <main.foo+116>
   0x6c1c4 <main.foo+84>:	ldr	r0, [r2, r0, lsl #2]
(gdb)
18	in /Users/drchase/work/tmp/arm.go
2: /x $cpsr = 0x10
4: /x $r0 = 0x0
5: /x $r1 = 0x9
6: /x $r2 = 0xd0100
7: /x $r3 = 0x0
8: /x $r4 = 0x0
9: x/3i $pc
=> 0x6c1c0 <main.foo+80>:	bls	0x6c1e4 <main.foo+116>
   0x6c1c4 <main.foo+84>:	ldr	r0, [r2, r0, lsl #2]
   0x6c1c8 <main.foo+88>:	b	0x6c1b0 <main.foo+64>
(gdb)
0x0006c1e4	18	in /Users/drchase/work/tmp/arm.go
2: /x $cpsr = 0x10
4: /x $r0 = 0x0
5: /x $r1 = 0x9
6: /x $r2 = 0xd0100
7: /x $r3 = 0x0
8: /x $r4 = 0x0
9: x/3i $pc
=> 0x6c1e4 <main.foo+116>:	bl	0x69d50 <runtime.panicIndex>
   0x6c1e8 <main.foo+120>:	andeq	r0, r0, r0
   0x6c1ec <main.foo+124>:	mov	r3, lr

I wonder if the (CMP x -y) => (CMN x y) is valid when the condition codes are used for unsigned comparison.
Nothing else in the rewrites looks the least bit suspicious.

Confirmed, the CMP to CMN transformation is not carry-bit-preserving.  The failing rewrite:
b11: (UGT (CMP x (RSBconst [0] y))) else --> panic
          (CMP x (RSBconst [0] y)) => (CMN x y)
b11: (UGT (CMN x y)) else --> panic

evaluated:
BEFORE:
b11: (UGT (CMP "9" (RSBconst [0] "0"))) else --> panic
b11: (UGT (CMP "9" "0")) else --> panic
UGT = if C&~Z then, else
C = "9" - "0" does not borrow (i.e. "0" U< "9") = true
Z = false (9 != 0)
UGT = true and not false = true ==> "then"

AFTER:
b11: (UGT (CMN "9" "0")) else --> panic
UGT = if C&~Z then, else
C = "9" + "0" unsigned-overflows (i.e. "0" + "9" exceeds max unsigned) = false
Z = false (9 != 0)
UGT = false and not false = false ==> "else" == panic

In context, the full chain of rewrites for both comparisons (that are the same comparison):
b7:  (If (Leq32 x (Neg32 y))) else --> b11;  b11: (If (IsInBounds  (Neg32 y) x)) else --> panic

  (Leq32 x y) => (LessEqual (CMP x y))
  (Neg(32|16|8) x) => (RSBconst [0] x)
  (IsInBounds idx len) => (LessThanU (CMP idx len))

b7: (If (LessEqual (CMP x (RSBconst [0] y)))) else  --> b11;  b11: (If (LessThanU (CMP (RSBconst [0] y) x))) else --> panic

 (If (LessThanU cc) yes no) => (ULT cc yes no)
 (If (LessEqual cc) yes no) => (LE cc yes no)

b7: (LE (CMP x (RSBconst [0] y))) else  --> b11;  b11: (ULT (CMP (RSBconst [0] y) x)) else --> panic

  (CMP x y) && x.ID > y.ID => (InvertFlags (CMP y x))

b7: (LE (CMP x (RSBconst [0] y))) else  --> b11;  b11: (ULT (InvertFlags (CMP x (RSBconst [0] y)))) else --> panic

   (ULT (InvertFlags cmp) yes no) => (UGT cmp yes no)

b7: (LE (CMP x (RSBconst [0] y))) else  --> b11;  b11: (UGT (CMP x (RSBconst [0] y))) else --> panic

  (CMP x (RSBconst [0] y)) => (CMN x y)

b7: (LE (CMN x y)) else  --> b11;  b11: (UGT (CMN x y)) else --> panic

   Because of block ordering rules, these assemble to

b7: (BGT (CMN x y)) then  --> b11;  b11: (BLS (CMN x y)) then --> panic

  Translating from GO asm to ARM asm (no surprises)

b7: (bgt (cmn x y)) then  --> b11;  b11: (bls (cmn x y)) then --> panic

This sounds an awful lot like https://go-review.googlesource.com/c/go/+/233097
@shawn-xdji

Actually, that's the arm64 CL. The arm one was https://go-review.googlesource.com/c/go/+/236637/

@randall77 It's similar.  I suppose I should figure out if the CMP <-> CMN rewrites are valid for the no-ov comparisons.


doc: improve example docs/code/progress/main.c

From the example:
percentage = downloaded*100.0/size;
async.data = (void*) &percentage;
uv_async_send(&async);

Multiple problems:

uv_async_send() is not a memory barrier
percentage goes out of scope when the thread exits, which can happen before the async callback runs
An optimizing compiler may opt to do the calculation in place, meaning other threads may see a partial result

There's a sleep(1) right after so (2) is not super likely but it's still a bad example.

You want is an example of async use?  Or a different one from this one


changed version number to 1.1.0

Coverage remained the same at 100.0% when pulling 589ae73d1ce974fc3f2a5c519153a409d2af706a on update-version-number into 9d44b09c62f569492ffd7ecd11fe60df31807de6 on master.


GNILC dust templates with polarization fraction tensor

View and comment on the notebook at: https://app.reviewnb.com/healpy/pysm/pull/72/files/
The plan as suggested by Jacques Delabrouille is to add small scales fluctuations directly in needlet space, and
Mathieu Remazeilles is working on it.
However I think it is useful to have a simpler model to compare to, so the plan here is to fit for a high-ell slope to the GNILC maps and then add small scale fluctuations.
The difference with PySM 2 is that now I am using the log of the polarization fraction tensor as suggested by Andrei Frolov, it is based on polarization fraction, so we get the polarization fluctuations automatically scale with amplitude of the large scale emission:

Masking

Resolution of the GNILC maps

see the gnilc_resolution.ipynb notebook at https://app.reviewnb.com/healpy/pysm/pull/72/files/

Fitting using GNILC R3
In the notebook attached to this PR I am loading the R3 maps (also for intensity), building the log of the polarization fraction tensor and fitting a slope to that.
I select manually a range of ell [100-300] which is clean from large scale fluctuations and before emission starts to flatten due to noise (I believe).

And that is exactly the problem, what section of the spectrum do we want to use for fitting?

A few quick thoughts, but hopefully others can chime in:

The masking looks good. Things get complicated (and possibly qualitatively different) too near the Galactic plane, so best to mask it even though there is a lot of signal there.
Hopefully Andrei has thoughts re mixing resolutions. Definitely sounds non-ideal, but it would also be a shame to smooth the I maps.
Is the reason that the TT spectrum flattens at the same ell as EE and BB just that you are using the polarization tensor formalism? It looks like the PySM2 model used ells well below 100 for their fits (see Figure 5). Are the differences due to the maps used or the polarization tensor approach?

Thank you for putting this together!

A few quick thoughts, but hopefully others can chime in:

The masking looks good. Things get complicated (and possibly qualitatively different) too near the Galactic plane, so best to mask it even though there is a lot of signal there.

very good thanks

Hopefully Andrei has thoughts re mixing resolutions. Definitely sounds non-ideal, but it would also be a shame to smooth the I maps.

ok wrote him email about this

Is the reason that the TT spectrum flattens at the same ell as EE and BB just that you are using the polarization tensor formalism? It looks like the PySM2 model used ells well below 100 for their fits (see Figure 5). Are the differences due to the maps used or the polarization tensor approach?

I'm investigating, I'll update this answer with what I find.

Power spectrum of the GNILC R2 II map

Do we understand the features in the spectrum above ell of 1000? Or do we not use that part anyway so it doesn't matter?

I have a new notebook with lots of improvements, will post on Monday

ok,
I have fixed the issue using the polarization fraction tensor, now the fit looks better (now properly in log-log):

I changed the mask, I noticed that GNILC had already sources removed, so chose a large galactic mask (GAL060 from HFI PR2):

This was due to the fact that when I want to remove the small scales from the GNILC temperature map, then when I apply the mask and run NaMaster, I see residual power at high-ell, this is improved if I use a mask which has no point sources and I don't apply too steep a cut (I am using a sigmoid so I can tweak the steepness of the cut).
Even now I can see the effect (please send suggestions on how to improve here, I would like to apply a steeper cut):

However the residuals are finally small enough that the output spectra look fine (before they showed a step in the II output spectrum around ell = 1400 because the residual small scale signal from the large scale signal map was added to the actual simulated gaussian small scales:

The full notebook with the analysis is at:
https://nbviewer.jupyter.org/gist/zonca/d00d3d2e17d902e5aac6f80e4c9f20f4
If anyone has suggestions on how to reduce the leaking of power to high ell, I would use a sharper transition between the input map and the gaussian realization, otherwise I think this is acceptable and I can produce the final templates for inspection from other people.

This is great! Are the spectra shown here in real units or in the pol. fract. tensor ones?

all of them in pol frac tensor, I'll also add a summary plot in standard IQU at the end of the notebook

TE
During the CMB-S4 meeting there was a mention of PySM 3 not having TE in the galactic models.

does it make sense even if we have different resolutions and temperature/polarization maps from different releases?
also, in the polarization fraction formalism, TE is related to how the T emission is correlated with the polarization fraction. Is this fine?

I made a first attempt at the fit:

the problem is that when I try to add the small scales generated using this fit, the polarization maps have way too much power and are completely off:

I'll keep investigating but it would be useful to have a general opinion about this and an answer to the 2 questions above.

Hi Andrea, I've been taking another look at this model, and was wondering about something. In this model we care about the polarization fraction, and so it is important to get the zero level of the intensity maps right. The GNILC maps do not seem to remove the CIB monopole, and get the overall zero-level contribution from dust slightly wrong (according to section 2.2 of 1807.06212).
I've taken a look at the effect of removing at least the CIB monopole from the GNILC intensity map, and combining this with gaussian realizations of Frolov (q, u) to produce (Q, U) with power tuned to the level in the BK maps (4.7 uK at ell=80). Whether or not you include the monopole in the intensity map seems to have a large qualitative effect on the simulations for a given power level, at these higher latitudes.
The small scale realizations you're doing here are slightly different, and I'm not sure yet what effect this has, if any. Have you already considered this?

no, haven't considered this at all, can you please share the code you used to fix the monopole level? or modify my notebook to incorporate that?

current issues slides: https://us02st1.zoom.us/web_client/ehjzr5/html/externalLinkPage.html?ref=https://docs.google.com/presentation/d/19Tcg_jccTgSg8KtZgf5PT0R2MVNFdl5_PM5mJjnN0EM/edit?usp=sharing

@bthorne93 will help with the monopole, also Jacques can provide maps for comparison of zero level
for covariance, start just using the IQU covariance itself
once those 2 are fixed, evaluate again TE, also send to Jacques to see if he finds anything wrong in the computation

@bthorne93 just a reminder to send me details about fixing the monopole level.

Based on Section 2.2 of Planck 2018: XII, the contributions to the monopole in the 353 GHz GNILC intensity map are CIB and dust. They remove the CIB contribution, and apply some process to improve on the estimate of the dust monopole the maps already contained.
The numbers for each contribution can be found in Table 12 of Planck 2018: III, which I've pasted below. This seems to indicate there is a relevant (but very small) contribution from zodiacal emission as well, which is not discussed in XII.
The simplest first thing to do is just remove the CIB contribution, for which I've been using the number in Table 12: 0.13 MJy/sr at 353 GHz. Perhaps we also need to remove zodiacal emission, and perhaps we also need to refine the dust monopole model as was done in XII.

Resolution with GAL080 mask

0,27.70%
10,0.04%
15,0.44%
20,4.26%
30,16.09%
60,45.93%
80,5.54%

ok, I have made a few changes:

Using GAL080 instead of GAL060 so we have more sky coverage (same used in PySM 2)
Using 2 degree apodization masks as provided by Planck
Removed TE: I believe that the output maps will have TE correlations anyway because of transforming back from polarization fraction tensor to IQU
Fixed monopole as suggested by @bthorne93
Focusing on lower ell range for fitting both in T and P
Use cosmic variance in the fitting
Using a sharper sigmoid for the transition

Need to do:

T small scales are uniform across the sky, they should be multiplied by a normalized smoothed T map?
Understand why fitted polarization slopes are so flat

the full notebook executed is at:
https://nbviewer.jupyter.org/gist/zonca/b398f85f701088ba60e3e66a20e0f48f
Spectra fitting plots

Fitted slopes are very close to zero:
'TT': -1.001
'EE': -0.053
'BB': -0.012

Thanks @brandonshensley and @seclark for the chat today, some notes:

fitted slopes are very close to zero, it could be because we are using variable resolution, we could either try to use the unires version at 80 arcmin or directly switch to 353GHz where it is easier to control the beam. So we could use GNILC temperature which is good because it has less contamination from CIB. Those are all options to explore
the sigmoid transition is possibly too sharp and could cause a strange feature in the spectrum, better use a standard beam/inverse beam.
The Planck paper about dust has fitted slopes, which are for example around -0.5 for BB (they quote -2.5 but then add 2 in the modelling). So another option is to keep that slope and just fit for the amplitude (this is even better if we are using the 353 maps in polarization).
As noted above, the small scales for temperature are uniform, they should be instead scaled for total amplitude normalized properly.
It looks fine to skip doing inverse noise weighting, signal is not uniform, so this could create issues in the spectra
I had previously ran a simple simulation to check that if you take the log of a powerlaw the new map has the same slope in the spectrum: https://gist.github.com/dd2a68b424333f14d37b47cc6df75ebf

see #74


Refactoring

More improvement.

I am refactoring.
https://github.com/Code-Hex/pget/tree/feature/refactoring


[Feature] Create home page

Is your feature request related to a problem? Please describe.
The application needs a home page. In the future the application will have an exam mode.
Describe the solution you'd like
A new home page for the application with two buttons for selecting one of the modes.
Additional context
I created a figma design file here

@eduardconstantin I have a few questions:

Is this task only about design, or should some functions also be integrated into the app?
Are only Tailwind classes allowed or desirable to complete it?
Thank you in advance for your answer.

@eduardconstantin I have a few questions:

Is this task only about design, or should some functions also be integrated into the app?
Are only Tailwind classes allowed or desirable to complete it?
Thank you in advance for your answer.

It also needs to have some functionality on the buttons like when you press on practice mode it will redirect to the current page that is in the app. When pressing on exam mode it should redirect to an empty page for now.
Yes, the design should be made only using tailwind classes that are found here.
Would you like to work on this issue?

@eduardconstantin yes, I would like to

Great! Make sure to make a new branch from the dev branch


Add configuration for recording time

Allow a user to configure the recording time for top, allowing for aggregation of requests longer than 5 seconds.
Also added a check in the ServiceProvider to only register event listeners when a valid recording time is set. This provides the added bonus disable this package by setting the recording time to 0.

Thanks for contributing.
Disabling or tuning the recording time makes sense.
But since top is a CLI tool, in a future release, I'll manage this value through command arguments like artisan top -i 5 and add modes for the event listening strategy to the config.

Saw the new code addition that defaults to recording only when top is running.  That's a great addition and something I was thinking of while implementing this.  Let me know if you'd be interested in a PR adding the command line argument, I see where you've already started planning for setting the recording time and a lot of my code here can be used.


Implement HTMLHint and any remaining CI/CD testing (Cypress)

Task

Implement HTMLHint
Implement Cypress aspects in CI/CD

Deliverables

Additions to the pipeline code that integrates and adds HTML linting

Delivery: 12/6

Added the HTML linter and an HTML parser with standard rules. Tested with npm lint and received error output.

Changed to HTML hint.

Completed.


Test fail: ApiPlatformCustomRoutesWithParamConverterTest::testBookSpecialSample3WithErrors

I don't know if it's because of a version of PHP i'm using...
I have to find since when this test is failing
It should not coz this is the reviver library that manage this part normally
1) App\Tests\Normalization\ApiPlatformCustomRoutesWithParamConverterTest::testBookSpecialSample3WithErrors
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
-'book.editors[0].editor: jsonOrArray for editor must be string or array'
+'book.editors[0].editor: Only variables should be passed by reference'

fixed by an update of rebolon/ApiJsonParameter...


Fix: progress bar reset & remove proposals chart

Summary of changes

Both the progress bar on keys page and authorities page starts from start.

Proposed area of change

[ ] apps/bridge-dapp
[ ] apps/hubble-stats
[x] apps/stats-dapp
[ ] apps/webbsite
[ ] apps/faucet
[ ] apps/tangle-website
[ ] libs/webb-ui-components

Screen Recording
https://github.com/webb-tools/webb-dapp/assets/53374218/4d1bf6e2-c0ed-4572-88aa-3891188bdee3

Can we animate it @devpavan04 so it grows from 0 to its progress?

After spending quite some time in debugging this issue @AtelyPham and I have decided to pick this up in the next sprint.
Issue - https://github.com/webb-tools/webb-dapp/issues/1422


Missing Characters

Since the last update, my characters are no longer available, being replaced with the 2 default characters.

I'm sorry to hear that, it may be a bug. Do you know if your browser cache was cleared?

Backed up some, unfortunately they players didnt.
Interestingly, when I open it on my phone, it shows the character for a moment, then switches back to the demo characters.

That's unfortunate. I'm investigating the problem now. I can't seem to recreate the issue. Can you send me a copy of your backed up (JSON files) characters please? My email should be on my profile. Or attach them to a post here?

Just sent by email.
Yeah pretty strange bug. We can recreate them pretty easily, so aside from wealth its not that big an issue.

Thank you, received. I'm using it to try and recreate the issue.
I've also pushed out a bug fix to address an issue with the recent update to the Event (xp and wealth) logging. Can you refresh awesomeSheet (but don't clear your cache as your character data may still be saved there) and see if your characters still don't load?

If you're familiar with Chrome and it's Dev Tools, there is a Console command you can run to copy all character data in local storage to your clipboard: copy(sheet.getAllCharacters()).
(this is what that command does)

I'll try it in but a moment. Currently GMing as we speak.
Upon refreshing it only shows the default characters.

Understood, when you get time, GMing comes first.
Unfortunately, I've not yet been able to recreate the issue. I'll keep at it.

I think I got it. I recently moved the app over to HTTPS and from last week started redirecting user to the new url (https://zombiefox.github.io/awesomeSheet/). Up until then some users were using the insecure address (HTTP [no s]) and their character data is still there. I've released a fix to disable redirecting.
To recover your character data from the HTTP site follow these steps:

Enter this address into your browser address bar (take note that there is no s in the HTTP): http://zombiefox.github.io/awesomeSheet/
Hit enter and then hit ESC to stop the browser from redirecting to HTTPS.
If awesomeSheet is loaded and you're still on HTTP [no s] then hold shift and hit Refresh. This will update that instance of awesomeSheet with the fix to stop redirecting.

Your character data should be there. I'm sorry for this unforeseen bug. I'll look into this problem some more and will try to address it in the future.

This was exactly the issue. Just tested and it works.
Good catch!

Fantastic, good to hear!
I'll investigating and work on a better transition from HTTP to HTTPS.


Is it possible to include conda environment in pex file?

I'm running into a fairly irritating issue where I'm able to install a package via conda install, but cannot install or collect the wheels via pip install or pip wheel... I'm using an anaconda virtual env, and everything works locally. I'd like to just package the entire conda virtual env inside of the pex file, but can't seem to find a way to do this...
Is what I'm looking to do possible? Are there alternative solutions?

From what I can tell after a quick install of miniconda3 and then a conda create -n scipy scipy, pex is not really close to being able to handle this. The conda create and conda install commands appear to install custom native libraries. This would make sense, since the target audience consumes a bunch of python libraries that are hard to install / depend on native libs! Currently, pex only deals in python standards: wheels, eggs, sdists and the associated PEPs. It does not handle installing non-python dependencies or now about external tools that can do so.
I don't of any alternative solutions.
I think it would be wonderful if pex or an offshoot / similar tool could create multiplatform pex's that could handle activating pre-packaged out of band native libs. This would be a good deal of work though and, like most OSS projects, we're strictly volunteer at this point, so it seems in the near term a solution that fits your needs would have to come from a motivated contributor.

I'm going to close this as "answered", but I'm happy to exchange ideas in this thread if you have more comments.

+1 for this feature


 This means a user must go to the CFP website and create an account before they can be made an admin in the system.
Suggested Enhancement
CLI users should be able to promote an email address to admin even if it isn't known to the system yet.
There are two ways to add this to the script:

Update admin:promote to create an account if the supplied email isn't found.
Add user:create to allow users to be created on the command line, which could then be promoted with admin:promote.

I prefer option 2 because it has more utility. It's very possible that an admin could want to create a user on the command line that isn't an administrator (for example, during development). #1 is a shortcut for the scenario in this ticket, but it doesn't provide any flexibility.
Considerations
The newly created account will have to have a password. The CLI user could create a password right there, but the most secure (and slickest) way would be to create a one-time password that had to be changed on first login. That, of course, is more work, and I don't personally think it needs to be a blocker in implementing this feature.
First Name and Last Name are currently required on the registration form. Either they'll have to be passed into the script, or someone will have to make sure it won't break something to leave them blank.

@eryno I'm going to assign this to me and do the user:create approach, with a requirement to add in a password, first name, and last name.
I will examine what it might take for a one-time password into the system and the additional flow to check this and get folks to enter a password

Added this ability.  The one-time password is a secondary feature to be added


Have bootkube self-shutdown once control-plane running

Mostly just a few nits, otherwise looks great.

LGTM -- only thing I noticed during testing was we get a bunch of EOF errors when it shuts down. We should try for a more graceful shutdown if we can, but doesn't have to be part of this PR. I'd post the logs but I closed the terminal window too soon before thinking about it.


WIP: Usage Features

THIS PR IS A WORK IN PROGRESS

Adds gopsutil based "backend monitor" service

Currently makes no attempt to support backends not hosted on the local machine...

Beginning of TokenUsage implementation

That's looking good!
I'll have a look later in the detail, but a quick glance looks ok. Maybe let's call it feature flags instead of Chicken? maybe it's more idiomatic.
Re remote gRPC, this is quite possible already (and I use it for debugging) (see relevant code at: https://github.com/go-skynet/LocalAI/blob/4aa5dac768a5255667097db1c043581196fa46b2/pkg/model/initializers.go#L144) so I'm leaving here a note for those who are interested in running remote gRPCs in LocalAI.
LocalAI have the --external-grpc-backends parameter in the CLI that can be used either to specify a local backend (a file) or a remote URL:
./local-ai --debug --threads 14 --image-path tmp-imgr --external-grpc-backends "huggingface.py:/home/mudler/_git/LocalAI/extra/grpc/huggingface/huggingface.py"

./local-ai --debug --threads 14 --image-path tmp-imgr --external-grpc-backends "my-awesome-backend:host:port"

To try it out, you can for instance, after running make build to run one of the grpc server in backend-assets/grpc and use the printed URL directly in there - to use then the backend you need to specify the backend name in the model config file.

this is looking good overall! just few nits, that should not make it a blocker as could be just follow-ups, the real one is testing this on a GPU
@dave-gray101 would you mind me merging meanwhile #906? It might conflict with your PR, but actually is just about running make protobuf again and committing the results
If you can't test on GPU I'll give it a shot if not today, tomorrow at max

this is looking good overall! just few nits, that should not make it a blocker as could be just follow-ups, the real one is testing this on a GPU as it supersedes #896 that I was holding off to merge for the same reason (manual QA)
@dave-gray101 would you mind me merging meanwhile #906? It might conflict with your PR, but actually is just about running make protobuf again and committing the results
If you can't test on GPU I'll give it a shot if not today, tomorrow at max

Hey @mudler ! From my testing, CUDA seems to work fine with this branch. Granted... my test criteria consists of "normal output comes out, and process manager shows GPU load"


What are the dimensions of the NSGT_sliced output

I get the following shape from the NSGT:
# 125 frequency bins
scl = MelScale(78, 22050, 125)

nsgt = NSGT_sliced(scl, 9216, 2304, 44100, real=True, matrixform=True, multichannel=True)

forward = np.asarray(list(nsgt.forward((audio.T,)))).astype(np.complex64)

# shape of forward

# T is number of frames in time - by what division? is it "sllen + (0 <= n <= trlen)"?
# I believe the hop size and/or trlen/transition area is expected to be variable to maintain perfect invertibility.

# 2 channels because audio is stereo

# second-last dimension is specified frequency bins+1, so 126

# the last shape is nsgt.coef_factor*sllen = 304 - what is this dimension?

T x (2 channels) x 126 x 304

In the analog to an STFT (which typically has a shape like I x F x T for chan x frequency_bins x time_frames), what are the two frequency dimensions of the NSGT_sliced?
Are the 126 frequency bins interpolated or duplicated to create a bigger vector of 304 values?

Is there a recommended way to compress the 126 + 304 dimensions into a single one? Simple concatenation? Such that I can pass around "T x F" rectangular time-frequency matrices?

https://github.com/grrrr/nsgt/blob/master/examples/spectrogram.py#L19
Is this the use of the assemble_coeffs function? Is there a corresponding possibility to disassemble?


Have to change field of name for showing menu item in menu.

Laravel Version: 5.3
Voyager Version: 0.10.2
PHP Version: 7.0.10
Database Driver & Version:

Description:
I set the prefix in env file in laravel ( lp_ ). After that, I want to create groups menu item.  I create it in database and menu item but it wont be shown. The field of name in data_type table will be saved as lp_groups and when I change it to groups , it works ...
I have to go to database and change the lp_groups to groups for showing groups menu item in the menu.
Steps To Reproduce:

This is exactly what's happening to me. The BREAD functionalities won't work correctly when using a database prefix. This is because of your database manager class, I'll try work on this if I can achieve something

@wuelcas can you please explain the issue with BREAD and the database manager?

Ok, I made a fresh installation of laravel and voyager for this.

I set a prefix for the database tables called 'voyager_'

I installed voyager with dummy data by running the php artisan voyager:install --with-dummy command. The voyager tables were created with the prefix I set up earlier, everything is going according to plan

I login into the voyager app and go to Tools -> Database. I see that all the tables are listed with the prefix and not one has a BREAD created (some tables supposed to have a BREAD created).

Why is this happening?
After digging into the source code I found out that the SchemaManager class (found in tcg\voyager\src\Database\Schema\SchemaManager.php) is the one responsable for listing, creating, deleting, updating and all that stuff for the tables and it works managing the table names with the prefix but... when voyager seed the tables, the data_types table (table used in the BREAD funcionalities) is filled with table name data without the prefix I set up, so in the controller when it tries to match the BREADs, it doesn't match. After digging the attribute of the data_types table that the application tries to match with the table name is name (I'm sure you guys know this but just trying to make my point more clearly)
data_type table

A solution ????
After find all this, I figure that making a change in the DataTypesTableSeeder class and concatenating the prefix (e.g. 'name' => DB::getTablePrefix().'posts',) will do the trick to make the BREAD works, and it did. I deleted all the data in the data_type table and seed it all over again with the table prefix and it appears in the database manager, I also seed the data_rows table to update the newly created data types.

Everything is working correctly with the dummy data. But let's try to create a new table.
Let's create a table, let's call it 'cars' and let's keep it simple, an id and a name attribute and let's create the model and migration for the table

Everything works fine, the table is created but without the prefix, so I need to manually change the table name and add the prefix and from now on if I'm using a prefix I need to add the prefix when I create the table with the database manager.
So I change the name of the table from 'cars' to 'voyager_cars' and I create a BREAD for the table, generating the permissions. No problem. Important: please look the Input Type values for each row

New bug
I'll try to edit the newly created BREAD for the cars table and the info for the rows is not showing. I you see the images above I set up the Input Type for the two rows to be Number for the id and Text for the name but in both it says Checkbox.

Why? because the data for the rows were not created in the data_rows table.
Why?.... I digged into the code again and the responsable to store that data is the storeBread method in the VoyagerDatabaseController.php file. Line 254: $data = $dataType->updateDataType($request->all(), true). So I digged a little more and I go to the updateDataType method in the DataType model. To get the fields of the table you guys do this, line 74: $fields = $this->fields(array_get($requestData, 'name')); the value for array_get($requestData, 'name') will be the name of the table with the prefix, this is important for what is lying ahead. I go to that fields method in the same file and to get fields is in line 125: $fields = Schema::getColumnListing($name); Ok so digging a little bit more into the getColumnListing method in the Schema class and there it is this code:
public function getColumnListing($table)
    {
        $table = $this->connection->getTablePrefix().$table;

        $results = $this->connection->select(
            $this->grammar->compileColumnListing(), [$this->connection->getDatabaseName(), $table]
        );

        return $this->connection->getPostProcessor()->processColumnListing($results);
    }

Basically debugging the code the $table parameter is going to be 'voyager_cars' but inside the method there's this line: $table = $this->connection->getTablePrefix().$table; which transalates to $table = 'voyagervoyager_cars' and that table doesn't exists, so the result for the fields is null or an empty string, I'm not really sure, so it will never fill the data_rows table.
The data_types created by voyager already have data_rows data but if you try to update the BREAD, it won't work because the names have the prefix included.
You know what is awesome? When you create or update a BREAD it doesn't show any errors (notice the sarcasm)
It took me hours to find all this, working with prefix in voyager doesn't work out, it has a lot of bugs. Hopefully you guys will take this information and do something about it, I really like Voyager and I'm using it (obviously without using a prefix) and I really like to use it prefixing my tables. Thanks.

@wuelcas thanks for the detailed explanation 👍
So basically all those issues are caused by using a prefix. Here is my suggestion: instead of relying on Laravel to implicitly add the prefix for you, you explicitly hard code it in your tables, since Laravel's prefix works well with tables managed by Laravel, but not with tables managed by other packages (in this case, Voyager).
On Voyager's part, I think we'll add an option to specify a prefix for Voyager's tables during installation, and if left empty, we'll use the prefix specified in Laravel's config.
What do you think about that?

Hey I think is great! Thanks for reading and open that PR, hopefully it will get merged. The thing is I don't like to hardcode the prefix, Laravel does an excellent work managing the prefix  of the tables, hopefully in a future release of Voyager prefixes won't be an issue anymore.


CLI for setting schema update policy

Allows administrator of a namespace to specify how new schemas
provided by producers are validated. The policies are backward,
forward, full and disabled. If disabled, the administrator can still
update the schema via the schemas admin tool.

rerun cpp tests
rerun integration tests

rerun integration tests


Support ca_file for http client in curl mode

With curl simulation mode the parameters from curl needs to be passed
The parameters from uaac target xxx are not used here.
Therefore, ca cert was missing. -b for --skip-ssl-validation is available
For uaac target --ca-cert there is was no setting.
Added -C or --cacert because these are the parameters from curl

@beyhan Added you because this issue causes problems so that I would like to release uaac but also dont want break the cross company rules (about review) so that you and @rkoster should judge, can I merge and release ?

The change looks straight forward to me.  Best would be if this could be coordinated in the UAA area itself. My observation is that the cooperation in the UAA area works good.

f. My observation is that the cooperation in the UAA area works good.

Yes sure, the problem is that if there is an escalation then we need to have a new release in a very short time and for this I ask for your approval.
The cross company reviews work, so I asked you not because I dont get a review from VMware colleagues, but because of an SAP internal escalation

Feel free to go ahead and merge this or indicate if it's ok for us to merge.


update to make work

these changes allow this work again after changes were made on mopar's end

Thanks!


Suppress some harmless scanbuild warnings, cmdline realloc() leak fix

Fix duk_cmdline.c incorrect realloc() error handling where original buffer being reallocated might leak. Other scanbuild warnings are harmless; there are a few remaining issues which are intentional and harmless (e.g. intentional segfault on panic).

Remaining errors on Linux x64:
Intentional dereference of a NULL pointer when intentionally causing a segfault on panic:
dist/src-separate/duk_error_macros.c:151:2: warning: Dereference of null pointer
        DUK_CAUSE_SEGFAULT();  /* SCANBUILD: "Dereference of null pointer", normal */
        ^~~~~~~~~~~~~~~~~~~~
dist/src-separate/duk_config.h:2435:70: note: expanded from macro 'DUK_CAUSE_SEGFAULT'
#define DUK_CAUSE_SEGFAULT()  do { *((volatile duk_uint32_t *) NULL) = (duk_uint32_t) 0xdeadbeefUL; } while (0)
                                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 warning generated.
dist/src-separate/duk_debugger.c:180:9: warning: Access to field 'heap' results in a dereference of a null pointer (loaded from variable 'thr')
        heap = thr->heap;
               ^~~~~~~~~
1 warning generated.

Potential NULL pointer in debugger code; NULL pointer is not actually possible because debugger code is only entered from bytecode executor where thr is always non-NULL (not sure why scanbuild flags only this one place in the debugger code, but it seems to be harmless in any case):
dist/src-separate/duk_debugger.c:180:9: warning: Access to field 'heap' results in a dereference of a null pointer (loaded from variable 'thr')
        heap = thr->heap;
               ^~~~~~~~~
1 warning generated.

Harmless read of garbage data in Date handling. This is caused by a shared helper which is intentionally given a dummy uninitialized array by some call sites when a certain output is not needed. The access is memory safe (i.e. within the allocated stack array) but the data is uninitialized and any results from dealing with uninitialized data are ignored:
dist/src-separate/duk_bi_date.c:781:5: warning: Assigned value is garbage or undefined
                d = dparts[i];
                  ^ ~~~~~~~~~
1 warning generated.


Write release notes for Datahub Catalog 1.42 release

Added release notes for Datahub Catalog 1.42 release on Thursday 12th of August

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.Nam seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.

@MariaShaposhnikova I'll review and publish these RN, for your awareness please take a look


leshan.eclipse.org demo server issues

The instance at leshan.eclipse.org seems to be having issues at the moment:
http://leshan.eclipse.org/ --> shows the demo bootstrap server UI. I thought it should have shown the (non-bootstrap) server UI?
http://leshan.eclipse.org/bs/ --> gives 503 service unavailable.

On leshan.eclipse.org, our launching script was using environement variable but this is not available anymore since #301. I fixed the script. Thx for reporting this @scop ;).


Fix binding of SQL_DECIMAL/SQL_NUMERIC type as character data

Binding SQL fixed precision and scale data to C float-point value may lead to unexpected rounding or truncation errors, especially during retrieval of min or max values of particular SQL data types.
For example:
Max value of SQL Server data type MONEY is<PHONE_NUMBER>85477.5807.
If the MONEY column is bound as SQL_C_DOUBLE, then the fetched value is rounded to an odd value of<PHONE_NUMBER>85477.6250.

I have tested the fix against SQL Server only with the relevant test included.

Fixing CI build failures...


Optical flow for Niagara smoke

Hi,
Can you please confirm if EasySynth can generate optical flow for 3D Niagara smoke particles in UE5? If not, is there any workaround? I am trying to generate an optical flow simulated dataset of smoke plumes from a drone's perspective in a forest environment.

Hi @fezsid,
It seems that such a use case would be too optimistic. Due to UE limitations, optical flow currently only works correctly when the camera is moving, but the whole scene is stationary.
The most recent UE releases may have changed something, so I will give it a try for the next release.

Thank you! Please let me know if I can contribute in any way to achieve this task.


Installing GAM on Catalina project error

The issue tracker is for reporting product deficiencies. "How do I?" questions should be posted to the discussion forum at https://groups.google.com/group/google-apps-manager. When in doubt, start at the discussion forum and return here only when instructed to do so.
Please confirm the following:

I have upgraded to the latest GAM release from https://git.io/gamreleases and I still have this issue. YES
I am typing the command as described in the GAM Wiki at https://github.com/jay0lee/gam/wiki
YES

Full steps to reproduce the issue:
Running through the installation of GAM. I get to that part where I need to create a project and it asks for the Gmail admin email. I enter my admin email, approve granting access to GAM, copy the link it gives me, paste in terminal and I end up with the following error. I have tried revoking the API permission and starting over and no luck.
Expected outcome (what are you trying to do?):
Actual outcome (what errors or bad behavior do you see instead?):
ERROR: {'name': 'operations/cp.7503222764418340346', 'metadata': {'@type': 'type.googleapis.com/google.cloudresourcemanager.v1.ProjectCreationStatus', 'ready': True}, 'done': True, 'error': {'code': 7, 'message': "Permission 'resourcemanager.projects.create' denied on parent resource 'organizations/724465662266'.", 'details': [{'@type': 'type.googleapis.com/google.rpc.ResourceInfo', 'resourceType': 'project', 'resourceName': 'projects/gam-project-9p2-un3-y5u'}]}}
jhesse@lmjgarcia061221 ~ %

Permission 'resourcemanager.projects.create' denied on parent resource 'organizations/724465662266
you don't have rights to create projects in your GCP organization. you can:

Create the project under a GCP folder you DO have access to create projects under
gam create project parent <name of folder resource>
Pre-create the project, make sure you have full rights to update the project and then run:
gam use project

Thank you for the quick response! That was it.

Just a note that despite having Owner and Organization Administrator roles in my organization, I somehow still lack the resourcemanager.projects.create permission. Adding the Project Creator role fixes it: https://console.cloud.google.com/iam-admin


Confusing lack of any output when putting docblock below decorators

@ObjectType({
  description: "A person who has composed opera.",
})
/** @gqlType */
export default class Composer  {
  @Field((type) => String)
  /** @gqlField */
  url(): string {
    return `/composer/${this._composer.id}`;
  }
}

Silently extracts zero things. Moving the docblocks above the decorators works, but we should error in this case if possible to show you where to find them.

Note that this is a special case of a broader problem. We don't have a way to discover detached comments. TypeScript does not expose these at all in the AST. We can look to typescript-eslint as a role model here. It (transitively) uses this:
https://github.com/JoshuaKGoldberg/ts-api-utils/blob/fcd3128b61dfb54978b30f07380ad1c87a5d093e/src/comments.ts#L76

Closed by https://github.com/captbaritone/grats/pull/107