Defect: "sync all" Doesn't Sync

[ ] I am reporting a bug others will be able to reproduce and not asking a question or requesting a new feature.

System information including:

OpenCoarrays Version: 2.8.0
Fortran Compiler: GNU Fortran 8.3.0
C compiler used for building lib: gcc 8.3.0
Installation method: I installed from a downloaded tarball
All flags & options passed to the installer: N/A (normall CMake build)
Output of uname -a:  Linux Computer1 5.0.0-13-generic #14-Ubuntu SMP Mon Apr 15 14:59:14 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
MPI library being used: mpiexec --version shows version 3.3
Machine architecture and number of physical cores: Dell PowerEdge R910 Server with 32 cores (4x 8-Core Xeon X7560 2.27GHz)
Version of CMake: 3.13.4

To help us debug your issue please explain:
What you were trying to do (and why)
I have written a short Fortran program which uses coarrays and the sync all statement.  The sync all statement doesn't appear to work as expected.
What happened (include command output, screenshots, logs, etc.)
I compiled and ran the following program with the number of processors equal to three.
program sync_issue
    implicit none

    integer :: n, noe

    noe = 3
    do n = 1,noe
        print *, 'red ', n, ' ', this_image()
        sync all
        print *, 'green ', n, ' ', this_image()
        sync all
    end do
end program sync_issue

It produces the following result on the screen:
 red            1             1
 green            1             1
 red            2             1
 green            2             1
 red            3             1
 green            3             1
 red            1             2
 green            1             2
 red            2             2
 green            2             2
 red            3             2
 green            3             2
 red            1             3
 green            1             3
 red            2             3
 green            2             3
 red            3             3
 green            3             3

What you expected to happen
I expected it to print something like the following:
red            1             1
red            1             2
red            1             3
green            1             1
green            1             2
green            1             3
red            2             2
red            2             1
red            2             3
green            2             3
green            2             2
green            2             1
red            3             1
red            3             3
red            3             2
green            3             3
green            3             1
green            3             2

Step-by-step reproduction instructions to reproduce the error/bug
Compile the short program with caf sync_issue.f08 -o sync_issue.  Then run it with cafrun -np 3 sync_issue.

Apparently, this isn't a bug.  The problem arises due to the fact that information is buffered when using the print command.
In order to solve the problem, place the following command after your print statements: call execute_command_line('').  This will cause the buffered information to be released so that you can immediately see your printed results on the screen.
So, ultimately, my revised program now looks like this:
program sync_issue
    implicit none

    integer :: n, noe

    noe = 3
    do n = 1,noe
        print *, 'red ', n, ' ', this_image()
        call execute_command_line('')
        sync all

        print *, 'green ', n, ' ', this_image()
        call execute_command_line('')
        sync all
    end do
end program sync_issue

It now wonderfully works as expected.

How about
flush(output_unit)

where output_unit is from
use iso_fortran_binding, only: output_unit

instead of
call execute_command_line('')

os is the flush() statement not applicable to stdout?

Thanks, shahmoradi.
flush() is applicable to stdout, and may very well work.  I learned about it while trying to get this program to function properly.  It's the suggestion that kept popping up, but I could never get it to work.
In all of my searches, no one mentioned the need for iso_fortran_binding, which, apparently, is why I couldn't get flush() to work.
I don't have iso_fortran_binding installed, so I still haven't been able to confirm.  You're probably right that it'll work.
Once I discovered that call execute_command_line('') works, I stopped looking for other solutions.
Thanks for shedding more light on flush().  It's always helpful to understand the proper approach.

@evansste12432 iso_fortran_binding was a typo.  You want
use iso_fortran_env, only : output_unit

(On a historical note, I recently heard that the Oracle Fortran compiler used to have something called iso_fortran_binding, but it was never part of the Fortran standard.  I don't recall the purpose -- I think it had something to do with linking to legacy Fortran code that used some Oracle extension to the language.

Thanks for clearing that up, @rouson.  In that case, flush() still didn't work.
I've been using iso_fortran_env as I try to work with teams.  I'm assuming it was installed, by default, when I installed OpenCoarrays 2.8.0, since I have team functionality.  Am I wrong about that?
If I'm right, then the flush() approach still doesn't seem to work.

@evansste12432 iso_fortran_env is part of the Fortran standard and is installed automatically whenever gfortran is installed.  And yes, if you are using team_type, then you have iso_fortran_env.

Damian is right, my apologies for the typo, I use iso_fortran_env and iso_c_binding intrinsic modules often, and I got confused for a moment. These two are intrinsic and should be available in at least Intel, Cray, and GFortran compilers by default, as far as I am aware (perhaps also including PGI, IBM, NAG, ... compilers). The statement flush (output_unit) has worked in applications that I have worked with so far. I'd be surprised if it is otherwise.

@shahmoradi , I don't know why it isn't working for me.  If the following program produces correct results for you, then there would have to be something wrong with my system:
program sync_issue
    use iso_fortran_env, only : output_unit
    implicit none

    integer :: n, noe

    noe = 3
    do n = 1,noe
        print *, 'red ', n, ' ', this_image()
        flush(output_unit)
        sync all

        print *, 'green ', n, ' ', this_image()
        flush(output_unit)
        sync all
    end do
end program sync_issue

Hi @evansste12432 , I tested your code and confirm the behavior. This was a very interesting post and counter-intuitive for me, so I decided to bring this up on Intel forum to get a response from the community as well, including the Fortran committee chair Steve Lionel,
https://software.intel.com/en-us/forums/intel-fortran-compiler/topic/837101#comment-1948312
I'd suggest you get involved in the discussion and report any observations you have had there too. In sum, according to Steve, the observed behavior with flush() is correct, although I am not quite convinced yet, still waiting for his further clarifications in the discussion on the Intel forum.

Hello @shahmoradi.
While previously searching for solutions to this problem, I came across the following link:
https://software.intel.com/en-us/forums/intel-fortran-compiler/topic/531146
You'll notice that this issue has, in deed, been discussed in the same Intel forum.  At that time, flush 6 was suggested, and seemed to work for one person who was involved in the discussion -- Though, as previously stated, I could never get it to work.
During the conversation, more is explained from the developer, which may shed more light on the issue.  It was the last post in the conversation, which makes me think the explanation may have put all concern to rest.


Design the styleguide

Design the style guide with all the "Discussify - Browser Extension" components.

@andreforsousa please continuously update this issue with the latest version of the styleguide because I will be using this issue to link to others.

@satazor the current version is here:
https://drive.google.com/drive/folders/1tAz3IeAlQ0a88SGw2Q1URdXyQ52b3Cia?usp=sharing
Already prepared slices for the iconography being used at the moment.

Cool, feel free to put the task into review.

@satazor I've updated the file with the "Feedback Icons" and other minor fixes related with folders and symbols organization:
https://drive.google.com/drive/folders/1Y80dN0WqjJ9tfwyCuCOt_DqTGYrcBfpN?usp=sharing

@andreforsousa let's move the styleguide to the repo, and then we close this.

Updated styleguide, included the disable states for Colored background primary and secondary buttons. Also fixed color names and added "monza" (red) and "ripe-lemon" (yellow) considering the multiple error and warning scenarios.
Latest version here:
https://drive.google.com/drive/folders/1tajbF7bS1d5FExUHyUXawUvALbp9ceUf?usp=sharing

Still missing acceptance criteria 3, specifically we need to buy the font license.
We're using Avenir, which can be acquired in one of the following links:

https://www.myfonts.com/fonts/linotype/avenir
https://www.fonts.com/font/linotype/avenir

I'm a bit worried with the licensing model that these services provide. There really isn't a font that is adequate to our needs, we'd probably need to get in touch to come to an agreement with them. For the time being, I would recommend trying to use a Google Font, and later on we can revisit if it's worth it. Let's try and find a similar font there first.


No jump to shortest matching path

rupa/z provides the following feature which apparently is also implemented in fisherman/z but does not work for me:

Common: When multiple directories match all queries, and they all have a common prefix, z will cd to the shortest matching directory, without regard to priority.

Test scenario
> mkdir -p foo/bar/a
> mkdir -p foo/bar/b
> cd foo/
> cd bar
> cd a
> cd ../b
> z -l
4          /home/jnns/foo/bar/b
4          /home/jnns/foo/bar/a
4          /home/jnns/foo/bar
> z -e foo
/home/jnns/foo/bar

I expect the directory to be changed to foo but I'm taken to the most frecent entry instead – not the shortest match.

@jnns did you confirm the same behaviour to exist in rupa's z? I don't think you'll ever be taken to /foo because it's not explicitly in the z file, and z only ever brings you to directories that are explicitly listed in the file.

@jethrokuan I'm not sure whether a line went missing during copy'n'paste but I retried the following:
> cd foo/bar
> z -l
8          /home/jnns/foo/bar/b
8          /home/jnns/foo/bar/a
8          /home/jnns/foo/bar
8          /home/jnns/foo
> z -e foo
/home/jnns/foo/bar

This is what happens with rupa's z:
> cp ~/.local/share/z/data ~/.z
> z -l
common:    /home/jnns/foo
8          /home/jnns/foo
8          /home/jnns/foo/bar
8          /home/jnns/foo/bar/a
8          /home/jnns/foo/bar/b
> z -e foo
/home/jnns/foo

The behavior differs in the two implementations. Rupa's approach comes in handy when you want to return to a project root quickly, for example.

I can see where you're coming from. I'll look into it!

If anyone else is having this problem: a workaround I found useful is to use
z 'foo$'

This makes use of a regular expression that matches for a line end.


Veracode scan complains the malloc is not checked for return value.

https://github.com/zeux/pugixml/blob/9e382f98076e57581fcc61323728443374889646/src/pugixml.cpp#L186
We use Veracode for scanning our mobile application. It complains the following when pugixml lib is used.
The result of this call to malloc() is not checked for success before being used. This can result in application instability or crashing if memory is not available.
Can this be fixed?

There's nothing to fix here - feel free to submit a bug report for Veracode. This code is perfectly valid; the check for null pointer is located in the callers of the allocate callback. There's nothing sensible that the default_allocate function can do.

We are starting to use Veracode, the thing is that they are not doing
static code analysis but are doing a data flow analysis using the binaries.
So it is IMHO (not at all an expert with their tools) likely that one of
the callers of default_allocate() is not checking the result  for nullptr.
Easiest way to check this is manual code inspection I guess. Maybe
something OP could do?
(apart from that I second your nothing sensible statement)
Arseny Kapoulkine @.***> schrieb am Do., 23. Sept.
2021, 11:59:

There's nothing to fix here - feel free to submit a bug report for
Veracode. This code is perfectly valid; the check for null pointer is
located in the callers of the allocate callback. There's nothing sensible
that the default_allocate function can do.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/zeux/pugixml/issues/440#issuecomment-925668298, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AFZFAYKBO7WLKTJ5I6YAJ3LUDL3ALANCNFSM5ETIIBJQ
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You can grep for xml_memory::allocate - the result is always getting checked for 0. I suspect their dataflow analysis gets confused because of a function pointer, the allocation callback is replaceable and default_allocate is just, well, the default.


Some tags are not persisted in state although applied to resources

Description
I'm using few tags when creating EKS resources. All of them applied, but when running apply again I see 13 changes, with modification to the tags attribute - adding a single tag out of the configured 3.
⚠ Note
Before you submit an issue, please perform the following first:

Remove the local .terraform directory (! ONLY if state is stored remotely, which hopefully you are following that best practice!): rm -rf .terraform/
Re-initialize the project root to pull down modules: terraform init
Re-attempt your terraform plan or apply and check if the issue still persists

Done. Issue still persist. I tried to destroy and re-create the cluster few times. Same issue.
Versions

Terraform:

Terraform v1.1.3 on darwin_amd64

Provider(s):

+ provider registry.terraform.io/hashicorp/aws v3.71.0
+ provider registry.terraform.io/hashicorp/cloudinit v2.2.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.7.1
+ provider registry.terraform.io/hashicorp/local v2.0.0
+ provider registry.terraform.io/hashicorp/null v3.1.0
+ provider registry.terraform.io/hashicorp/random v3.0.0
+ provider registry.terraform.io/hashicorp/template v2.2.0
+ provider registry.terraform.io/hashicorp/tls v3.1.0

Module: terraform-aws-modules/eks/aws @ 18.0.5

Reproduction
Steps to reproduce the behavior:
Define tags as follows
locals {
  tags = {
    Managed-by                      = "Terraform"
    Environment                     = "my-env"
    Cloudwatch_to_s3_lambda_enabled = "true"
  }
}

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "18.0.5"

  cluster_name                         = var.cluster_name
  cluster_version                      = var.cluster_version
..
..

  tags = local.tags
}

Run apply, let terraform create the resources.
Once its done, run apply again. Notice the planned changes. All of them are the same, adding the tag to multiple resources.
..
      ~ tags                   = {
          + "Cloudwatch_to_s3_lambda_enabled" = "true"
            # (3 unchanged elements hidden)
        }

Plan: 0 to add, 13 to change, 0 to destroy.

I checked multiple resources, the tag is actually applied. Terraform state just isn't aware of it.
Not using workspaces.
I've tried to clear the cache, reinstall modules, recreate EKS cluster.
Code Snippet to Reproduce
See above. Rest of the code is similar to the examples.
Expected behavior
Tag be applied and kept in state. Further plan/apply executions shouldn't present this change.
Actual behavior
Tag is applied, but every plan/apply presents this as a change.
Terminal Output Screenshot(s)
Additional context
This worked properly with eks module v17.20

Changing the tag to CloudwatchToS3LambdaEnabled     = "true" seems to solve the issue.

unfortunately without a full reproduction, this doesn't tell us anything. please see the examples section as these examples have tags and work as intended


test(skeleton): add skeleton test with isLoaded

Closes #2073
Description
add skeleton test with isLoaded
Current behavior (updates)
Nil
New behavior
Nil
Is this a breaking change (Yes/No):
No
Additional Information

Thanks!


Datefield in Firefox isn't quite working

The datefield in Firefox is behaving quite strange, once you press backspace you get NaN-aN-aN in the inputfield.
I think this issue exists in all browsers which don't support native Datepickers.

Maybe using https://github.com/cibernox/ember-power-datepicker would be a better option.

The bug is still present (*) but it will require a fix in ember-one-way-controls that will be replaced soon.
(*) firefox 52.7.3 - NaN; safari - with native datepicker works as expected.


[Accessibility]A11y_AzureDataStudio_MapDataSrcObjects_Keyboard: On expanding tree view items with the help enter key ,keyboard focus directly shift to the fifth "Summary" screen.

Check out Keros! - Identify accessibility bugs before check-in and make bug fixing faster and easier.
GitHubTags: #A11yMAS; #A11yTCS; #MAS3.2.2; #A11y_AzureDataStudio; #MacOs; #DesktopApp
Environment Details:
Application name: Azure Data Studio
Version: 1.9.0-insider (user setup)
Commit: 2ca39b5
Electron: 3.1.8
Node.js: 10.2.0
V8: 6.6.346.32
OS :MacOS Mojave 10.14.5
Additional Details:
Tools Name: Keyboard
MAS References: MAS3.2.2
Steps to Reproduce:
Install the app Azure Data Studio.
Enable SQL server2019 preview->select table from the database->Create External Table screen should open->Next button->Create Database Master Key screen should open->click on next button->fill all the required details->click on next button.
Select any tree view item from left panel using arrow key and press enter.
Observe that keyboard focus directly shift to fifth "Summary" screen.
Actual:
On expanding tree view items with the help enter key ,keyboard focus directly shift to the fifth "Summary" screen. But when same thing is performed using mouse ,keyboard focus remains on the same screen.
Expected:
On expanding tree view items with the help enter key ,keyboard focus remains on the same screen .
User Impact:
Keyboard user will get confuse with the unauthorize behaviour of the expand tree view.
Recommendation:
Please refer the following snippet link for fixes:
https://microsoft.sharepoint.com/teams/msenable/mas/Pages/browse-fixes.aspx
MAS Reference :
MAS3.2.2:
https://microsoft.sharepoint.com/:w:/r/teams/msenable/_layouts/15/WopiFrame.aspx?sourcedoc={24cbfff7-374e-42e6-b9ab-d87fcd53b3e8}
Attachment for Reference:
6156_AccessibilityA11y_AzureDataStudio_MapDataSrcObjects_Keyboard.pptx

Dup of https://github.com/microsoft/azuredatastudio/issues/6153

@AccessibilityTestingTeam-TCS - I'm closing this as a duplicate, see #6153 for the status of this issue


PasteSelection on macOS pastes text from clipboard

I'm running 2821902a4dc0145f40a1a2fcfda4537793436308 on macos 10.12, and the middle button  action PasteSelection, as configured here:
mouse_bindings:
  - { mouse: Middle, action: PasteSelection }

does not paste the selection, but pastes from the clipboard. To reproduce, do the following on macos:

in alacritty, run echo foobar
Mark "bar"
Hit Cmd+C (the default action will copy)
Mark "foo"
Middle-click

Result: The text inserted is "bar", not "foo".
This seems like it's intended (and documented) to work as I'd expected, so is possibly a regression.

From what I can tell, this is due to copypasta's macos implementation of Load for Clipboard: It believes macOS has no such thing as a selection, and so load_selection defaults to load_primary, loading the primary clipboard.
I think the fix for this is probably to extend copypasta to allow storing a selection manually, and alacritty doing that whenever the selection changes.

I know there's been a lot of changes to clipboard handling since was opened, but I've just got a mac to test on, and can confirm this is still an issue.

It's a slight question of definition. What the PasteSelection action does, is just paste from the selection clipboard. It has nothing to do with the active selection, so it's possible to paste the selection even after it doesn't exist anymore.
To support this on macOS, one would usually set it so that selections are stored to the primary clipboard. It's interesting to me that one would want to paste the currently active selection, which is kind of a completely different thing from how this is understood on Linux.


Fix: out-of-dated docker img & tzdata option bug

In order to close #1 .
Description

images in Dockerfile has been out-of-dated.
When build with dockerfile, it is a frequent bug that getting stuck when choosing time zone for tzdata package.

LGTM. thank you


CLI: migrate status should return a non-successful exit code (1) when a failed migration is found or an error occurs

Bug description
From https://github.com/prisma/prisma/issues/12349
migrate status returns a successful exit code (0) in unexpected situations like
when a failed migration is found
P1000: Authentication failed against database server at localhost
How to reproduce

when a failed migration is found

npx prisma migrate --create-only
# edit the .sql file and add "BROKEN"
npx prisma migrate deploy
npx prisma migrate status

P1000: Authentication failed against database server at localhost

# edit .env and change the username or password to an incorrect one
npx prisma migrate status

Expected behavior
It should return a non-successful exit code -> 1
Prisma Version
4.3.0-dev.27

It will now exit(1) = error in these cases
if database connection error
If databaseIsBehind (= unapplied migrations)
If historiesDiverge (= migration history diverged)
If we can't find a migration table (= not managed by Prisma)
If failed migration(s) are found

It's now in our internal dev version 4.3.0-dev.56 and up, and will be part of the 4.3.0 release.

Replying to this comment from the linked PR

If databaseIsBehind = unapplied migrations -> exit(1)

The other instances of errors make sense, but does this one? I'm not sure that unapplied migrations are error-worthy.
I only bring this up because my team used prisma migrate status as a part of our build process to list all unapplied migrations and then apply them. This change broke our pipeline by introducing a new error.


fix: handle context cancellation in start cmd

based off #293

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 7 (0.0%)  changed or added relevant lines in 2 files are covered.
3 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-0.005%) to 27.13%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

cmd/start.go
0
1
0.0%

internal/start/start.go
0
6
0.0%

Files with Coverage Reduction
New Missed Lines
%

internal/start/start.go
3
0%

Totals

Change from base Build<PHONE_NUMBER>:
-0.005%

Covered Lines:
1369

Relevant Lines:
5046

💛  - Coveralls


feat: added user submissions

#5

Can u post some screenshots of what you've built here?

hmm let me figure out the UX of this before committing to it.

hmm let me figure out the UX of this before committing to it.

alright np


[feat] start registry implementation - WIP

This PR should addresses issues #3 , #5, #7, #10, #17, and probably part of #20.
Note mostly to self: I'm currently assuming we'll use OpenZeppelin's AccessControl Component, but it makes it cumbersome to use u256 instead of felt252 for the profiles/roles. I guess it'll make sense down the road or I'll ask.

Hello @trevis-dev , I'm really happy that you've taken this issue. But this is yet to be announced and is for a closed community for onboarding new dev contributions

Hey @Akashneelesh , thank you for the info.
I saw this on OnlyDust and thought I could help. Please let me know if anything changes then.
I've enjoyed working on this anyway :)


Unable to create jqueryUI select menu, etc. inside w2ui layout

Love this framework but running into issues porting existing code into w2ui layouts... basic example is a simple jqueryUI selectmenu:
W2UI Layout pane content:
<select id="sel_menu">
    <option>one</option>
    <option>two</option>
</select>'

Then inside script:
$( function() {
    $('#sel_menu').selectmenu();
});

The select menu is styled after the jqueryUI them but the dropdown is never invoked when the select menu is clicked.  This works perfectly when --outside-- of the w2ui layout, but doesn't work within it.  Am I missing something (entirely possible!) or is this a bug?  Thanks so much.

Only bugs here. All questions to w2ui.com or stackoverflow.com. Attach to http://w2ui.com/web/docs/common.onRefresh and you should be fine.


[<IP_ADDRESS>] TSM1 [cluster] 2015/12/30 21:46:39 drop write request: shard=36. shard group does not exist or was deleted

hello,
I deploy a influxdb cluster in a test environment, by node1, node2, node3.
Influxdb cluster running for three days in the normal state.At this time, a large number of frequent writes with a little read.
Just, I found the data query by influxdb-python, sometimes can be found, sometimes can not find. So, I see the log information of a three node influxdb cluster. I found in the node 2 on the following error message.
[write] 2015/12/30 21:11:15 write failed for shard 28 on node 3: engine: cache maximum memory size exceeded
[tsm1] 2015/12/30 21:11:15 error writing snapshot: error opening new segment file for wal (1): invalid argument

I checked the issue column in the GitHub influxdb project, with reference to the following issue.

https://github.com/influxdb/influxdb/issues/5191
https://github.com/influxdb/influxdb/issues/4167

After I order the closure of the influxdb cluster, adjusting influxdb value as follows:
[data]
Max-wal-size =<PHONE_NUMBER>0
Wal-ready-series-size = 256000
Wal-max-series-size = 20971520
Wal-partition-size-threshold = 209715200
Cache-max-memory-size =<PHONE_NUMBER>0
Cache-snapshot-memory-size =<PHONE_NUMBER>
[hinted-handoff]
Max-size =<PHONE_NUMBER>0

After the restart, after 2 minutes, and began to report the following information in node 1 and node 2.
Node 3 is normal.
node 1 :
[write] 2015/12/30 21:29:13 write failed for shard 32 on node 3: read message type: EOF
[cluster] 2015/12/30 21:33:00 drop write request: shard=37. shard group does not exist or was deleted
[cluster] 2015/12/30 21:33:10 drop write request: shard=37. shard group does not exist or was deleted
[cluster] 2015/12/30 21:33:40 drop write request: shard=37. shard group does not exist or was deleted
[cluster] 2015/12/30 21:33:50 drop write request: shard=37. shard group does not exist or was deleted
[cluster] 2015/12/30 21:34:00 drop write request: shard=37. shard group does not exist or was deleted
[metastore] 2015/12/30 21:34:00 rpc: rpc connection from: <IP_ADDRESS>:56769
[metastore] 2015/12/30 21:34:00 rpc: recv FetchData request on: <IP_ADDRESS>:56769
[cluster] 2015/12/30 21:34:10 drop write request: shard=37. shard group does not exist or was deleted

node 2 :
[cluster] 2015/12/30 21:46:49 drop write request: shard=36. shard group does not exist or was deleted
[cluster] 2015/12/30 21:46:59 drop write request: shard=36. shard group does not exist or was deleted
[cluster] 2015/12/30 21:47:09 drop write request: shard=36. shard group does not exist or was deleted
[cluster] 2015/12/30 21:47:19 drop write request: shard=36. shard group does not exist or was deleted
[cluster] 2015/12/30 21:47:29 drop write request: shard=36. shard group does not exist or was deleted
[cluster] 2015/12/30 21:47:39 drop write request: shard=36. shard group does not exist or was deleted
[cluster] 2015/12/30 21:47:49 drop write request: shard=36. shard group does not exist or was deleted
[cluster] 2015/12/30 21:47:59 drop write request: shard=36. shard group does not exist or was deleted
[cluster] 2015/12/30 21:48:09 drop write request: shard=36. shard group does not exist or was deleted
[cluster] 2015/12/30 21:48:19 drop write request: shard=36. shard group does not exist or was deleted
[cluster] 2015/12/30 21:48:29 drop write request: shard=36. shard group does not exist or was deleted
[cluster] 2015/12/30 21:48:39 drop write request: shard=36. shard group does not exist or was deleted
[cluster] 2015/12/30 21:48:49 drop write request: shard=36. shard group does not exist or was deleted
[cluster] 2015/12/30 21:48:59 drop write request: shard=36. shard group does not exist or was deleted
[cluster] 2015/12/30 21:49:09 drop write request: shard=36. shard group does not exist or was deleted
[cluster] 2015/12/30 21:49:19 drop write request: shard=36. shard group does not exist or was deleted
[cluster] 2015/12/30 21:49:29 drop write request: shard=36. shard group does not exist or was deleted
[cluster] 2015/12/30 21:49:39 drop write request: shard=36. shard group does not exist or was deleted

My /etc/influxdb/influxdb.conf profile information as follows.
reporting-disabled = false
[registration]
[meta]
  dir = "/home/influxdb/meta"
  hostname = "influxdb1.xxx.com"
  bind-address = ":8088"
  retention-autocreate = true
  election-timeout = "1s"
  heartbeat-timeout = "1s"
  leader-lease-timeout = "500ms"
  commit-timeout = "50ms"
  cluster-tracing = true
  raft-promotion-enabled = true
[data]
  dir = "/home/influxdb/data"
  engine = "tsm1"
  wal-dir = "/home/influxdb/wal"
  wal-logging-enabled = true
  wal-ready-series-size = 256000
  wal-max-series-size = 20971520
  wal-flush-cold-interval = "10m"
  wal-partition-size-threshold =<PHONE_NUMBER>
  cache-max-memory-size =<PHONE_NUMBER>0
  cache-snapshot-memory-size =<PHONE_NUMBER>
[hinted-handoff]
  enabled = true
  dir = "/home/influxdb/hh"
  max-size =<PHONE_NUMBER>0
  max-age = "168h"
  retry-rate-limit = 0
  retry-interval = "1s"
  retry-max-interval = "1m"
  purge-interval = "1h"
[cluster]
[retention]
  enabled = true
  check-interval = "30m"
[shard-precreation]
  enabled = true
  check-interval = "10m"
  advance-period = "30m"
[monitor]
[admin]
  enabled = true
  bind-address = ":8083"
  https-enabled = false
  https-certificate = "/etc/ssl/influxdb.pem"
[http]
  enabled = true
  bind-address = ":8086"
  auth-enabled = false
  log-enabled = true
  write-tracing = false
  pprof-enabled = false
  https-enabled = false
  https-certificate = "/etc/ssl/influxdb.pem"
[[graphite]]
  enabled = false
[collectd]
  enabled = false
[opentsdb]
  enabled = false
[[udp]]
  enabled = false
[continuous_queries]
  log-enabled = true
  enabled = true
  recompute-previous-n = 2
  recompute-no-older-than = "10m"
  compute-runs-per-interval = 10
  compute-no-more-than = "2m"

Please help me under investigation, thanks！！

@beckettsean
Can you tell me where is the problem?
Thanks!

@holau no, I do not know.
@kostasb or @rossmcdonald any ideas?

When I writing a lot of data into influxdb cluster, I restart influxdb node 2 directly, the bug is following！
CMD Log : /var/log/messages：
Jan 15 17:35:01 localhost -bash: HISTORY: PID=2947 PPID=2944 SID=2947 User=root USER=sucan163 CMD=service influxdb restart
Influxdb Log : /var/log/influxdb/influxd.log:
[cluster] 2016/01/15 17:35:32 drop write request: shard=39. shard group does not exist or was deleted

@beckettsean Can you try when cluster writing lot of data then restart some of node, the bug will be come out.

@mjdesa @rossmcdonald here's a use case to probe with the cluster testing. During sustained write volume, restart a node.

The drop write request: shard=37. shard group does not exist or was deleted type errors are likely due to hinted handoff writes queued on other nodes for shards that were removed.  They could be removed due to retention policy, dropping DBs, etc..

Closing since this code no longer exists in master.


implemented DetoneCovarianceTransformer

It does, and it should. By the way, we get the reordered corr matrix if we
use the recorded w and v to do v*diag(w)*v'
On Wed, May 27, 2020 at 2:22 PM stevey628<EMAIL_ADDRESS>wrote:

@stevey628 commented on this pull request.
In mcos/covariance_transformer.py
https://github.com/enjine-com/mcos/pull/46#discussion_r431449977:

Derive the correlation matrix from a covariance matrix

:param cov: covariance matrix
:return: correlation matrix
"""
std = np.sqrt(np.diag(cov))
corr = cov / np.outer(std, std)
corr[corr < -1], corr[corr > 1] = -1, 1  # numerical error
return corr

   corr = cov_to_corr(cov)

   w, v = linalg.eig(corr)

   # sort from highest eigenvalues to lowest

   sort_index = np.argsort(-np.abs(w))  # get sort_index in descending absolute order - i.e. from most significant

   w = w[sort_index]

   v = reorder_matrix(v, sort_index)

reorder_matrix reorders both columns and rows of the matrix. won't
reordering the rows of the eigenvectors change the vectors themselves?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/enjine-com/mcos/pull/46#pullrequestreview-419618332,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAXTIXOM52ABFAUUJ7ZJW7TRTWABRANCNFSM4NMIZTKA
.

*reordered
On Wed, May 27, 2020 at 2:25 PM Jin Won Choi<EMAIL_ADDRESS>wrote:

It does, and it should. By the way, we get the reordered corr matrix if we
use the recorded w and v to do v*diag(w)*v'
On Wed, May 27, 2020 at 2:22 PM stevey628<EMAIL_ADDRESS>wrote:

@stevey628 commented on this pull request.
In mcos/covariance_transformer.py
https://github.com/enjine-com/mcos/pull/46#discussion_r431449977:

Derive the correlation matrix from a covariance matrix

:param cov: covariance matrix
:return: correlation matrix
"""
std = np.sqrt(np.diag(cov))
corr = cov / np.outer(std, std)
corr[corr < -1], corr[corr > 1] = -1, 1  # numerical error
return corr

   corr = cov_to_corr(cov)

   w, v = linalg.eig(corr)

   # sort from highest eigenvalues to lowest

   sort_index = np.argsort(-np.abs(w))  # get sort_index in descending absolute order - i.e. from most significant

   w = w[sort_index]

   v = reorder_matrix(v, sort_index)

reorder_matrix reorders both columns and rows of the matrix. won't
reordering the rows of the eigenvectors change the vectors themselves?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/enjine-com/mcos/pull/46#pullrequestreview-419618332,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAXTIXOM52ABFAUUJ7ZJW7TRTWABRANCNFSM4NMIZTKA
.

actually, I guess I wouldn't have to reorder the corr matrix if I only
swapped the eigenvectors around. I'll update the code.
On Wed, May 27, 2020 at 2:25 PM Jin Won Choi<EMAIL_ADDRESS>wrote:

*reordered
On Wed, May 27, 2020 at 2:25 PM Jin Won Choi<EMAIL_ADDRESS>wrote:

It does, and it should. By the way, we get the reordered corr matrix if
we use the recorded w and v to do v*diag(w)*v'
On Wed, May 27, 2020 at 2:22 PM stevey628<EMAIL_ADDRESS>wrote:

@stevey628 commented on this pull request.
In mcos/covariance_transformer.py
https://github.com/enjine-com/mcos/pull/46#discussion_r431449977:

Derive the correlation matrix from a covariance matrix

:param cov: covariance matrix
:return: correlation matrix
"""
std = np.sqrt(np.diag(cov))
corr = cov / np.outer(std, std)
corr[corr < -1], corr[corr > 1] = -1, 1  # numerical error
return corr

   corr = cov_to_corr(cov)

   w, v = linalg.eig(corr)

   # sort from highest eigenvalues to lowest

   sort_index = np.argsort(-np.abs(w))  # get sort_index in descending absolute order - i.e. from most significant

   w = w[sort_index]

   v = reorder_matrix(v, sort_index)

reorder_matrix reorders both columns and rows of the matrix. won't
reordering the rows of the eigenvectors change the vectors themselves?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/enjine-com/mcos/pull/46#pullrequestreview-419618332,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAXTIXOM52ABFAUUJ7ZJW7TRTWABRANCNFSM4NMIZTKA
.


[SP-4501] Backport of PDI-17427 - Job Executor Step doesn't read fiel…

…ds correctly (8.1 Suite)

Build Completed
:fire: This pull request has some issues. It would be preferable to fix them in order for it to be just perfect. See below for more details. Some links are also available below for further assistance in addressing those issues.
Build Commands
mvn -B -fn -f 'pom.xml' -pl 'engine' -P '!assemblies' -Dsurefire.runOrder=alphabetical -Daudit -amd clean install

Cleanup Commands
mvn -B -f 'pom.xml' -pl 'engine' -P '!assemblies' -amd build-helper:remove-project-artifact

Changed files
  engine/src/main/java/org/pentaho/di/trans/steps/jobexecutor/JobExecutor.java

Unit Test Coverage
These statistics help you identify how your changes have affected the coverage of the following files. If a file is not in this list, then its coverage was not affected by your changes. To get some help interpreting these metrics, please refer to Jacoco's documentation.

:warning: Coverage Changes (click to expand)
org.pentaho.di.job.entries.evalfilesmetrics.JobEntryEvalFilesMetrics

Branch Change: -0.4% :small_red_triangle_down:
Complexity Change: -0.5% :small_red_triangle_down:
Instruction Change: -0.1% :small_red_triangle_down:

org.pentaho.di.trans.steps.constant.ConstantMeta

Instruction Change: -0.1% :small_red_triangle_down:

org.pentaho.di.trans.steps.datagrid.DataGridMeta

Branch Change: -2% :small_red_triangle_down:
Complexity Change: -1.7% :small_red_triangle_down:
Instruction Change: -0.2% :small_red_triangle_down:

org.pentaho.di.trans.steps.replacestring.ReplaceStringMeta

Instruction Change: -0.1% :small_red_triangle_down:

org.pentaho.di.trans.steps.rowgenerator.RowGeneratorMeta

Branch Change: -2.6% :small_red_triangle_down:
Complexity Change: -1.5% :small_red_triangle_down:
Instruction Change: -0.1% :small_red_triangle_down:

org.pentaho.di.trans.steps.sort.SortRowsMeta

Instruction Change: -0.1% :small_red_triangle_down:

org.pentaho.di.trans.steps.sortedmerge.SortedMergeMeta

Instruction Change: -0.2% :small_red_triangle_down:

org.pentaho.di.trans.steps.uniquerows.UniqueRowsMeta

Instruction Change: -0.1% :small_red_triangle_down:

org.pentaho.di.trans.steps.gpload.GPLoadMeta

Branch Change: -0.9% :small_red_triangle_down:
Complexity Change: -0.8% :small_red_triangle_down:
Instruction Change: -0.2% :small_red_triangle_down:


Adding download links

Environment
not related to deployment
Issue description
In the Readme's development prerequisite section only the download link of Golang is made available. We can the downloads links for Git, and others also.
Error log
nothing
Expected result
No response

Please assign this to me, I'd like to fix it.


ReleaseAtLeastNPages sometimes take up 1 second ?

Hi, tcmalloc seems to have a long interval step within pageheap lock, which causes multiple threads in the process stuck at SpinLock::SlowLock to wait for the pageheap lock.
the problem code is https://github.com/gperftools/gperftools/blob/master/src/page_heap.cc#L204
    ReleaseAtLeastNPages(static_cast<Length>(0x7fffffff));

Our program is a memory intensive process, and when it executes this code during runtime, causing multiple threads to get stuck and not respond for a long time. Is there a good solution?
We added some trace logs and found that a ReleaseAtLeastNPages call freed 11GB of memory and made 7w system calls madvise. And this take up to 1 second.
Out env:

linux kernel: 5.10
arch: aarch64
gperftools: 2.6.90

The tcmalloc stat like this:
MALLOC:<PHONE_NUMBER>8 (14930.6 MiB) Bytes in use by application
MALLOC: +<PHONE_NUMBER>2 (18490.2 MiB) Bytes in page heap freelist
MALLOC: +    766933264 (  731.4 MiB) Bytes in central cache freelist
MALLOC: +      9185792 (    8.8 MiB) Bytes in transfer cache freelist
MALLOC: +    109744464 (  104.7 MiB) Bytes in thread cache freelists
MALLOC: +     65929216 (   62.9 MiB) Bytes in malloc metadata
MALLOC:   ------------
MALLOC: =<PHONE_NUMBER>6 (34328.6 MiB) Actual memory used (physical + swap)
MALLOC: +<PHONE_NUMBER> ( 1115.4 MiB) Bytes released to OS (aka unmapped)
MALLOC:   ------------
MALLOC: =<PHONE_NUMBER>6 (35443.9 MiB) Virtual address space used
MALLOC:
MALLOC:         272458              Spans in use
MALLOC:            204              Thread heaps in use
MALLOC:           8192              Tcmalloc page size

Thanks for the report.
So what is "7w system calls"? Is that 7 million (1e6) or billion (1e9) or something else ?
In general, yes, this place have potential to cause significant delays. The comment around the line you referenced explains why this is done. Making it much better is a genuine challenge.
There are couple directions of attack I can think of:
a) lean a bit more about those MADV_DONTNEED calls. Does look like your workload might be producing a ton of small chunks and we cannot pass them all in one swoop to kernel. So it takes a ton of syscalls (and syscalls might be quite expensive this days). So perhaps there is something in your workload we could tweak to make it do less page heap fragmentation.
b) depending on workload we might make help it have fewer small spans by configuring tcmalloc with larger logical page size (--with-tcmalloc-pagesize flag of configure)
c) as step 0 you should definitely update to newest gperftools because we've learned how to do those madvise calls without holding "giant" page heap lock. It should help, but whether it'll be enough help remains to be seen.

Sorry, the words I described before was too localized. "7w system calls" means 70 thousand system calls.
I have some questions about your proposed solution:
a) We understand this and will optimize the program to some extent. However, this scenario is quite extreme and difficult to optimize. Our program may continuously request memory during operation, and then release all memory in a short time when the program is about to exit, which leads to this situation.
b) As mentioned in a), this situation will not occur when the program runs normally, so we do not plan to adjust the tcmalloc-pagesize parameter specifically for this situation.
c) In the latest version 2.15, I see that "ReleaseAtLeastNPages(0x7fffffff)" is still called in function PageHeap::NewLocked within page heap lock. It's just that madvise may be replaced by mmap, but it is still a system call within the lock. Am I wrong?
Finally, I have two questions:

If we want to fix this problem quickly, can we directly delete the code near "ReleaseAtLeastNPages(0x7fffffff)"? That is just this link

How is 0x7fffffff in "ReleaseAtLeastNPages(0x7fffffff)" defined? This is a very large number, which will almost certainly release all the free memory of the process. Have you considered the impact of it occupying the pageheap lock for a long time?

for (a): it is up to you what to do, but here is one "heretical idea": perhaps just exit and don't free anything? Also do keep in mind that this exact code path happens when we are allocating memory and we couldn't find big enough span of free or released memory. So this should be near peak usage (modulo fragmentation). I.e. if don't do it, we'd 100% surely grab more memory from OS.
for (b): sure
for (c): Ah, I somehow forgot that I haven't haven't returned "unlock-around-release" logic. So please ignore my original comment.
(1). Sure. But please if you ever file more tickets or update existing ones, please, mention that you have made this modification. I.e. you'll re-introduce (arguably rare) issue fixed by original commit that added this logic.
(2). The idea is to release everything.
I am closing this, assuming this is understood. Please reopen if you think some work is needed here or if something isn't yet fully understood.


Make everything available on the API websocket

TL;DR: for all models (with security measures), send creates and updates to the API websocket.
Currently, we only send a select amount of stuff down the API websocket. I've heard a number of times over the past few timescales that X wants Y other thing sent down it. Debate then ensues about whether it's worth doing it for the number of people who will actually use it.
I propose we make everything (within limits, of course - no sending secure fields) available across the websocket, filterable by clients on connection. For example:
GET /cable?events=flag_logs;posts;feedbacks HTTP/1.1
Host: metasmoke.erwaysoftware.com

would result in a WS upgrade to a socket that only sends that specific client creates and updates for flag logs, posts, and feedbacks - and nothing else.
This solves issues of both bandwidth and having to make things available case by case.
I have an idea or few of how to do this, but if I'm doing it it'll need to wait until I get back from holiday.

I’d recommend passing parameters like the API key is currently passed so official ActionCable clients can still consume it easily.

Sure, I just don't know how that is so I couldn't use it in an example. That's just the initial HTTP request, not the WS upgrade handshake.

https://github.com/rails/rails/tree/master/actioncable#passing-parameters-to-channel:

You can pass parameters from the client side to the server side when creating a subscription. For example:
# app/channels/chat_channel.rb
class ChatChannel < ApplicationCable::Channel
  def subscribed
    stream_from "chat_#{params[:room]}"
  end
end

If you pass an object as the first argument to subscriptions.create, that object will become the params hash in your cable channel. The keyword channel is required.
# Client-side, which assumes you've already requested the right to send web notifications
App.cable.subscriptions.create { channel: "ChatChannel", room: "Best Room" },
  received: (data) ->
    @appendLine(data)

  appendLine: (data) ->
    html = @createLine(data)
    $("[data-chat-room='Best Room']").append(html)

  createLine: (data) ->
    """

      #{data["sent_by"]}
      #{data["body"]}

    """

# Somewhere in your app this is called, perhaps from a NewCommentJob
ActionCable.server.broadcast \
  "chat_#{room}", { sent_by: 'Paul', body: 'This is a cool chat app.' }

Would have to confirm, but I suspect those end up as the query string on the HTTP request. Anyway, we probably don't need to go into implementation details yet.


Allow "unable to find workflow type" error to be configured to hard-fail the workflow

Is your feature request related to a problem? Please describe.
Currently we allow this error to support retry so a worker can be deployed with the proper workflow for it to work.
Describe the solution you'd like
Need to investigate options about making it non-retryable.

The best approach for this may be to go ahead and support #543 so a catch-all workflow implementation can be written


undefined: commandeer.ImplementsRunner

I tried to run myapp-cobrafy which gives undefined: commandeer.ImplementsRunner at line 31 in cobrafy/cobra.go
full error
~/dev/golang/src/github.com/jaffee/commandeer/examples/myapp/cmd/myapp-cobrafy master
» go run main.go
# github.com/jaffee/commandeer/cobrafy
../../../../cobrafy/cobra.go:31:5: undefined: commandeer.ImplementsRunner

that's odd... can you make sure you have the latest version of the master branch?
I am not able to reproduce this issue and ImplementsRunner is clearly defined in com.go

I pulled and I'm up to date with master. The latest commit I have is 719d29f8107ee2337fef4e1680770ea58780ff32 which is the same one github shows, and I still don't have that function. I couldn't find it with the github search either (or just by looking at com.go on github)

as far as I can tell it looks like it was removed in #4

ah ha! I didn't have the latest master 🤦♂
Thank you! I pushed a fix to master


Support for reverse proxies

Currently, both cluster_health.go and nodes.go are overwriting the path provided by the es.uri flag, assuming that it will always be empty. However, this assumption is not always correct: In some cases it's useful to access the Elasticsearch endpoint behind a reverse proxy (e.g., Kubernetes) for testing purposes. This fix adds support for this scenario.

Just realized I missed indices.go. Working on the fix

Hi,
thanks for your contribution! It makes sense to support reverse proxying. I'm currently on holidays with my kids, so I can't review it before the weekend

looks good, thanks for your contribution!


S3File.info() -- performance problems under prefix with a huge amount of keys

Currectly, s3.head_object is used only if version_aware flag of S3FileSystem is set to True. In other cases, it defaults to fsspec implementation, which is based on ls(). In case when a large amount of keys are stored under a specific prefix, this leads to very slow procedure of getting object information or even getting Access Denied errors due to AWS API rate limiting, since many consequent requests should be made in order get the complete object listing.
Is there any advantage of using fsspec's implementation over boto3 s3.head_object implementation?
If not, I'd be happy to provide a PR for using s3.head_object for S3File.info() regardless of version_aware flag.

This is an old discussion, which predates fsspec. The ls() call caches the file detilas for the given directory, so in the case where you have not too many entries, or mean to get details about most of them, it is right to list everything. You case, where you want a small number out of a much larger number, is the opposite extreme. I don't know how to accommodate both, except of course info() itself could have an extra kwargs.
The reason this is different for versions, is that the standard bucket listing does not provide version information, but head does.
By the way, this argument was also had with regards to listing buckets by prefix ("directory"), as we do now, versus getting the complete bucket list, as we did some time in the past. In some cases, the single call for the whole bucket is better, but there is no way to guess ahead of time when that might be.


Add info on how to run/debug a test via VSCode

Also, the predefined launch.json can be added directly into the repository (/.vscode/launch.json) instead of being mentioned in Readme but I don't know if we want to have IDE specific file directly in the repo, so, I only mentioned it in the Readme.

Look great for me. Really nice to have that explanation on the read me file

@fjglira sorry, I totally missed the notification from this pr.
I noticed that some jobs are still waiting. Do I need to add openshift-ci bot into my fork repo in order to move this forward?

/ok-to-test

Hi @mkralik3 , sorry for the delayed answer. Checking if I can add you to the organization

I can't add you as member of the QE team (I do not have the privileges) but I added you as a mantainer. Please take a look


support osx arm64

Checklist

[ ] Used a personal fork of the feedstock to propose changes
[ ] Bumped the build number (if the version is unchanged)
[ ] Reset the build number to 0 (if the version changed)
[ ] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[ ] Ensured the license file is being packaged.

@danielnachun - thanks very much for this!  Would you like to be added as a maintainer as well?

Yes, please feel to add me.


Redis ACL, how to set username in Redis connection?

System information:

Operating system MacOS Big Sur
DBeaver version EE 7.3.0

Connection specification:

Redis server 6.0.7
Redis
Redis with username/password with SSL

Describe the problem you're observing:
Unable to connect to Redis with ACL (username, password). There is no dialogue box to input username. Also the "Edit driver settings" seems read only. Is this supported?
Include any warning/errors/backtraces from the logs
Could not get a resource from the pool
WRONGPASS invalid username-password pair
WRONGPASS invalid username-password pair

This feature will be added in DBeaver EE 21 (the next version).


Can I somehow temporarily disable Popper?

I just have a simple question. Is it possible to temporarily prevent popper from calculating position (on scroll mainly) when i need to? E.g I have tooltip component and when element isn't visible i see that popper is still calculating position when I scroll and onUpdate callback is called like 100 times. Can I prevent this from happening without initializing instance before tooltip is shown and destroying it after is hidden? (that sounds like a really bad idea though :/). Unfortunately I don't see any option which could help me with this, so is there currently any solution for this problem of mine? Thanks for the any help/tips/anything.
P.S. I am using latest version - v1.0.0-beta.1

hi! first, latest beta is beta.2 😛
focusing on your question:
what's the problem in destroying and recreating the instance when you toggle the visibility?

what's the problem in destroying and recreating the instance when you toggle the visibility?

Hmm not sure.. just feel that it's unnecessary to do it every time I toggle my tooltip, I would like to initialize an instance just once. Also i guess it would be more memory efficient to initialize popper just once. What you think about adding disabled option that would turn off all modifiers? It wouldn't be so difficult to implement I think. If you want, I can try to do it.

Sure, feel free to send a PR with this feature. Thanks!


Pickle Pint Quantities

Install copy_reg handlers to pickle and unpickle Pint objects properly.

This PR fixes an issue where pickled Pint Quantity objects were unpickled pointing to a different UnitRegistry than the current one. @cmccoy please take a look.

+1

@cmccoy fixed the whitespace issue. PTAL.

LGTM.


M2 compiling issues

General information

Corrfunc version: 2.5.0
platform: M2 Mac, macOS Monterrey
installation method (pip/source/other?): pip, then git clone

Issue description
pip:
ERROR: Failed building wheel for Corrfunc
Running setup.py clean for Corrfunc
Failed to build Corrfunc
ERROR: Could not build wheels for Corrfunc, which is required to install pyproject.toml-based projects
GitHub directory results in the following error after "make install":
clang: error: the clang compiler does not support '-march=native'
What have you tried so far?
Changing the "-march=native" lines in common.mk to "-mpu-apple=a14", adding make CC=gcc, updating my gsl/gcc/python/pip packages - still I get the same error.
Minimal failing example
** could not install

make install

# issue

Makefile:9:  Not compiling C extensions for mocks. Either python or numpy not available
gcc  -DVERSION="2.5.0" -DUSE_UNICODE -std=c99 -m64 -g -Wsign-compare -Wall -Wextra -Wshadow -Wunused -fPIC -D_POSIX_SOURCE=200809L -D_GNU_SOURCE -D_DARWIN_C_SOURCE -O3  -funroll-loops -march=native -fno-strict-aliasing -Wformat=2  -Wpacked  -Wnested-externs -Wpointer-arith  -Wredundant-decls  -Wfloat-equal -Wcast-qual -Wcast-align -Wmissing-declarations -Wmissing-prototypes  -Wnested-externs -Wstrict-prototypes   -Wno-unused-local-typedefs  logbins.c  -lm -o logbins
clang: error: the clang compiler does not support '-march=native'
make[1]: *** [logbins] Error 1
make: *** [install] Error 2

Thanks for using Corrfunc and for reporting the issue. I have been meaning to add the compilation-on-m1/m2 fix from the arm64 branch into the master branch and this gives me the reminder to do so now.
Even after the compilation is fixed, you still will have another issue with the non-detection of python/numpy. Can you add a bit more detail about your python/numpy setup - are you using conda/macports/brew?

@quasars100 Will you please do the following to check if the new commits fix your compilation issue? The steps would be:
  1. git clone <repo>
  2. git checkout -b fix-compile-on-apple-soc
  3. make clean
  4. make

This should compile all of the command-line executables. If you encounter any warning/errors, please report back.

Even after the compilation is fixed, you still will have another issue with the non-detection of python/numpy. Can you add a bit more detail about your python/numpy setup - are you using conda/macports/brew?

Thank you so much for the quick reply.  My python version is Python 3.10.12 (installed using brew) and NumPy version is 1.24.3 (installed using pip).  Local environment (no virtualenv), terminal, packages installed using brew (no condo/macports).

@quasars100 Will you please do the following to check if the new commits fix your compilation issue? The steps would be:
  1. git clone <repo>
  2. git checkout -b fix-compile-on-apple-soc
  3. make clean
  4. make

This should compile all of the command-line executables. If you encounter any warning/errors, please report back.

I tried running the code block above and unfortunately got the following error:
In file included from ../../utils/cpu_features.c:13:
../../utils/cpu_features.h:41:10: error: couldn't allocate output register for constraint 'a'
__asm("cpuid" : "=a"(a),"=b"(b),"=c"(c),"=d"(d) : "a"(functionnumber),"c"(0) );
^
../../utils/cpu_features.h:41:10: error: couldn't allocate output register for constraint 'a'
../../utils/cpu_features.h:69:10: error: couldn't allocate output register for constraint 'a'
__asm("xgetbv" : "=a"(a),"=d"(d) : "c"(ctr) : );
^
../../utils/cpu_features.h:41:10: error: couldn't allocate output register for constraint 'a'
__asm("cpuid" : "=a"(a),"=b"(b),"=c"(c),"=d"(d) : "a"(functionnumber),"c"(0) );
^
../../utils/cpu_features.h:41:10: error: couldn't allocate output register for constraint 'a'
5 errors generated.
make[2]: *** [../../utils/cpu_features.o] Error 1
make[1]: *** [DD] Error 2
make: *** [theory] Error 2

That is odd - those lines of code should never be seen by the compiler if the environment variable __ARM_NEON is defined - which is supposed to be the case. For reference, I just did a fresh repo clone and tried out my instructions and those worked. I also notice that there is line number mis-match between what I see and what you have reported in the error - is it possible that you accidentally skipped the checkout branch line? There was an error in what I quoted - you should do git checkout fix-compile-on-apple-soc (i.e., without the -b )
If you were indeed in the correct branch, will you please report back on the following steps:

please copy-paste the output generated during the (unsuccessful) compilation process. Should look something like this:

clang -DOUTPUT_RPAVG   -DENABLE_MIN_SEP_OPT  -DCOPY_PARTICLES  -DLINK_IN_DEC -DLINK_IN_RA  -DDOUBLE_PREC -DUSE_OMP  -DVERSION=\"2.5.0\" -DUSE_UNICODE -std=c99 -m64 -g -Wsign-compare -Wall -Wextra -Wshadow -Wunused -fPIC -D_POSIX_SOURCE=200809L -D_GNU_SOURCE -D_DARWIN_C_SOURCE -O3  -fopenmp=libomp -funroll-loops -fno-strict-aliasing -Wformat=2  -Wpacked  -Wnested-externs -Wpointer-arith  -Wredundant-decls  -Wfloat-equal -Wcast-qual -Wcast-align -Wmissing-declarations -Wmissing-prototypes  -Wnested-externs -Wstrict-prototypes   -Wno-unused-local-typedefs  -I../../io -I../../utils -I/opt/local/include -I../../mocks/DDrppi_mocks -I../../mocks/DDsmu_mocks -I../../mocks/DDtheta_mocks -I../../mocks/vpf_mocks -c run_correlations_mocks.c -o run_correlations_mocks.o

And then, can you please report back on these two things at the command-line:

which <compiler> (where you replace the  with whichever compiler that you are using).
<compiler>  -mcpu=apple-m1 -dM -E - < /dev/null | grep -i ARM

Here is what the output is after the following steps.
In terminal, in the ~/Corrfunc directory:
(1. git clone ) --> already cloned and in the Corrfunc directory
2. git checkout fix-compile-on-apple-soc
3. make clean
4. make
Output (in terminal):
In file included from ../../utils/cpu_features.c:13:
../../utils/cpu_features.h:41:10: error: couldn't allocate output register for constraint 'a'
__asm("cpuid" : "=a"(a),"=b"(b),"=c"(c),"=d"(d) : "a"(functionnumber),"c"(0) );
^
../../utils/cpu_features.h:41:10: error: couldn't allocate output register for constraint 'a'
../../utils/cpu_features.h:69:10: error: couldn't allocate output register for constraint 'a'
__asm("xgetbv" : "=a"(a),"=d"(d) : "c"(ctr) : );
^
../../utils/cpu_features.h:41:10: error: couldn't allocate output register for constraint 'a'
__asm("cpuid" : "=a"(a),"=b"(b),"=c"(c),"=d"(d) : "a"(functionnumber),"c"(0) );
^
../../utils/cpu_features.h:41:10: error: couldn't allocate output register for constraint 'a'
5 errors generated.
make[2]: *** [../../utils/cpu_features.o] Error 1
make[1]: *** [DD] Error 2
make: *** [theory] Error 2
achen@MAC0008879 Corrfunc %

Just to confirm: there are no matches to the ARM grep command for either of the compilers?

Also, what happens if you do make CC=gcc (after a make clean)
What’s happening is that the compiler is not supposed to see that chunk of code on ARM, based on a variable usually defined by the compiler. That assumption of defined variable seems to break down on your setup. Since I don’t use brew (macports person) here, I hadn’t encountered this issue before

Since @quasars100 probably ran the original instructions with git checkout -b fix-compile-on-apple-soc, they probably have a local branch that's not tracking the remote. I think the steps to fix this are:
$ git checkout master
$ git branch -d fix-compile-on-apple-soc
$ git checkout fix-compile-on-apple-soc

I think at that point, you'll have a local branch that's tracking the remote, which you can confirm with git branch -vv or git status. Then hopefully the code will compile.

Also, what happens if you do make CC=gcc (after a make clean)
What’s happening is that the compiler is not supposed to see that chunk of code on ARM, based on a variable usually defined by the compiler. That assumption of defined variable seems to break down on your setup. Since I don’t use brew (macports person) here, I hadn’t encountered this issue before

This seems to be able to run, since the last few lines of output are:
ar rcs libcountspheres_mocks.a countspheres_mocks.o countspheres_mocks_impl_float.o countspheres_mocks_impl_double.o ../../utils/gridlink_impl_float.o ../../utils/gridlink_impl_double.o ../../utils/gridlink_utils_float.o ../../utils/gridlink_utils_double.o ../../utils/utils.o ../../utils/progressbar.o ../../utils/cpu_features.o ../../utils/avx512_calls.o ../../utils/set_cosmo_dist.o ../../utils/cosmology_params.o
make[2]: Nothing to be done for `lib'.
/Library/Developer/CommandLineTools/usr/bin/make -C examples
gcc -DOUTPUT_RPAVG   -DENABLE_MIN_SEP_OPT  -DCOPY_PARTICLES  -DLINK_IN_DEC -DLINK_IN_RA  -DDOUBLE_PREC -DUSE_OMP  -DVERSION="2.5.0" -DUSE_UNICODE -std=c99 -m64 -g -Wsign-compare -Wall -Wextra -Wshadow -Wunused -fPIC -D_POSIX_SOURCE=200809L -D_GNU_SOURCE -D_DARWIN_C_SOURCE -O3  -funroll-loops -fno-strict-aliasing -Wformat=2  -Wpacked  -Wnested-externs -Wpointer-arith  -Wredundant-decls  -Wfloat-equal -Wcast-qual -Wcast-align -Wmissing-declarations -Wmissing-prototypes  -Wnested-externs -Wstrict-prototypes   -Wno-unused-local-typedefs  -I../../io -I../../utils -I/opt/homebrew/Cellar/gsl/2.7.1/include -I../../mocks/DDrppi_mocks -I../../mocks/DDsmu_mocks -I../../mocks/DDtheta_mocks -I../../mocks/vpf_mocks -c run_correlations_mocks.c -o run_correlations_mocks.o
gcc run_correlations_mocks.o ../../io/ftread.o ../../io/io.o ../../utils/utils.o ../../utils/progressbar.o ../../utils/cosmology_params.o ../../mocks/DDrppi_mocks/libcountpairs_rp_pi_mocks.a ../../mocks/DDsmu_mocks/libcountpairs_s_mu_mocks.a ../../mocks/DDtheta_mocks/libcountpairs_theta_mocks.a ../../mocks/vpf_mocks/libcountspheres_mocks.a   -lm -L/opt/homebrew/Cellar/gsl/2.7.1/lib -lgsl -lgslcblas -Xlinker -rpath -Xlinker /opt/homebrew/Cellar/gsl/2.7.1/lib -o run_correlations_mocks
achen@MAC0008879 Corrfunc %
However, in the middle, there was the following warning:
If you want to permanently set the default compiler to gcc for all future compilations, please update the "CC" variable in "common.mk"
../common.mk:276: Compiler is Apple clang and does not support OpenMP
If you want OpenMP support, please install clang with OpenMP support
For homebrew, use "brew update && (brew outdated xctool || brew upgrade xctool) && brew tap homebrew/versions && brew install clang-omp"
For Macports, use "sudo port install clang-3.8 +assertions +debug + openmp"
../common.mk:320:  $CC = gcc does not support OpenMP - please use gcc/icc for compiling with openmp. Removing "USE_OMP" from compile options.
If you are sure your version of "clang" ( must be >= 3.7, NOT Apple clang) does support OpenMP, then comment out the line  "OPT:=$(filter-out -DUSE_OMP,$(OPT))"  in the file "common.mk"
You might have to add in the include path (path to "omp.h") to "CFLAGS" and the runtime library path to "CLINK" at the top of "common.mk"
/bin/sh: python: command not found
/bin/sh: python: command not found
/bin/sh: line 0: [: too many arguments
/bin/sh: line 0: [: too many arguments
../common.mk:497:  Found python version .. but minimum required python is 2.7
../common.mk:502:  Found NUMPY version .. but minimum required numpy is 1.7
-------COMPILE SETTINGS------------
MAKE		 = ["/Library/Developer/CommandLineTools/usr/bin/make"]
CC		 = ["gcc"]
OPT		 = ["-DOUTPUT_RPAVG -DENABLE_MIN_SEP_OPT -DCOPY_PARTICLES -DLINK_IN_DEC -DLINK_IN_RA -DDOUBLE_PREC"]
CFLAGS		 = ["-DVERSION="2.5.0" -DUSE_UNICODE -std=c99 -m64 -g -Wsign-compare -Wall -Wextra -Wshadow -Wunused -fPIC -D_POSIX_SOURCE=200809L -D_GNU_SOURCE -D_DARWIN_C_SOURCE -O3 -funroll-loops -fno-strict-aliasing -Wformat=2 -Wpacked -Wnested-externs -Wpointer-arith -Wredundant-decls -Wfloat-equal -Wcast-qual -Wcast-align -Wmissing-declarations -Wmissing-prototypes -Wnested-externs -Wstrict-prototypes -Wno-unused-local-typedefs -mcpu=apple-m1 -mtune=apple-m1"]
CLINK		 = [" -lm"]
PYTHON		 = ["python"]
GSL_CFLAGS	 = [""]
GSL_LINK	 = [""]
PYTHON_CFLAGS	 = [""]
-------END OF COMPILE SETTINGS------------
Makefile:7:  Not compiling C extensions for mocks. Either python or numpy not available

However, when I check my python version, it was 3.10, which is > python2.7
achen@MAC0008879 Corrfunc % which python
python: aliased to /opt/homebrew/bin/python3.10
And numpy:
Python 3.10.12 (main, Jun  7 2023, 00:37:42) [Clang 14.0.0 (clang-14<IP_ADDRESS>)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

import numpy
numpy.version
'1.24.3'

which is >numpy 1.7.
Sorry for how long this issue seems to be, and thank you all again for all your help.

Since @quasars100 probably ran the original instructions with git checkout -b fix-compile-on-apple-soc, they probably have a local branch that's not tracking the remote. I think the steps to fix this are:
$ git checkout master
$ git branch -d fix-compile-on-apple-soc
$ git checkout fix-compile-on-apple-soc

I think at that point, you'll have a local branch that's tracking the remote, which you can confirm with git branch -vv or git status. Then hopefully the code will compile.

Thank you, and after running these steps, the code did seem to compile, however, this was the end output:
Makefile:7:  Not compiling C extensions for mocks. Either python or numpy not available
/Library/Developer/CommandLineTools/usr/bin/make -C DDrppi_mocks
make[2]: Nothing to be done for all'. /Library/Developer/CommandLineTools/usr/bin/make -C DDtheta_mocks make[2]: Nothing to be done for all'.
/Library/Developer/CommandLineTools/usr/bin/make -C DDsmu_mocks
make[2]: Nothing to be done for all'. /Library/Developer/CommandLineTools/usr/bin/make -C vpf_mocks make[2]: Nothing to be done for all'.
for dir in DDrppi_mocks DDtheta_mocks DDsmu_mocks vpf_mocks examples; do
/Library/Developer/CommandLineTools/usr/bin/make -C $dir lib;
done
make[2]: Nothing to be done for lib'. make[2]: Nothing to be done for lib'.
make[2]: Nothing to be done for lib'. make[2]: Nothing to be done for lib'.
make[2]: Nothing to be done for lib'. /Library/Developer/CommandLineTools/usr/bin/make -C examples make[2]: Nothing to be done for all'.
achen@MAC0008879 Corrfunc %
Also, when I try calling "Corrfunc" in python and using its commands, I get the following error:

pairs = xi(L, 2, bins, pos[:,0], pos[:,1], pos[:,2])
Traceback (most recent call last):
File "/Users/achen/Corrfunc/Corrfunc/theory/xi.py", line 187, in xi
from Corrfunc._countpairs import countpairs_xi as xi_extn
ModuleNotFoundError: No module named 'Corrfunc._countpairs'

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "", line 1, in
File "/Users/achen/Corrfunc/Corrfunc/theory/xi.py", line 191, in xi
raise ImportError(msg)
ImportError: Could not import the C extension for the correlation function.

The package does import in python without issue, it's just when I try running the commands they don't work.
Thanks again, and sorry for the myriad of issues this seems to have on brew.

Hoorrray - that's definitely progress! At least the code seems to be compiling (which I am surprised by, given the lack of the (usually) defined ARM variables but I will put that down to a bad grep implementation).
@quasars100 Please don't apologize - you are helping us improve Corrfunc, and that, in turn, helps everyone - so thank you!
@lgarrison Not really sure why python is not being detected - the detection is simply a shell calling python! I see that the output says  /bin/sh: python: command not found. My guess is that python is defined as an alias and, therefore, only available on the interactive terminal (i.e., where @quasars100 is working but not through the non-interactive terminal invoked by make).
@quasars100 What do you get if you try alias python? And regardless of the alias status, using python -m pip install -e .  --verbose should work. Even if python is aliased, that specific command should use the correct python.
Related, I see that OpenMP is not enabled; ideally, that also needs to be fixed. @quasars100 Will you please run brew install llvm to install the OpenMP-enabled version of clang and then recompile (after make clean) by invoking make (or equivalently, make CC=clang)

@quasars100 To explicitly clarify, as long as python and numpy is not detected at compile time, the Corrfunc C extensions will not be compiled, and therefore, you will not be able to use Corrfunc from python

Hoorrray - that's definitely progress! At least the code seems to be compiling (which I am surprised by, given the lack of the (usually) defined ARM variables but I will put that down to a bad grep implementation). @quasars100 Please don't apologize - you are helping us improve Corrfunc, and that, in turn, helps everyone - so thank you!
@lgarrison Not really sure why python is not being detected - the detection is simply a shell calling python! I see that the output says  /bin/sh: python: command not found. My guess is that python is defined as an alias and, therefore, only available on the interactive terminal (i.e., where @quasars100 is working but not through the non-interactive terminal invoked by make). @quasars100 What do you get if you try alias python? And regardless of the alias status, using python -m pip install -e . --verbose should work. Even if python is aliased, that specific command should use the correct python.

Here is what I get when running "alias python" in terminal:
alias python
python=/opt/homebrew/bin/python3.10
(which python
python: aliased to /opt/homebrew/bin/python3.10 )

Related, I see that OpenMP is not enabled; ideally, that also needs to be fixed. @quasars100 Will you please run brew install llvm to install the OpenMP-enabled version of clang and then recompile (after make clean) by invoking make (or equivalently, make CC=clang)

So prior to opening this issue, I actually ran "brew install llvm", and that is the version of clang my current clang compiler is aliased to, which is why the warning of "OpenMP is not available with apple clang" was confusing, since clang should be aliased away from Apple clang now.
After running "make CC=clang", OpenMP is enabled, but I still get this error:
/bin/sh: python: command not found
/bin/sh: python: command not found
/bin/sh: line 0: [: too many arguments
/bin/sh: line 0: [: too many arguments
../common.mk:497:  Found python version .. but minimum required python is 2.7
../common.mk:502:  Found NUMPY version .. but minimum required numpy is 1.7
However when I check the python version:
which python
python: aliased to /opt/homebrew/bin/python3.10
achen@MAC0008879 Corrfunc % python
Python 3.10.12 (main, Jun  7 2023, 00:37:42) [Clang 14.0.0 (clang-14<IP_ADDRESS>)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

import numpy
numpy.version
'1.24.3'

Python > 2.7 and numpy > 1.7.  Do you know if this is something to be changed in "common.mk" or my local environment?
Thank you again so much for all your help!

Ohh good - so python is defined as an alias. No wonder make can't find python.
Try and run export CC=clang && python -m pip install -e . - that should do the trick

Yes, that worked! (Has to be in the Corrfunc cloned directory though).
Thank you so much!

Great - glad this all worked :)! Can you confirm that you have OpenMP support, i.e., that you can use multiple threads now?
Sidenote: The reason why you were facing the compilation and non-detection of llvm-clang/python/numpy is all because you (or homebrew) have the relevant aliases set. The aliases work fine for you since you are always accessing from an interactive terminal - i.e., a terminal where you can send keyboard input to. The shell sessions spawned by common.mk are non-interactive and, therefore, do not know about the aliases.
I also suspect your PATH variable does not contain /opt/homebrew/bin (for llvm-clang) ahead of /usr/bin (for apple-clang), and thus, when the non-interactive shell resolves clang as /usr/bin/clang. Since I don't use homebrew, I am unsure if it is an intentional design that /opt/homebrew/bin is not added to your PATH environment variable. If you ever run into such odd issues of not-detecting-the-correct-tool, then it might be worthwhile adding the full path to whichever tool that you are wanting to detect (or equivalently, add /opt/homebrew/bin to PATH either in the terminal you are working in, or in the startup shell file - .bashrc/.zshrc etc)


Error running extract_lora_from_models.py

Good day.
I ran into a strange issue running extract_lora_from_models.py, it seems to be unable to calculate the safetensors hashes:
calculating by svd
100%|█████████████████████████████████████████████████| 264/264 [00:44<00:00,  5.94it/s]
enable LoRA for text encoder
enable LoRA for U-Net
LoRA has 792 weights.
Loading extracted LoRA weights: <All keys matched successfully>
Traceback (most recent call last):
  File "/mnt/DATA/AI/sd-scripts/networks/extract_lora_from_models.py", line 164, in <module>
    svd(args)
  File "/mnt/DATA/AI/sd-scripts/networks/extract_lora_from_models.py", line 144, in svd
    lora_network_o.save_weights(args.save_to, save_dtype, metadata)
  File "/mnt/DATA/AI/sd-scripts/networks/lora.py", line 231, in save_weights
    model_hash, legacy_hash = train_util.precalculate_safetensors_hashes(state_dict, metadata)
AttributeError: module 'library.train_util' has no attribute 'precalculate_safetensors_hashes'

This works when saving as ckpt, but safetensors fails for some reason. I've verified library.train_util.precalculate_safetensors_hashes exists from a python console:
>>> train_util.precalculate_safetensors_hashes
<function precalculate_safetensors_hashes at 0x7f7a14a24700>

so I don't know why it fails, everything looks fine.
It worked with a safetensors output on January 18th.

That fixed it, thank you very much!
I am not sure what could have caused it, this local clone is the same I do training into without errors, and I did successfully output to safetensors with this script before.
Regardless, it is fixed so if it happens again I know what to do. Thanks a lot! I'll close the issue now.


add jsx linting

Add jsx linting, so the linter also works for *.jsx files.

Oww, nice catch thank you, going to merge it and release it


CUDA_ERROR_OUT_OF_MEMORY AND UNABLE TO LOAD CUDA

Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-07-11 12:42:11.226602: W tensorflow/compiler/xla/service/platform_util.cc:240] unable to create StreamExecutor for CUDA:0: failed initializing StreamExecutor for CUDA device ordinal 0: Internal: failed call to cuDevicePrimaryCtxRetain: CUDA_ERROR_OUT_OF_MEMORY: out of memory; total memory reported:<PHONE_NUMBER>6
2019-07-11 12:42:11.399046: F tensorflow/stream_executor/lib/statusor.cc:34] Attempting to fetch value instead of handling error Internal: no supported devices found for platform CUDA
Aborted (core dumped)

Hi,
This is an error that usually shows when the GPU resources are already locked by another process. Do you by any chance have two parallel Python instances in which Tensorflow (or PyTorch, etc.) is being used?
If so, you can retain one of those instances from using the GPU by first exporting the environment variable CUDA_VISIBLE_DEVICES to -1 (or 1 if you have multiple GPUs and want to use the second one, etc.).
If that does not answer your issue, could you please give some more information as to the system you are using? (OS, TF version, etc.)

yes!! thanks that could be a reason as multiple programs are executing on server by different members!!

 I am not a sysadmin, but I believe in such cases a good practice can be to have users queue their jobs that require access to the GPU, and automatically prevent access to it in other cases. At any rate, if you want to use tensorflow without accessing the GPU, export CUDA_VISIBLE_DEVICES=-1 before launching python and you should be fine :)

Please provide details about what platform you are using (operating system, architecture). Also include your TensorFlow version. Also, did you compile from source or install a binary?
Make sure you also include the exact command if possible to produce the output included in your test case. If you are unclear what to include see the issue template displayed in the Github new issue template.
We ask for this in the issue submission template, because it is really difficult to help without that information. Thanks!

Automatically closing due to lack of recent activity. Please update the issue when new information becomes available, and we will reopen the issue. Thanks!

did anyone find the solution to this problem?
Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-08-01 14:26:33.107400: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency:<PHONE_NUMBER> Hz
2019-08-01 14:26:33.107815: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x55714b51c9f0 executing computations on platform Host. Devices:
2019-08-01 14:26:33.107827: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): ,
2019-08-01 14:26:33.215296: W tensorflow/compiler/xla/service/platform_util.cc:240] unable to create StreamExecutor for CUDA:0: failed initializing StreamExecutor for CUDA device ordinal 0: Internal: failed call to cuDevicePrimaryCtxRetain: CUDA_ERROR_OUT_OF_MEMORY: out of memory; total memory reported:<PHONE_NUMBER>
2019-08-01 14:26:33.215397: F tensorflow/stream_executor/lib/statusor.cc:34] Attempting to fetch value instead of handling error Internal: no supported devices found for platform CUDA
Aborted (core dumped)

@anil-bit Is it possible that, as was the case for the author of this issue, you already have an instance of python / tensorflow opened that reserves the entire GPU?
On a unix system you can check which programs take memory on your GPU using the nvidia-smi command in a terminal.
To disable the use of GPUs by tensorflow (which should be a workaround), you can run (in 2.0) tf.config.set_visible_devices([], 'GPU') ; I do not have the 1.14 equivalent in mind at the moment but you can easily look it up.

@anil-bit I think you were running some program over GPU and if you have forcefully stooped it and then you are again executing it over the same GPU then this problem arises as the GPU memory was already locked by the previous program. To solve this issue terminate the terminal and again open the terminal and execute the code. Hope this helps!!


TypeError dndSpellData.components

Im getting the following error, if i try to export anything:

from

Do you have any idea, how i can fix this?

Hello,
What is your foundry and dnd 5e versions ?

Foundry is Version 11 Build 315 and Dnd is 3.2.1

Yes it seems the object dndSpellData.system.components has become dndSpellData.system.properties in the recent dnd 5e (was working on 3.1).
I will do a fix for this.
If you want a quick fix, you could add this code before the error line :
dndSpellData.system.components = dndSpellData.system.properties.reduce((acc,propertiy) => { acc[propertiy] = true return acc} ,{})

Fix done in 2.5.1


[EPIC] Restructure the flutter_templates

Why
Currently, the flutter_templates repository is a mix between the template and Flutter project. It could confuse a new-dev (ex. why do we have Makefile inside a Flutter project?), and it's hard to maintain when the template stuff grows.
As discussed in the Flutter-guild meeting, we came to an agreement to re-structure the flutter_templates with the following:

Create a .template directory and move Flutter-related stuff into this dir.
Keep template-stuff (e.g. scripts/, Makefile, etc) at root.
For optional features, we may store them in .features directory and add them to the generated project via the scripts. (need more info later 🐕🦺 )
The team decided not to create an example project to present our MVVM structure but implement it directly to our .template. WHY?

When creating the example project, we have to implement the features in more detail (List screen, detail screen). It requires more effort but brings less value for us.
When initializing a project, we don't need to manually copy & paste repository, service, use case, etc. We may also need to update the repository name or service name that matches our project context via script.

Please let me know if I'm missing something. I welcome comments for improvement 🙇
More tasks will come for this epic 🚀
Who Benefits?
Developers, maintainers.

@manh-t I'm a bit concerned with the last idea:

The team decided not to create an example project to present our MVVM structure but implement it directly to our .project. WHY?

The main concern to me is NOT about enforcing us to have it, but rather the later phase that might make us fall in the same trap as before:

present our MVVM structure but implement it directly

There are a few things we need to reconsider:

If we don't want to have example/ - fine, but then we shouldn't implement a concrete implementation at the root level because it might not match with our expectation later on. Let's say you want to switch to the VIPER pattern, we can't make it happen unless developer has to:

Init the project with MVVM
Remove the ViewModel
Adding Router...
=> As you can see, that is cumbersome.

My suggestion is that we should keep the root clean, only just to have the implementation ready at the:

Dependencies declaration (flutter version, lib version...)
Networking or HTTP service ready to use.
Logging, crashlytics, utility lib setup...
=> Anything above that (repository, use case) should be added by the developers who init a project from the template. The other way is that we should add mvvm as a feature option (so later you can add viper if anyone wants to support it). This can ensure the project will be extensible when technology evolves.

Now back to the example:

we have to implement the features in more detail (List screen, detail screen).

First, we should NOT implement this in the root project either. Remember that it might cost us removal effort later when we don't want to have them.
So what option do we have to demonstrate the templated project? Imo, that's why I suggested we have multiple examples inside example/. If an example is outdated, or when we add a new major feature, instead of changing it we can have another examples directory:
---examples/
|--mvvm/
|--viper/
|--xyz/

If you are still unsure about it, I recommend NOT taking the example/ idea into it yet, but rather, just cleaning up the project and getting everything ready at first with architecture as a feature option (mvvm or default == None).
@markgravity @doannimble @luongvo please feel free to debate 🙇

@sleepylee your concern is valid 🤗 we already discussed the structures used in flutter_templates too, then it comes to challenges which are:

If the mvvm has a piece of code that need to append/ modify an existing file (e.g., change network_module.dart to add service class into this), we need to build a script (or find a third-party lib) that can update the existing file properly (maybe the file will be maintained to fix or add more code, the script is still able to edit it).
How to verify the code we added into mvvm is correct via CI/CD.
So we fall back to a more straightforward option, as mentioned in the description above 🙇

If you are still unsure about it, I recommend NOT taking the example/ idea into it yet, but rather, just cleaning up the project and getting everything ready at first with architecture as a feature option (mvvm or default == None).

Okay, I will not take the example idea into the project until the idea becomes possible to me 🙇 perhaps I'll create a PoW first 😁

If the mvvm has a piece of code that need to append/ modify an existing file (e.g., change network_module.dart to add service class into this), we need to build a script (or find a third-party lib) that can update the existing file properly

I think if the change is small, we should just update it to the current example repo.

How to verify the code we added into mvvm is correct via CI/CD.

If you do it manually as above, then you can verify it. The idea is that CI/CD should only verify a project init from the template but NOT committed to the vcs.
So, what happens if there are more updates that you can't bring to the example repo we already had?
👉  create a new example from the latest state of the template. Adding the same small demo features that can ensure you will have the latest example code.

@luongvo I think we can close this epic, since the project was re-organized in 1.0.0. What do you think?

@manh-t I'm planning a short flutter-templates meeting every 2 weeks or 1 month to discuss the plan & backlog. I think we can have decisions from that meeting, I will send you guys the invitation for the first one soon 👋


remove partial ignored directory

Hello.
I think plugin do not undestand ignore paths with subdirectory.
For example:
in "jquery", plugin remove all "dist" directory instead of "dist/cdn"

For jquery, only stable versions have a dist directory.

I try to use make composer-mirror throw satis. In satis config I use "bower-asset/jquery": ">=1.7@stable", but the result package have not dist files only for jquery. for example "bower-asset/bootstrap" has dist directory.

composer/cache has package with dist directory but installed package has not

The differnce between bootstrap and jquery, is that the Bootstrap branches contain the distdirectory, whereas this is not the case for jquery. Tags only contain this directory... (it is the choice of the developers).

Jquery Branch Master
Jquery Tag 2.1.4

composer/cache?

~/.composer/cache/files/bower-asset/jquery/7751e69b615c6eca6f783a81e292a55725af6b85.zip - it a cache for jquery 2.1.4

Ok, I thought of a package name ... Strange, I use jquery with my composer plugin, and I don't have this bug.
Do you used the last stable version of my plugin (1.0.2) and last dev version of Composer?

You always have the ability to disable or change the configuration for removing the ignored files of Bower, see the doc Disable or replace the deleting of the ignore files for Bower.

I find out why the dist was removed. It is not a bug of you plugin.
thanks for your attention.


Deduplicate code for simple wiring

As @ychen-atlassian suggested, extracted repetitive code for simple wiring.

I'm okay with leaving the code as is. I literally just was thinking about having svccatentangler except as a util method rather than as a struct that you embed to get WireUp (which imo was the confusing part)


M3-1489 deploy image to disk (#4206)

Note to Reviewers
Includes M3-1489 (deploy disk from image) and ...looks like that's it. Final review before merging to develop.

Coverage increased (+0.03%) to 31.94% when pulling c2c2a59a5c92ca49a13732198206993a16815e9d on feature/deploy-disk-from-image into 85b542ce54cf89842c262b66d89f79fbd417d119 on develop.


Textfield text

Hello, How do I change the hintText of the textField, I'm Brazilian and I need to change the text from 'search' to 'pesquisar'

Hey @joaothimotheo, this can't be possible in current version of the package. But I will be releasing the next version soon with a lot of new customisation options including this!

Now available in 2.0.0. Thanks :+1:


Remove getSnippet inline GLSL and move code where necessary to shader chunks

The legacy 'getSnippet' to get GLSL snippets is still in the codebase. This should now be removed now that we have the shader chunk system.

Removed in https://github.com/playcanvas/engine/pull/636


Remove the 2.1. templates from the rel/2.0.x branch

The templates will stay in the dev branch.
FYI - I'll do a non-merging merge after this goes in so we can keep rel/2.0.x work on the 2.0 templates in sync with the dev branch.

@natemcmaster,
Thanks for having already signed the Contribution License Agreement. Your agreement was validated by .NET Foundation. We will now review your pull request.
Thanks,
.NET Foundation Pull Request Bot

It's not technically necessary. It's more about keeping branches clean. I figured it would be better to keep the rel/2.0.x branch free of any netcoreapp2.1 stuff.


Evaluation results

After training for 200 epochs with the Chinese datasets, I got the following results:

The result here is that I just used the Chinese datasets for training, but did not retrain CPC. Maybe this is the reason why the results are very different from the results in your paper?
But if I remember correctly, I saw in CPC-audio's paper that the author mentioned that the model can be transferred to other languages and perform well?

Hi
I wonder if you re-trained the vocoder part.
The CPC's author said it transfers well, but fine-tuning is better for adaptation.
If you have re-trained the vocoder, you can check using mel-spectrogram features instead of using CPC.

Yes, I'm currently retraining the vocoder, except for some parameters like sample rate etc. I follow the source code settings.
Do you mean that if I want to perform well on Chinese or other datasets, I can use the following two methods?

Use the new datasets to fine-tuning CPC, and use the new datasets to train TriAANVC and vocoder;
The CPC does not make any changes, the input of TriAANVC uses the mel-spectrogram of the new datasets, and the vocoder is re-trained using the new datasets.

Thank you very much for taking the time to reply me, it helps me a lot.

Yes.
If you want to use another dataset (especially different languages), you need to re-train TriAAN-VC and vocoder.
Of course, it might be better to fine-tune CPC, but if it is difficult, I recommend using mel-spectrogram as inputs for TriAAN-VC.


Wrong version number comment in release notes

There is a new release of Blazor, the 0.4.0. But I think there is an error in the release notes.
It is written: We are pleased to announce the release of Blazor 0.3.0. It should be 0.40.

Whoops! Should be fixed now. Thx!


accessibility in facility/node

Coverage increased (+0.002%) to 83.235% when pulling 1140147c03aea52b6a97def62faf91fdbd2f2792 on ShinjiTanimoto:facility-node into 6ec96025d2fda7875385a6380130dbd53b2144d5 on shirasagi:master.

Coverage increased (+0.2%) to 83.473% when pulling ea671e33485538f098e968dc6bf0864446951a53 on ShinjiTanimoto:facility-node into 6ec96025d2fda7875385a6380130dbd53b2144d5 on shirasagi:master.


Verra VM0044 Webinar

Problem description
Verra’s VM0044: Methodology for Biochar Utilization in Soil and Non-Soil Applications, v1.0 measures the removal of carbon through the use of biochar resulting from improved waste handling and disposal practices. These practices involve converting waste biomass into biochar at new biochar production facilities. The greenhouse gas (GHG) benefits are only credited for biochar that is used in eligible soil and non-soil applications, such as crop- and grasslands, as well as emerging non-soil products like biochar-amended concrete and building materials. This methodology can be applied worldwide and utilizes a standardized activity method to determine additional benefits.
In this webinar we will go over the VM0044 methodology, policy creation, and have a demo of the Guardian.
Requirements
Design schemas for the VM0044 methodology, create a PowerPoint presentation, and conduct webinar.
Definition of done
Designing Schema
Development of Schema
Updating ReadMe and documentation
Acceptance criteria
100% completion on design, development and documentation

@carbontarzan ^


[bug] http fetch fails to send form with correct image encoding

Describe the bug
If you call the http form function to produce a Body that includes both a Byte array file the correct mime type isn't used and what gets uploaded is a unicode text file instead of an image.
Example:
Reproduction
Call the fetch api with a body created by Body.form(...) with an end result similar to:
{someKey: {file: byteArray, mime: 'image/png', fileName: 'aFile.png'}

Where "someKey" is what your server expects to be the key in the form containing the file and byteArray is an array of numbers, each number between (-128, 127).
An example byte array (in plain text) is attached (for the Tauri logo):

The file will be correctly uploaded, however it will simply be a text file, not binary (image) if in linux you run the command file -i img.png it will show a mime type of plain text.
As a control simply perform the upload specifying an absolute path to the image file:
{someKey: {file: 'path/to/file', mime: 'image/png', fileName: 'filename.png'}}

And this works great! So we know it is only an issue when supplying a byte array as file
Expected behavior
The file should have mime type of image/png
Platform and versions
Linux (Ubuntu 20.04, Centos 7, Windows 10)

[build-dependencies]
tauri-build = { version = "1.0.0-rc.12", features = [] }

[dependencies]
tauri = { version = "1.0.0-rc.14", features = ["fs-read-file", "fs-write-file", "http-all", "http-multipart", "shell-open"] }

Stack trace
No response
Additional context
I tried adding reqwest-client but that resulted in it completely not uploading at all with my server rejecting:

org.apache.commons.fileupload.FileUploadException: the request was rejected because no multipart boundary was found

Note: an example server to test this aganst is Jira. Run, setup a project, create an issue:
https://hub.docker.com/r/atlassian/jira-software
Then post an attachment to an issue:
https://docs.atlassian.com/software/jira/docs/api/REST/8.22.3/#issue/{issueIdOrKey}/attachments-addAttachment

Can you try changing the content-type header to multipart/form-data and enabling the http-multipart Cargo feature?

@lucasfernog sorry I'm not misunderstanding, don't I already show the http-multipart feature flag above? Or is there another step necessary to enable besides adding it to cargo.toml?
Also, I add the following headers to the request (in addition to auth):

"content-type" -> "multipart/form-data", "X-Atlassian-Token" -> "no-check", "accept" -> "application/json", "user-agent" -> "custom-agent"

Keep in mind I show that this works with specifying a path for file, it;s just specifying a byte array when it fails to encode properly. I would use the path if I could, but browser file pickers intentionally do not share file path for security reasons to javascript.

ahh you have the feature, sorry didn't see that. I'll investigate this issue, thanks for the report.

@evbo I just tried it and it work both with a file path and a byte array. Here's my code:
const file = await readBinaryFile('/path/to/image.png')

result = await fetch('http://localhost:8080/rest/api/2/issue/TEST-25/attachments', {
  method: 'POST',
  body: Body.form({
    'file': {
      file, mime: 'image/png', fileName: 'image.png'
    }
  }),
  headers: { 'Content-Type': 'multipart/form-data', Authorization: 'Basic bHVjYXNmZXJub2c6MDQwODk2NTA0MjI=', 'X-Atlassian-Token': 'no-check' }
})

Note that if you use each number between (-128, 127) as you said it'll end up being a weird attachment in Jira anyway since that's not a valid image byte array.
Also, the multipart logic is the same when using a file path or a byte array (internally we just read the file and get its bytes to send), so both ways should work. If you want me to try something else, please share some code with us.

Actually there's an issue when using file as a path, i'll push a change to fix it.

Also found the issue for the reqwest usage, thanks for catching that :)

Thanks @lucasfernog and @FabianLars I'm glad I provided some assistance in the end!
Now I wish I read this sooner! Here's the proper way to encode files as byte arrays using FileReader in JS:
https://stackoverflow.com/a/31435538/1080804
There's a lot of examples online instructing to read files as URLs or Text - that's only useful for HTML tags like <img> to reference as their src, etc where they need plaintext representations. Never try to re-encode binary files starting from text (lesson learned!)