Missing genotype, PC, kinship data in MVP.Data output from .vcf

I'm having trouble with my data in MVP that leads to outputs lacking all genotype information per individuals in the GWAS population.
My SNP data was in .tped/.tfam, so I converted it to .vcf with PLINK before loading this data into MVP.Data. Here's the first 10 items from the last line of my .vcf input:
scaffold_724	20159	724_20159	2	1	.	.	PR	GT	0/1
I used this command:
MVP.Data(fileVCF="my_prefix.vcf",
         # used grep in bash and found 1529 header lines with ##, confirmed with head (also bash)
         vcf.jump=1529,
         sep.vcf="\t",
         filePhe="my_pheno_data.txt",
         fileKin=TRUE,
         filePC=TRUE,
         out="my_out")

Everything appeared to be working fine, until I realized when the processing was nearly finished that the output files were only about 200MB combined (and my .vcf input is 28GB). Then, this happened:
... [1] "Number of Markers Written into File: 1: 8252000" [1] "Number of Markers Written into File: 1: 8252993" [1] "Preparation for numeric data is done!" [1] "Preparation for PHENOTYPE data is done!" [1] "Output mvp genotype..." [1] "Calculate KINSHIP using Vanraden method..." [1] "Preparation for Kinship matrix is done!" [1] "Principal Component Analysis Start ..."  means for first 10 snps:  [1] 0 0 0 0 0 0 0 0 0 0 [1] "Preparation for PC matrix is done!" [1] "MVP data prepration accomplished successfully!" Warning message: In big.PCA(bigMat = big.geno, pcs.to.keep = pcs.keep) :   selected too many PCs to keep [5], changing to 2
> Covariates <- attach.big.matrix("my_prefix.pc.desc")
> head(Covariates)
     [,1] [,2]
[1,]    1    0
[2,]    0    1
> head(Kinship)
     [,1] [,2]
[1,]    0    0
[2,]    0    0
--

The genotype file (.map) contains only 3 columns and doesn't show the genotypes for each individual, just the Chr, SNP position and SNP ID.
Thanks for making this tool and for the help!

Hi naglemi,
Thank you for using MVP. Please use dim() function in R to show me the dimension of the data generated by MVP.Data.
Please attention that .map is SNP information file, and the genotype file is suffixed with ".desc" and ".bin".
Best,
Xiaolei

Thanks for taking a look.

genotype <- attach.big.matrix("mvp.917geno_a4_fromvcf.geno.desc")
dim(genotype)
[1] 8252993       2
phenotype <- read.table("mvp.917geno_a4_fromvcf.phe",head=TRUE)
dim(phenotype)
[1] 917  15
map <- read.table("mvp.917geno_a4_fromvcf.map" , head = TRUE)
dim(map)
[1] 8252993       3
Kinship <- attach.big.matrix("mvp.917geno_a4_fromvcf.kin.desc")
dim(Kinship)
[1] 2 2
Covariates <- attach.big.matrix("mvp.917geno_a4_fromvcf.pc.desc")
dim(Covariates)
[1] 2 2

I get a different issue when loading my data as .bed/.bim... I get an error saying SNP IDs must be unique (and they are). I'll check everything over, try again and post a separate issue if that still doesn't work for me.

Please double check "vcf.jump" and "sep.vcf", and also check if the vcf file you used have the same format as displayed in the user manual

I made sure there are 1529 header lines, so vcf.jump has the correct setting, right?
> grep -i "##" all_emmax_format_12_no_dup_real.vcf | wc -l
1529

The file is tab-seperated. Below are the first 10 columns of lines 1530 and 1531, which appear to match the format in the user manual.
#CHROM  POS     ID      REF     ALT     QUAL    FILTER  INFO    FORMAT  my_genotype_1
1       52      01_52   2       1       .       .       PR      GT      0/0
Perhaps the issue is a results of my file size (28GB) and limited memory available (64GB)? This seems to be causing an issue when working with my .bed file, which I'm about to make a separate issue post for.
Thanks for the help

It is not a memory issue. The big vcf file would be read into memory simultaneously. Could you please send me a small vcf file for the test. I am curious that why the top 1529 lines are all annotations.

I sent the top 1600 lines of the vcf. As I mentioned in my message, there are many unassembled contigs, and some of these are as characters instead of numeric (as was necessary to prepare the vcf with PLINK due to limit of numeric contigs allowed)... Maybe the first thing I should do is make them all numeric?

Hi Michael Nagle,
Thank you for sharing the demo data, and you are right that there are 1529 header lines. I run following codes and it works:
MVP.Data(fileVCF="nagle_poplar_SNPs_head1600.vcf",
#filePhe="Phenotype.txt",
vcf.jump=1529,
sep.vcf="\t",
#sep.phe="\t",
fileKin=FALSE,
filePC=FALSE,
out="mvp.vcf",
#maxLine=10000
)
But I find a problem in the data, the last header line, which starts with "#", there are 918 sample IDs while there are only 917 samples on the genotypic data lines (0/0,0/1,1/1 line).
Best,
Xiaolei

Hi Xiaolei,
Thanks again for looking at the data. I don’t know why an extra genotype wound up in the .vcf, but I don’t see one in my .tped or .bim files. For now I’ll work with the PLINK binaries.
I added memory to the virtual machine we’re working on, so we’re at 96GB now. When working with plink binaries, however, I’m still getting this error:
[1] "Principal Component Analysis Start ..."
Error: cannot allocate vector of size 56.4 Gb
Please let me know if you have other suggestions. Since everything except the PCA seems to be working for me, maybe I should try using PCAs from PLINK  and importing those into R to run MVP with?
Thanks,
Michael Nagle
PhD student, Molecular and Cellular Biology
From: Xiaolei Liu<EMAIL_ADDRESS>Reply-To: XiaoleiLiuBio/MVP<EMAIL_ADDRESS>Date: Tuesday, June 26, 2018 at 2:13 AM
To: XiaoleiLiuBio/MVP<EMAIL_ADDRESS>Cc: Nagle, Michael F<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [XiaoleiLiuBio/MVP] Missing genotype, PC, kinship data in MVP.Data output from .vcf (#21)
Hi Michael Nagle,
Thank you for sharing the demo data, and you are right that there are 1529 header lines. I run following codes and it works:
MVP.Data(fileVCF="nagle_poplar_SNPs_head1600.vcf",
#filePhe="Phenotype.txt",
vcf.jump=1529,
sep.vcf="\t",
#sep.phe="\t",
fileKin=FALSE,
filePC=FALSE,
out="mvp.vcf",
#maxLine=10000
)
But I find a problem in the data, the last header line, which starts with "#", there are 918 sample IDs while there are only 917 samples on the genotypic data lines (0/0,0/1,1/1 line).
Best,
Xiaolei
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/XiaoleiLiuBio/MVP/issues/21#issuecomment-400238694, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ATZtb3KLpe6wnpa9VVUqF9lsnnXhBpj_ks5uAfsagaJpZM4UxrEH.

Closed because there is no activity for too long. if you have any related new messages, feel free to open it.


Merge scale layer

Hello.
Great work!
In all  models you merge BatchNorm layer, but save Scale layer with lr_mult: 0. Why we cant merge Scale layer too. (I try this --  but have poor resault, why?)
Maybe BatchNorm layer after roi_pooling is usefull

If we want the update the conv layer parameters, so we couldn't merge the scale layer into conv layer for training phase.
I didn't do the experiment that not merge the batchNorm layer after roi_pooling, but I don't think it will make a big difference. Because the parameters obtained from the Imagenet have very good generalization ability.

@soeaver Thnak your for answer!
Can you explain more detailed why "we couldn't merge the scale layer into conv layer for training phase". I can imagine only one reason: Scale layer with lr_rate=0 are constant when we merge with convolution layer with Scale some weights became bigger and it penalties by weight_decay.

If we only merge bn into scale, the feature map can still be normalized by a single scale layer. But we merge both bn and scale into conv layer, and we update the conv layer parameters  at training phase  that will destroy the normalization parameters (because we only have conv layer now).
So, at inference phase we can merge bn and scale into conv layer.


Update CI Image

Overview

Add newly released CPython 3.12 to built CI image for testing and verification.
Expand Makefile for local image testing
Fix ARM build GHA syntax

Related Github Issue
Should streamline #897
Testing

Test CI build is here: https://github.com/newrelic/newrelic-python-agent/actions/runs/6409413149

Codecov Report

Merging #930 (5a71a13) into main (17cd48d) will decrease coverage by 0.40%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main     #930      +/-   ##
==========================================
- Coverage   81.74%   81.35%   -0.40%
==========================================
  Files         190      190
  Lines       19581    19581
  Branches     3401     3401
==========================================
- Hits        16007    15930      -77
- Misses       2593     2671      +78
+ Partials      981      980       -1

see 7 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Implement CRC CLI update

We need to implement update flow with podman-desktop API for update of installed binary

We do not have a proper way to detect this as output from the command... perhaps extend crc version for this from the API?

I see this as a lower priority for now.

We do not have a proper way to detect this as output from the command... perhaps extend crc version for this from the API?

Technically it should be possible, need to look on it

Can/should crc be shipped together with the openshift local extension image? (just a question I had in the back of my mind for a while :)

Yes, it's an option and we do it for most of the podman extensions. You'd still want to 'copy/install crc out' so that there's a proper copy on the system path in a normal location. This option is better for offline installs or someone who can't reach the download by the time you're trying, but bigger extension size especially if you do it for all OSes.

The question is:

would this be an installer of CRC with an extension
or, an extension installer with CRC embedded

The first option to me sounds more reasonable... although; perhaps this should be the "OpenShift Toolbox"-installer as we have more interesting extensions that can be collected in a nice and convenient package ;-)

Yeah, I agree a full offline installer is out of scope due to 2 GB size. :) Might not be a bad idea to include just the crc cli in the extension though to make sure it can always be installed?

Some additional comments after dropping this in the team sync:
Containers for extensions need to be platform specific if we want to include the crc binaries. This is currently not an issue for Kind as they are not an external extension, as it is the PD building/bundling process that handles this platform selection.

Perhaps picking up the need to 'update', as given in the command output might be a good first step.
New version available

Current state for podman: after you update the extension it knows about the new version, and the only prompt is an option to update on the dashboard.
IMHO I'd just worry about detecting and supporting a similar flow with this issue, and we figure out separately the best way to handle this for multiple extensions and engines. It wouldn't be very usable if Podman, Kind, OpenShift Local and others all add a status bar item flagging updates, and possibly with different behaviours. To make sure we cover this I've opened two issues:

Updating extensions: https://github.com/containers/podman-desktop/issues/2184
Updating container/Kube engines: https://github.com/containers/podman-desktop/issues/2185

right, there are two issues

informing the user of a new version of the extension
have the extension perform a dependency update

they are related, but very different issues. PD should not handle updates
of kubernetes or OpenShift local by itself, as this is the responsibility
of the extension as one of its dependencies. PD should in my opinion just
show extension updates that are available and perhaps offer an onupdate
event, or activate to be informed about an update flow? otherwise,
extensions need to track their own versions that were used/installed and
determine an update happened.
I still believe the extension should be informing the user somehow, and
think it is up to to the extension what is the best way: this might be a
dialog, a status bar notification, of other. for example, a security issue
most likely should be 'forced' by a dialog informing the user more
actively,while a minor upstate can be a notification or exclamation
somewhere in the UI. perhaps thought from UI/UX?
On Thu, Apr 20, 2023, 23:44 Tim deBoer @.***> wrote:

Current state for podman: after you update the extension it knows about
the new version, and the only prompt is an option to update on the
dashboard.
IMHO I'd just worry about detecting and supporting a similar flow with
this issue, and we figure out separately the best way to handle this for
multiple extensions and engines. It wouldn't be very usable if Podman,
Kind, OpenShift Local and others all add a status bar item flagging
updates, and possibly with different behaviours. To make sure we cover this
I've opened two issues:

containers/podman-desktop#2184
https://github.com/containers/podman-desktop/issues/2184
containers/podman-desktop#2185
https://github.com/containers/podman-desktop/issues/2185

—
Reply to this email directly, view it on GitHub
https://github.com/crc-org/crc-extension/issues/37#issuecomment-1516558728,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAAOZR3AIGVTEHOVMD4ZW3XCFKXBANCNFSM6AAAAAAWTWEA7A
.
You are receiving this because you commented.Message ID:
@.***>

right, there are two issues

informing the user of a new version of the extension
have the extension perform a dependency update

they are related, but very different issues. PD should not handle updates
of kubernetes or OpenShift local by itself, as this is the responsibility
of the extension as one of its dependencies. PD should in my opinion just
show extension updates that are available and perhaps offer an onupdate
event, or activate to be informed about an update flow? otherwise,
extensions need to track their own versions that were used/installed and
determine an update happened.
I still believe the extension should be informing the user somehow, and
think it is up to to the extension what is the best way: this might be a
dialog, a status bar notification, of other. for example, a security issue
most likely should be 'forced' by a dialog informing the user more
actively,while a minor upstate can be a notification or exclamation
somewhere in the UI. perhaps thought from UI/UX? or just show a notification for the extension tab and the specific extension when they report an updateavailable or by version check.
Note that the installation and updates for different engines will still inherit different behaviour: from an msi that needs to run to just a replacement of a binary.

@gbraad Some thoughts / ideas :) I have a quick & dirty mockup below.
I agree that a security issue could be handled differently than a minor update. Right now our task manager
might be an ideal place, might need to rephrase it (as below) to be more generally a notification area... anyway this is how a minor update could be.

For a security update... I am not sure how forceful we can/should be? Will the user get an oppty to migrate stuff off? (I know we say don't put anything on OpenShift Local you care about but...) Could be something where there's a specific time period we give until it's forced (kind of like how some mobile OSes handle updates now?) We could warn them about the ticking clock via notifications. Maybe when the time's run out we shut down OpenShift Local and don't allow it to be used unless it's updated? (Not sure if this is too harsh?)
Any update should be queued up since it's a large download, can be thru the task manager like so:

@gbraad agreed - the extension should remain solely responsible for both deciding that there is an update available and performing the update. My concern is if each extension decided on a different UI mechanism to notify users and update, and was suggesting we just keep that part simple until we've had a chance to talk/design.
https://github.com/containers/podman-desktop/issues/2019 will allow extensions to create tasks, and we might need a second issue for notifications, but then I think the pattern above could work well.


Geo widget error: Map container is already initialized

See console on https://enke.to/::widgets (and https://odk.enke.to/::YYY5).
This may be the cause of the missing (Google Maps) map "terrain" layer on enke.to as well.
There is also a report here, which may or may not be the same issue: https://forum.opendatakit.org/t/openstreet-maps/22941

As far as I can see this bug actually had no negative consequences, so the missing layer and forum issue must be something else.


add cache cleaner routine and do not fire requests to alchemy on non existent token

For unknown tokens a request to alchemy is fired. This prevent to slow down markets endpoint

Please check the nil case of token metadata when we integrate

Already checked. I don't like this implementation, it's risky


TVL2Deconv solution shifted by half kernel size

As a toy example I try to use TVL2Deconv (sporco version 0.1.11) to restore an image convolved with a 2D Gaussian. Interestingly, the deconvolved image ends up being circularly shifted by half of the size of the kernel. Here is a MWE.
I am out of ideas what could introduce such a phase ramp. Am I misusing something?

Sorry, this should probably be addressed in the documentation. The strange behaviour you're observing is due to a phase shift between the convolution implemented by ndimage.convolve and the frequency domain convolution used within tvl2.TVL2Deconv. If you replace
lp = ndimage.convolve(img, kernel)

in your example with
import sporco.linalg as spl
lp = spl.fftconv(kernel, img)

so that the forward operation is also performed in the frequency domain, the shift between the forward and inverse disappears.


title-body mismatches

 Issue by xoffey
Sat Jul 14 18:52:23 2018
Originally opened as https://github.com/codelucas/newspaper/issues/595

I'm not saying this is a bug. It's more of a thought on possible improvement and a question about whether anyone has worked on this issue.
Newspaper does a great job of parsing articles from high-quality news sites. I use it to extract title and body text, after I do a web crawl of major US news providers. It is especially good at extracting titles. Of course, it occasionally fails to extract the correct body text, because that's insanely hard to do.
Recently, I was looking at mismatches of titles with bodies. I found that title-body disagreement is a pretty reliable indication that the body-extraction did not work well. Essentially, if very few of the title words are found in the body text, it means something is probably wrong
I wonder, would it be feasible for newspaper to use information about the extracted title to help it decide which block(s) to use as the body text? Has anyone experimented on this?

site does not work anymore


corrected normalisation for deseq2 alignment free quantification

Hania and I fixed a bug in the was the normalisation was being performed for sleuth. Hania tested it and it works

Many thanks!


docstring formatting problem

I just noticed that the docstrings for the new MCXcorStacking module that was just merged have a formatting problem with the results sphynx is generating from the docstrings.   I see a lot of the docstring that I created have similar problems.   When I read the sphynx documentation and a few posts I found with a google search I can't see why there is a formatting problem.   I'm going to be working on the documentation in the next couple weeks anyway so this would be a good time to fix this this problem.
Also, if any of you have any experience with this I found one web page that said vscode has a nice plug in to help create sphynx format docstrings.  Any one in the group have experience with that?

I (@pavlis ) have been looking into this issue a bit and realize we have two fundamental problems we need to address to improve the look and readability of our documentations:

There are many formatting errors in our docstrings.
The look of our pages is less than ideal. Compare mspass.org and current obspy pages and you will see what I mean.

The fundamental cause of 1 is the inability to preview the formatted output from the variant of rst that sphynx uses to generate api pages from docstrings.  It is made worse by the fact that the special docstring directives like :param: and :type: seem to have some special formatting properties linked to sphynx that I do not understand.   Writing docstrings in our environment is like writing code in a computing language you are not fluent in writing.  That is fundamentally why we have a lot of formatting errors.  The solution to this problem is that someone in the group needs to commit to learning the tricks of the sphynx variant of rst and got over all the docstrings in the mspass code base to fix the many formatting errors.
Item 2 is almost certainly a configuration setup for how sphynx is run on the code base.   I think this one lands on the lap of @wangyinz to fix as he  set this up originally.   I am not sure how that is working with the makefile in docs.  I think we need to have function pages that have the look of obspy pages like  this one:


ladybird: unstable-2023-01-17 -> unstable-2023-08-10

Description of changes

Update to current master.

Pre-Download required Unicode, Common Locale Database, Timezone database and Root CA certificates to build with the new option -DENABLE_NETWORK_DOWNLOADS=OFF
Copy required downloaded files to the expected SERENITY_CACHE_DIR layout

For the above two, hopefully someone can point out a more elegant way to create the Caches directory

Remove workarounds for resolved upstream issues.
Add new missing qtmultimedia package
Remove unused unzip package
Point homepage at the serenity repository, as the ladybird one is archived now. Alternatively, we could use https://ladybird.dev

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

cc @emilytrau @networkException @fgaz

Fixes #212792

Alternatively, we could use https://ladybird.dev

Is that the official website? I can't find any links to it in the repo or at serenityos.org

Yes its official

Result of nixpkgs-review pr 248456 run on x86_64-linux 1

1 package built:

ladybird

Yes its official, though only really documented on Discord I guess.

Confirmed. Please use https://ladybird.dev then

Does not appear to build on darwin, please mark it as broken (or fix it)

I tried testing it on my apple silicon MacBook and I couldn't get past trying to get the build to use clang-11. No matter what hax I tried it kept wanting to use clang-11, which is far too old to compile the project (it actually crashes the compiler). I was able to force it to use one of the llvmPackages_16 derivations, but then the compile couldn't find the Apple SDK $SYSROOT or something, <assert.h> was not found.
Either way, adding
    substituteInPlace CMakeLists.txt \
      --replace "if (APPLE)" "if (FALSE)"

let the CMake configure, so I'll add that and see if it fixes the CI? If that fails I'll mark it as broken :\

Well that's super annoying. Why does nix pass an absolute path for both CMAKE_INSTALL_BINDIR  and CMAKE_INSTALL_LIBDIR? Aren't those both supposed to be relative to CMAKE_INSTALL_PREFIX? I pushed a fix to serenity upstream that I figured would fix the macOS builds but ended up breaking the Linux ones :/

All right, x86_64 Darwin is now working, but aarch64-darwin CI is seeing the same "using clang-11 even though we told it to use a different compiler" issue I was seeing locally. So not sure if I should mark that as unsupported, report an issue/discourse thread, or accept it as expected?

Perhaps try using darwin.apple_sdk_11_0.callPackage instead of callPackage (last bullet at https://nixos.org/manual/nixpkgs/stable/index.html#sec-darwin )

Perhaps try using darwin.apple_sdk_11_0.callPackage instead of callPackage (last bullet at https://nixos.org/manual/nixpkgs/stable/index.html#sec-darwin )

all-packages.nix currently has this:
 ladybird = qt6Packages.callPackage ../applications/networking/browsers/ladybird {
    stdenv = if stdenv.isDarwin then darwin.apple_sdk_11_0.llvmPackages_14.stdenv else stdenv;
  };

I did a check of env | grep PATH with the current setup, and saw that clang-wrapper-11 was before clang-wrapper-14 in the path on my machine.
Which seems... guaranteed not to work?
PATH=/nix/store/32w82ry9cb7bsxdx324wjnbdgh989dqk-cmake-3.26.4/bin
/nix/store/69dxkkb3vwx73n332fldjvll393cpydp-ps-adv_cmds-119/bin
/nix/store/jcd8f778j0ckdmp50h9wn8iakkclf2cd-ninja-1.11.1/bin
/nix/store/7dkbcs6vgwbkb6rafh03j9j0bn8qpki0-clang-wrapper-11.1.0/bin
/nix/store/78g551gc2wkzr06ar571fgvj6qn5gi14-clang-11.1.0/bin
/nix/store/r30ljn3w4317s0xhf8s6wf3c3kx5cfiv-coreutils-9.3/bin
/nix/store/dnvdnqc5cdx499j2y0pyqlsdvd868y08-cctools-binutils-darwin-wrapper-11.1.0-973.0.1/bin
/nix/store/6j9rsw4086h5g8db7anxy6bbrsv4zqz7-cctools-binutils-darwin-11.1.0-973.0.1/bin
/nix/store/3h3nfz7qbf8f9j2acvsdaqs59dd0pkkp-lndir-1.0.4/bin
/nix/store/fgap77lk5a7q9xjan91cf135bvymvb4m-clang-wrapper-14.0.6/bin
/nix/store/n8zdf0a7zz7jk4md936j97g5gykhl5sc-clang-14.0.6/bin

If I change it to this (with the required with qt6Packages; additions to nativeBuildInputs and buildInputs):
  ladybird = darwin.apple_sdk_11_0.callPackage ../applications/networking/browsers/ladybird {
    stdenv = if stdenv.isDarwin then darwin.apple_sdk_11_0.llvmPackages_16.stdenv else stdenv;
  };

Then I see the same, but with clang-wrapper-16 at the position that clang-wrapper-14 was at:
PATH=/nix/store/32w82ry9cb7bsxdx324wjnbdgh989dqk-cmake-3.26.4/bin
/nix/store/69dxkkb3vwx73n332fldjvll393cpydp-ps-adv_cmds-119/bin
/nix/store/jcd8f778j0ckdmp50h9wn8iakkclf2cd-ninja-1.11.1/bin
/nix/store/7dkbcs6vgwbkb6rafh03j9j0bn8qpki0-clang-wrapper-11.1.0/bin
/nix/store/78g551gc2wkzr06ar571fgvj6qn5gi14-clang-11.1.0/bin
/nix/store/r30ljn3w4317s0xhf8s6wf3c3kx5cfiv-coreutils-9.3/bin
/nix/store/dnvdnqc5cdx499j2y0pyqlsdvd868y08-cctools-binutils-darwin-wrapper-11.1.0-973.0.1/bin
/nix/store/6j9rsw4086h5g8db7anxy6bbrsv4zqz7-cctools-binutils-darwin-11.1.0-973.0.1/bin
/nix/store/3h3nfz7qbf8f9j2acvsdaqs59dd0pkkp-lndir-1.0.4/bin
/nix/store/3dl7pdy82jdzbxixgbvvdl89hq5mh8dz-clang-wrapper-16.0.6/bin
/nix/store/c52xq08a10zqkazj4q0a3y4md672505g-clang-16.0.6/bin

Is this supposed to work?

@fgaz  I've updated the ladybird PR here to the latest current rev and tested it on both macOS Sonoma (apple silicon) and Ubuntu 22.04 (x86_64).
There were a few extra upstream changes since the last time I looked at this package:

The app name was changed to Ladybird with a capital L
The macOS app bundle should be fully self-contained now
Most internal helper process executables were moved to $PREFIX/libexec. The way that Ladybird/CMakeLists.txt reads CMAKE_INSTALL_LIBEXECDIR to embed in the binary really really wants the path to be relative to $CMAKE_INSTALL_PREFIX rather than an absolute path. I'm assuming that nixpkgs would rather we patch the CMakeLists.txt than pass a relative path with -DCMAKE_INSTALL_LIBEXECDIR=libexec?
A few more downloaded files were added, including another Unicode file (IDNAMappingTable.txt), the public suffix list (upstream cmake just downloads master, but I pinned it), and a standard ICC profile from Adobe.
The macOS port by default uses a custom AppKit/Cocoa UI instead of Qt. More patches are needed to properly support both UI toolkits in the nix file.
OpenGL is used for graphics-accelerated painting, and WebGL features. I could not get this to work properly on GNU/Linux (Ubuntu), so I disabled the feature on Linux.
There's CMake feature detection on non-macOS to use pulse audio if available. If pulse audio is not available, it falls back to Qt multimedia.

Most internal helper process executables were moved to $PREFIX/libexec. The way that Ladybird/CMakeLists.txt reads CMAKE_INSTALL_LIBEXECDIR to embed in the binary really really wants the path to be relative to $CMAKE_INSTALL_PREFIX rather than an absolute path. I'm assuming that nixpkgs would rather we patch the CMakeLists.txt than pass a relative path with -DCMAKE_INSTALL_LIBEXECDIR=libexec?

If you add something like this to make the absolute path relative into the upstream CMakeLists.txt, then it should just work for both cases: https://git.dblsaiko.net/nucom/tree/CMakeLists.txt#n32

Thanks, I'll have a look tomorrow. Merging this is really long overdue :sweat_smile:
-DCMAKE_INSTALL_LIBEXECDIR=libexec is also fine if you can't use an absolute path, we have many packages doing that


RecursiveTextSplitter creates a list of strings that don't play well with Pinecone.from_documents()

The RecursiveTextSplitter creates a list of strings.
The CharacterTextSplitter creates a list of langchain.schema.Document
The Pinecone.from_documents() loader seems to expect a list of langchain.schema.Document
As such, if you try to feed it a "documents" object created by the RecursiveTextSplitter, you get this error:
--> 181         texts = [d.page_content for d in documents]
AttributeError: 'str' object has no attribute 'page_content'

This is a bug on the RecursiveTextSplitter, right?

Can workaround by rebuilding the docs:
from langchain.schema import Document
new_docs = [Document(page_content=doc) for doc in docs]`


Clock divider calculation

Hello,
i'm trying to connect to a ws2812e led strip with my pi_pico board.
I'm having a rather terrible level shifter on my setup which leads to messy signals, the level shifters struggles to push the voltage to 5v5 on the output side which leads to most of the pwm signal beeing closer to 3v3 for most of the high phases of the signal.
That's my problem really but just for the explanation.
The issue is i tested this setup with the ws2812 micropython library and had no issues with my led strip.
Once i switched to rust and your library, i started to notice issues with the signal and incorrect outputs on my led strips.
After diving into both code bases pio state machine setup i noticed a difference in calculating the clock divider value.
this is how the c implementation of the micropython code calculates the divider values.
// Frequency given in Hz, compute clkdiv from it.
uint64_t div = (uint64_t)clock_get_hz(clk_sys) * 256ULL / (uint64_t)args[ARG_freq].u_int;
if (!(div >= 1 * 256 && div <= 65536 * 256)) {
  mp_raise_ValueError(MP_ERROR_TEXT("freq out of range"));
}
clkdiv_int = div / 256;
clkdiv_frac = div & 0xff;

and this is your solution.
// Configure the PIO state machine.
let bit_freq = FREQ * CYCLES_PER_BIT;
let mut int = clock_freq / bit_freq;
let rem = clock_freq - (int * bit_freq);
let frac = (rem * 256) / bit_freq;
assert!(
    (1..=65536).contains(&int) && (int != 65536 || frac == 0),
    "(System Clock / {}) must be within [1.0, 65536.0].",
    bit_freq.to_kHz()
);

// 65536.0 is represented as 0 in the pio's clock divider
if int == 65536 {
    int = 0;
}
// Using lossy conversion because range have been checked
let int: u16 = int as u16;
let frac: u8 = frac as u8;

the problem with your solution might be that dividing the clock freq  and after that multiplying it by 256 is that the calculation increases the interger maths error.
By expanding the calculation to 64 bits as in the python base solution and then multiplying it by 256 and dividing it by 256 again after the divisions is that the incluence of calcualtion errors goes down.
After i have to confess rather clumsily trying to port the python code to rust i came up with this solution and experience no further errors in my led output.
let bit_freq = FREQ * CYCLES_PER_BIT;
let div: u64 = ((clock_freq.to_Hz() as u64)  * 256u64) / (bit_freq.to_Hz() as u64);

assert!(div >= 1 * 256 && div <= 65536 * 256,
    "(System Clock / {}) must be within [1.0, 65536.0].",
    bit_freq.to_kHz()
);

// Using lossy conversion because range have been checked
let int: u16 = (div / 256u64) as u16;
let frac: u8 = div as u8;

Have to test this further as my setups still isn't working. Only with an active debugger connection...


Unsupported scenario: Can we please update if custom images/SIG are supported? Also VMSS? Uni/Flexible orchestration mode?

Azure documentation issue guidance
Thanks for opening an issue in the Azure technical documentation repository.
We use GitHub issues as the primary channel for customer and community feedback about the Azure documentation.
Creating an issue
We prefer that you create documentation feedback issues using the Feedback link on the published article - the feedback control on the doc page creates an issue that contains all the article details so you can focus on the feedback part.
You can also create a feedback issue here in the repo. If you do this, please make sure your issue lists:

[ ] The relevant Azure service or technology.
[ ] A link to the published documentation article that you have feedback about.
[ ] Clear, specific feedback that the author can act on.

Pull requests and article contributions
If you know the change that is needed in an article, we encourage you to submit the changes directly using a pull request. If the change is large, or if you want to contribute an entire article, follow these guidelines:

[ ] Don't surprise us with a big pull request or a pull request with a new article! Submit an issue that describes the details of the proposed large change or new article.
[ ] Include the service or technology area.

We'll route the issue to the appropriate content team for review and discussion.
Tech support and product feedback
If you would like to contact Microsoft about other things, such as product feedback or tech support, please review these guidelines:

If you need technical support using Azure, the paid and free support options are described here: https://azure.microsoft.com/support/options/.

Each article in the Azure technical documentation contains a product feedback button - it's best to submit product feedback directly from a relevant article. Otherwise, you can submit product feedback for most Azure products in the following product feedback forum: https://feedback.azure.com/d365community/forum/79b1327d-d925-ec11-b6e6-000d3a4f06a4.

@ElenaMST
Could you please elaborate the query for better understanding. So, we can assist you further.
It would be great if you could add a link to the documentation you are following for these steps? This would help us redirect the issue to the appropriate team. Thanks!!

Reference link: https://learn.microsoft.com/es-es/azure/virtual-machines/workloads/canonical/ubuntu-pro-in-place-upgrade
Will be possible to update if the procedure is supported only for Marketplace images or will apply also for custom/SIG images.
As well for VMSS. If it supported only for regular VMSS or if is applicable for VMSS as well.
Thank you!
Elena Stefan
Support Engineer
Microsoft Azure Technical Support
Customer Service and Support
<EMAIL_ADDRESS>Working hours: Mon-Fri, 9AM - 6PM EET/UTC +2
If you need to work with another Support Engineer outside of my working hours, please send email to<EMAIL_ADDRESS>with your case number, and availability.We are always interested to hear your feedback. Please feel free to reach my manager regarding the level of service you have received<EMAIL_ADDRESS>[Microsoft Azure logo horizontal]
Microsoft Azurehttps://azure.microsoft.com/en-us | Azure Statushttps://status.azure.com/en-us/status | Support Planshttps://azure.microsoft.com/en-us/support/plans | Create a Casehttps://azure.microsoft.com/en-us/support/create-ticket | Privacy Policyhttps://privacy.microsoft.com/en-us/PrivacyStatement
From: AjayBathini-MSFT @.>
Sent: Tuesday, September 19, 2023 7:09 PM
To: MicrosoftDocs/azure-docs @.>
Cc: Elena Stefan @.>; Mention @.>
Subject: Re: [MicrosoftDocs/azure-docs] Unsupported scenario: Can we please update if custom images/SIG are supported? Also VMSS? Uni/Flexible orchestration mode? (Issue #114919)
@ElenaMSThttps://github.com/ElenaMST
Could you please elaborate the query for better understanding. So, we can assist you further.
It would be great if you could add a link to the documentation you are following for these steps? This would help us redirect the issue to the appropriate team. Thanks!!

Reply to this email directly, view it on GitHubhttps://github.com/MicrosoftDocs/azure-docs/issues/114919#issuecomment-1725992396, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AXOVDCM7XVZDWIHBDB3HZ6LX3G7S5ANCNFSM6AAAAAA46OM2AY.
You are receiving this because you were mentioned.Message ID: @.***>

@ElenaMST
Thanks for your feedback! I've assigned this issue to the author who will investigate and update as appropriate.


Adds documentation about updating icons

As a result of having to do these PRs here:
PR 1041 static
PR 305 govuk_template
I have documented what steps should be taken when updating icons.

Removed DNM as this is good to merge now that #305 is merged.


TEST FOR BOT, DO NOT MERGE

- What I did
- How I did it
- How you can verify it

This pull request fixes #issue_number_here

Hello, I'm a bot! I can make mistakes, notify gabrieldemarmiesse if I made one.
I see that you modified the following file{plural}:
{files_changed}
The owner of those file is @{owner}
@{owner} could you please take a look at it whenever
you have the time and add a review? Thank you in advance for the help.

Hello, I'm a bot! I can make mistakes, notify gabrieldemarmiesse if I made one.
I see that you modified the following files:
stuff
The owner of those file is @gabrieldemarmiesse
@gabrieldemarmiesse could you please take a look at it whenever
you have the time and add a review? Thank you in advance for the help.


HDFS-16214. Lock optimization for deleteing, no locks on the collection block

Lock optimization for deleteing, no locks on the collection block
HDFS-16214

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
1m  6s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
38m 18s

trunk passed

+1 :green_heart:
compile
1m 43s

trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
compile
1m 32s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
checkstyle
1m  2s

trunk passed

+1 :green_heart:
mvnsite
1m 40s

trunk passed

+1 :green_heart:
javadoc
1m 11s

trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javadoc
1m 39s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
4m  1s

trunk passed

+1 :green_heart:
shadedclient
27m 26s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 29s

the patch passed

+1 :green_heart:
compile
1m 33s

the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javac
1m 33s

the patch passed

+1 :green_heart:
compile
1m 22s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
javac
1m 22s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 57s

the patch passed

+1 :green_heart:
mvnsite
1m 29s

the patch passed

+1 :green_heart:
javadoc
0m 59s

the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javadoc
1m 33s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
4m  5s

the patch passed

-1 :x:
shadedclient
26m 44s

patch has errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
396m  9s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 58s

The patch does not generate ASF License warnings.

513m 25s

Reason
Tests

Failed junit tests
hadoop.hdfs.TestDFSShell

hadoop.hdfs.TestFileCreation

hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot

hadoop.hdfs.TestHDFSTrash

hadoop.hdfs.server.namenode.TestCheckpoint

hadoop.hdfs.server.namenode.TestDeleteRace

hadoop.hdfs.TestDistributedFileSystem

hadoop.hdfs.TestListFilesInDFS

hadoop.fs.viewfs.TestViewFileSystemLinkMergeSlash

hadoop.fs.viewfs.TestViewFileSystemLinkFallback

hadoop.fs.contract.hdfs.TestHDFSContractDelete

hadoop.hdfs.server.namenode.snapshot.TestSnapshot

hadoop.fs.viewfs.TestViewFSOverloadSchemeWithMountTableConfigInHDFS

hadoop.hdfs.TestViewDistributedFileSystem

hadoop.fs.contract.hdfs.TestHDFSContractRootDirectory

hadoop.hdfs.TestDFSPermission

hadoop.fs.viewfs.TestViewFsWithAcls

hadoop.fs.contract.hdfs.TestHDFSContractRename

hadoop.hdfs.server.namenode.TestFSDirectory

hadoop.hdfs.TestListFilesInFileContext

hadoop.fs.TestSymlinkHdfsFileSystem

hadoop.fs.viewfs.TestViewFsAtHdfsRoot

hadoop.fs.contract.hdfs.TestHDFSContractGetFileStatus

hadoop.hdfs.server.namenode.TestProtectedDirectories

hadoop.hdfs.TestQuota

hadoop.hdfs.server.namenode.TestNNThroughputBenchmark

hadoop.fs.TestSymlinkHdfsFileContext

hadoop.fs.viewfs.TestViewFileSystemOverloadSchemeWithHdfsScheme

hadoop.hdfs.tools.TestViewFSStoragePolicyCommands

hadoop.fs.viewfs.TestViewFsWithXAttrs

hadoop.fs.TestGlobPaths

hadoop.fs.viewfs.TestViewFileSystemWithTruncate

hadoop.fs.viewfs.TestViewFileSystemWithAcls

hadoop.hdfs.server.balancer.TestBalancer

hadoop.hdfs.TestHDFSFileSystemContract

hadoop.hdfs.TestViewDistributedFileSystemContract

hadoop.hdfs.TestViewDistributedFileSystemWithMountLinks

hadoop.fs.TestHDFSFileContextMainOperations

hadoop.fs.viewfs.TestViewFsHdfs

hadoop.hdfs.server.namenode.TestNestedEncryptionZones

hadoop.hdfs.server.namenode.TestAuditLoggerWithCommands

hadoop.hdfs.TestAclsEndToEnd

hadoop.fs.viewfs.TestViewFileSystemHdfs

hadoop.hdfs.server.namenode.TestNamenodeRetryCache

hadoop.fs.viewfs.TestViewFileSystemWithXAttrs

hadoop.fs.viewfs.TestViewFileSystemOverloadSchemeHdfsFileSystemContract

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3885/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3885

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell

uname
Linux d18e14931ebb 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 0d0b152bf9a26b5187efa9c087feeeb835c45741

Default Java
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3885/1/testReport/

Max. process+thread count
2002 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3885/1/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
1m  0s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
32m 34s

trunk passed

+1 :green_heart:
compile
1m 27s

trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
compile
1m 22s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
checkstyle
1m  3s

trunk passed

+1 :green_heart:
mvnsite
1m 29s

trunk passed

+1 :green_heart:
javadoc
1m  1s

trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javadoc
1m 32s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
3m 11s

trunk passed

+1 :green_heart:
shadedclient
22m 21s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 14s

the patch passed

+1 :green_heart:
compile
1m 17s

the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javac
1m 17s

the patch passed

+1 :green_heart:
compile
1m 10s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
javac
1m 10s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 51s

the patch passed

+1 :green_heart:
mvnsite
1m 18s

the patch passed

+1 :green_heart:
javadoc
0m 50s

the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javadoc
1m 27s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
3m 10s

the patch passed

+1 :green_heart:
shadedclient
22m  0s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
373m  2s

hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 45s

The patch does not generate ASF License warnings.

471m 51s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3885/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3885

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell

uname
Linux 69c0b6f13276 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / efaeb849105015d3045f1d1635c16d2f33cb54a2

Default Java
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3885/2/testReport/

Max. process+thread count
3002 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3885/2/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
1m  4s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 26 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
33m  1s

trunk passed

+1 :green_heart:
compile
1m 27s

trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
compile
1m 24s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
checkstyle
1m  9s

trunk passed

+1 :green_heart:
mvnsite
1m 30s

trunk passed

+1 :green_heart:
javadoc
1m  1s

trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javadoc
1m 32s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
3m 15s

trunk passed

+1 :green_heart:
shadedclient
22m 33s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 19s

the patch passed

+1 :green_heart:
compile
1m 19s

the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javac
1m 19s

the patch passed

+1 :green_heart:
compile
1m 10s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
javac
1m 10s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
0m 59s
/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs-project/hadoop-hdfs: The patch generated 18 new + 1046 unchanged - 1 fixed = 1064 total (was 1047)

+1 :green_heart:
mvnsite
1m 17s

the patch passed

+1 :green_heart:
javadoc
0m 52s

the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javadoc
1m 22s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
3m 19s

the patch passed

+1 :green_heart:
shadedclient
22m 10s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
365m 59s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 47s

The patch does not generate ASF License warnings.

466m 33s

Reason
Tests

Failed junit tests
hadoop.hdfs.TestLease

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3885/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3885

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell

uname
Linux 354fe4a9a7c0 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 24dd0b4355be75b1642000a84971b468e34d6e1a

Default Java
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3885/3/testReport/

Max. process+thread count
2947 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3885/3/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 51s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 26 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
33m 44s

trunk passed

+1 :green_heart:
compile
1m 29s

trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
compile
1m 23s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
checkstyle
1m  8s

trunk passed

+1 :green_heart:
mvnsite
1m 27s

trunk passed

+1 :green_heart:
javadoc
1m  3s

trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javadoc
1m 34s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
3m 16s

trunk passed

+1 :green_heart:
shadedclient
22m 34s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 15s

the patch passed

+1 :green_heart:
compile
1m 20s

the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javac
1m 20s

the patch passed

+1 :green_heart:
compile
1m 15s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
javac
1m 15s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
0m 58s
/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 1047 unchanged - 0 fixed = 1049 total (was 1047)

+1 :green_heart:
mvnsite
1m 19s

the patch passed

+1 :green_heart:
javadoc
0m 53s

the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javadoc
1m 29s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
3m 17s

the patch passed

+1 :green_heart:
shadedclient
22m 25s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
375m 10s

hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 43s

The patch does not generate ASF License warnings.

476m 14s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3885/6/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3885

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell

uname
Linux e1c6c0735a7a 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 50d8750e40c12a9e9497fc6e76afa289e0175acc

Default Java
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3885/6/testReport/

Max. process+thread count
2907 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3885/6/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.


Clean up orphaned files which are removed from form.

@matthewford
Currently we do not remove form attachment files (stored in FormAnswerAttachment) if it removed from Form.
It explained as we need to click "Save and Continue" on JS form in order to apply removing attachment from Form document hash.
Looks like we need to remove unused FormAnswerAttachments's via for example cron task?

Think we need to discuss this.

@matthewford gonna take a look to this one.
So possible ways to solve:

Add cron script, which would clean up Attachments related to forms, but not saved in document of form (probably weekly)
Add ajax removing of attachments on click "Remove"

think step 2) would be preferable, although when you say ajax I assume you mean a way with a non-js fallback?

In non - js version already implemented removing of attachment

Ok, gonna add removing of attachment via AJAX for JS version


Fix different UI issues

:tada: This PR is included in version 2.7.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


arsenal: 1.0.2 -> 1.1.0

Description of changes
Update to latest upstream release 1.1.0

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 173634 run on x86_64-linux 1

3 packages built:

arsenal
python310Packages.pyfzf
python39Packages.pyfzf


Flashing bug in iPhone devices

Package(s) version(s): 1.1.0
Description of the issue/feature:
In iPhone devices the application is continuously flashing.
The issue is because Mobile Safari is not supporting media queries using rem units: issue.

The issue was not part of this project, it was related to a specific style of adidas.github.io.


Uncaught Error: EROFS, read-only file system '...'

During trying to open file from external NTFS device (via USB) mounted as readonly I have got:
/Applications/Atom.app/Contents/Resources/app/src/workspace.js:504
--
Error: EROFS, read-only file system '/Volumes/data/trizeni/getconfig757/getconfig757/getcfg'
  at Error (native)
  at Object.fs.openSync (fs.js:455:18)
  at Object.module.(anonymous function) [as openSync] (/Applications/Atom.app/Contents/Resources/atom/common/lib/asar.js:422:20)
  at Project.module.exports.Project.open (/Applications/Atom.app/Contents/Resources/app/src/project.js:287:31)
  at Workspace.module.exports.Workspace.openURIInPane (/Applications/Atom.app/Contents/Resources/app/src/workspace.js:492:31)
  at Workspace.module.exports.Workspace.open (/Applications/Atom.app/Contents/Resources/app/src/workspace.js:418:19)
  at Ipc.<anonymous> (/Applications/Atom.app/Contents/Resources/app/src/window-event-handler.js:45:32)
  at Ipc.emit (events.js:110:17)
  at process.<anonymous> (/Applications/Atom.app/Contents/Resources/atom/renderer/api/lib/ipc.js:22:29)
  at process.emit (events.js:118:17)

OS:
System Version: OS X 10.10.1 (14B25)
Kernel Version: Darwin 14.0.0
Darwin XXXX 14.0.0 Darwin Kernel Version 14.0.0: Fri Sep 19 00:26:44 PDT 2014; root:xnu-2782.1.97~2/RELEASE_X86_64 x86_64
Atom:
Version 0.174.0 (0.174.0)
FS:
/dev/disk2s1 on /Volumes/data (ntfs, local, nodev, nosuid, read-only, noowners)

Same happend to me.
All the same but I'm on OSX 10.8.5

Should have been fixed by https://github.com/atom/atom/pull/5157. Let us know if you can still reproduce. :bow:


Update ROSCO Simulink Example

Major Changes:

Added scripts to Matlab-Toolbox/Utilities/ for running simulations in matlab/simulink
Updated Test_Cases/5MW_Land_Simulink example to work with OpenFAST 2.4.0

Minor Changes:

Added output to Test_Cases/NREL-5MW/NRELOffshrBsline5MW_Onshore_ElastoDyn.dat for simulink simulations
Exclude temp matlab/simulink files from git tracking

This seemed to clear up the issue #25 and worked for me.  It still depends on the OpenFAST version.  Maybe we keep this open until theres another release of OpenFAST.  I still need some dev features to run OpenFAST with Simulink on mac.

This seemed to clear up the issue #25 and worked for me.  It still depends on the OpenFAST version.  Maybe we keep this open until theres another release of OpenFAST.  I still need some dev features to run OpenFAST with Simulink on mac.


Project cookiecutter template?

Was rp2 created from a cookiecutter template? If so, may I ask, where could I find the template?

No, it was written from scratch. I am now using it as a template for new Python projects though (e.g. [DaLI]( uses uses RP2's structure).

Thanks. I'm trying to get up-to-speed on the latest python packaging best practices.

Ok: RP2 should be fairly close to Python packaging state of the art (as of about 1 year ago, when I did the research). Python packaging is a moving target though... If you see anything that should be updated in RP2 or DaLI let me know.

In my projects, I've been using the audreyfeldroy pypackage cookiecutter, here: https://github.com/audreyfeldroy/cookiecutter-pypackage
You might consider forking her project and customizing it with your preferences. Your work has some differences in the directory structure and the pre-commit hooks, black, isort, mypy, etc.  I'm considering adding some of these tools to my own projects.

Yes, looks like there are already plenty of cookie cutter projects: there may not be need for a new one. Thanks for the pointer to Audrey's project. RP2 and DaLI are src-based projects: based on a quick look at Cookiecutter PyPackage, it seems to me it doesn't generate src-based structures, so it would not be compatible with RP2 and DaLI, but correct me if I'm wrong. Regardless, I'll take a look at it to see if there are configuration ideas I can bring into RP2 and DaLI.

I was wondering why you had a src directory. Always something new to learn.  Cookiecutter is flexible enough for a wide variety of project layouts.  I stumbled upon the hypermodern-python cookiecutter template tonight -- not endorsing it since I have never used it, but it does have the src layout: https://github.com/cjolowicz/cookiecutter-hypermodern-python

Interesting, thanks for sharing.

Closing the issue.


Documentation update

Moved examples into ipython notebooks and rendered them as part of the documentation. They are statically rendered now (execution is set to 'off'), but once the library is stable, we can have them automatically run as part of the CI pipeline. That ensures that tutorial code actually works! Another possibility is then to combine our examples folder with something like Binder, which then has all those notebooks available to play with
Copied over the primitives currently described on overleaf into a table format
Used the nice NIR schema figure from overleaf in the 'what' section
Used sphinx auto-api, which generates a much nicer API reference
Light logo in the sidebar, some text fixes here and there

Difference between before and after

Regarding the execution, I think you have a great point about directly providing runnable examples. I also have high hopes for stuff like thebe to run the notebooks directly in the browser. It's not quite there, but we should definitely keep an eye out for ways to make the code as accessible as possible.


Update help, add, list and edit commands in user guide

Help command has no change in terms of usage or details, but verified against our user guide draft.
Update Add command to change phone number field to telegram handle. Address and tag fields removed.
Group name field added as a compulsory field because we want students to be created with an assigned group so no student is left without a group.
Updated List command to reflect that we store, specifically, students in our app, CSBook.
Update Edit command to change phone number field to telegram handle, address and tag fields are removed, and group name field added. Examples are updated accordingly to draft user guide.

Codecov Report

Merging #44 (a990a6c) into dev (fad62fa) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##                dev      #44   +/-   ##
=========================================
  Coverage     71.45%   71.45%
  Complexity      377      377
=========================================
  Files            68       68
  Lines          1156     1156
  Branches        112      112
=========================================
  Hits            826      826
  Misses          302      302
  Partials         28       28

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fad62fa...a990a6c. Read the comment docs.


Provide policy to demonstrate all compilation stages

It would be cool to have a demo policy where all — or as many as possible — of the compilation stages were involved. Let's try and add one :)

https://github.com/srenatus/opa-explorer/pull/3 not all, but more


[Broker] [WIP]support split bundle by specified topic

Fixes #13761
Motivation
Support split partitions belonging to specified topic in a bundle
Modifications

add a new algorithm for bundle split named SpecifiedTopicCountEquallyDivideBundleSplitAlgorithm
abstract split arguments to BundleSplitOption

Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
(or)
This change is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end deployment with large payloads (10MB)
Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:
If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): ( no)
The public API: ( no)
The schema: ( no)
The default values of configurations: ( no)
The wire protocol: ( no)
The rest endpoints: ( no)
The admin cli options: ( no)
Anything that affects deployment: ( no)

Documentation
Check the box below or label this PR directly (if you have committer privilege).


Complete design specification for date range option

Is your feature request related to a problem? Please describe.
There is an existing design file for a date range feature.
Describe the solution you'd like
This file needs to be completed and presented to the engineers in order to finalize the requested development spec.

Date range specs are outlined here: https://www.figma.com/file/mZzZNE0cEPnphZozbaZfDy/Calendar?node-id=1839%3A23842&t=gkfCMlaUKKa7v6F3-1


Strange Textures when shifting

Mod loader
Fabric
Minecraft version
1.21
SkinShiftter version
1.2
CraftedCore version
5.5
Modloader version
0.16.0
The latest.log file
latest.log
Issue description
Skins Appear Garbled for some players? I Initially thought it was from different player model types (thin vs classic) but idk
Steps to reproduce
/skinshifter set @s Notch
/skinshifter set @s Herobrine
Other
Video of Weirdness Here

Hey! Please try resetting your cache by running the command /craftedcore clear cache or by deleting the directory called craftedcore in your configuration directory. Let me know if it worked!

Neither Worked, still displays the same as before

Ok, I've tested several times now. It only occured for Notch in my testing. I think the skin png of his Skin isn't in a valid format for 1.21. This isn't fixable for now, since the skins are still interpreted by Minecraft's vanilla code.

most of the players with mojang capes have the same effect, and some without it as well, such as "dwi" "wanderingtrader" "mcci" (wrong skin type, slim, not wide), "cape" and a bunch more


Bug? 2 virus at same location.

Here the screenshot, i use my client (not official agar client),

I found this in the latest update.

This happens on all virus.

This might be because your client is a bit buggy

In agario client too..

Yes

First thing that comes to my head is...  that the willCollide check is inverted.

@ZfsrGhS953  @RasyiidWho  @AlexHGaming @PheoniXAgario
https://github.com/Megabyte918/MultiOgar-Edited/commit/05ee9d6a21e31fc871bf3e7d0e8b308c1f7c94ab
this bug are created in this update

@Jhair2015 ikr.

Yep, same issue here (with agar client and my custom client)

@AlexHGaming Same profile picture lol
The virus bug is either a client issue, or a bug with the latest version.

Fixed; sorry for the wait


Add 2 parameters in LLAMALEND bot creation

Description
I added a delegate parameter for price prediction bot. Also I added a controller index so that we can use different controller of the same collateral. We need to update frontend for the update.
Changed Vyper function is as follows.
https://github.com/VolumeFi/curve-llamalend-leverage-bot-vyper/blob/6829c853909295f1b5bfd6b443b4a72baabe5bae/contracts/factory.vy#L192

Fixed with https://github.com/VolumeFi/palomabot/pull/464


If we have assets define dependencies, then I think they should all be installed at the same level (in addons folder). For dependency version conflicts I think a simple resolution would be best, such as always going for the latest version.
On the otherhand, if we really want to implement dependency management with deduplication and version conflict resolution by allowing multiple versions of the same asset to be installed, then that's ok too! As if it is really robustly made, then it will be able to support expansion of assets ecosystem long term. Somehow I don't like the idea of having multiple versions of the same asset in my game project though.

I don't think putting dependencies as subfolder of assets is a good idea as it is leading to a path that will be very complex.
If we have assets define dependencies, then I think they should all be installed at the same level (in res://addons folder

I agree with that. I would also prefer addons to exist once in the projec in a standard location, not as many times as versions required by various other sources and in various folders. Dependencies are a thing, but we can find a middle-ground too instead of going maximum fine-grained.

These problems are why my initial suggestions said not to let assets define dependencies yet, but just have a way to list dependencies of the project, and let user manually install any dependencies of assets.

To me if assets define dependencies it would be primarily for the user to be quicker informed than having to look at docs or reviewing all plugins before installing. And eventually, have an option to download the dependencies at the same time, for convenience (especially when you install from scratch). We probably don't need to resolve dependency conflicts too, at least the install tool could inform the user if there is one.

I feel like this whole dependency issue is being way over-thought about

@LikeLakers2 Proper dependency management and conflict avoidance is an important and hard issue. It's being discussed so much because of that

I don't foresee this conversation ending with the way it's heading currently.

Trying to just push this through without figuring out all the problems that dependency brings would be a very bad idea and just cause way more problems.  This is a hard issue and it will take time.

For dependency version conflicts (two assets depend on the same asset but different versions of it) I think a simple resolution would be best

@chanon My ideas are simple.  What's complex about naming a folder and changing the relevant file paths in text documents? It's a one time cost for the user, makes it apparent what version of what you have installed, and is fairly easy to write.

such as always going for the newer version

That will lead to plug-ins breaking without warning as their dependencies update and change how they work. EG Plugin 1 and 2 both depend on Dep 1, but different versions.  Plugin 2 depends on the current, latest version but plugin 1 depends on an older version that behaves differently than the latest version.  Plugin 1 no longer works as intended or has bugs it didn't ship with.

We probably don't need to resolve dependency conflicts too, at least the install tool could inform the user if it can detect one.

@Zylann Then we have the issue where the user can't install x plugin because y uses a different version.  That will not be looked upon as a good thing by users

I agree with @willnationsdev and @LikeLakers2

I feel like this whole dependency issue is being way over-thought about. Maybe we should come up with a few simple systems and have a discussion on each, or a vote? I don't foresee this conversation ending with the way it's heading currently.

Too much bike shedding. This is a game engine, not a game-engine-addon-delivery-platform. Just vendor dependencies in each of the addons' directories and call it a day. No stress about version conflict or management or w/e at the cost of some inefficiency. If the user of the addon is concerned about performance then they should be in a position to consider REAL performance optimizations like GDNative or modules. I also can't imagine why any addon would have a deep dependency graph. The only reason I can see an addon having a dependency on another is if it extends it in some way: SweetFpsController -> SweetFpsController w/ flying. In that case I would say it's up to the two projects or their users to reconcile their coupling rather than giving the responsibility to the engine.

@Two-Tone if plugin A needs version 1 of X and plugin B needs version 2 of X, I would say Godot should not try to magically fix it. Normally, version 2 of X should be backward-compatible with version 1, and that's how things should be to keep it simple (remember when I mentionned the mindset when developing plugins). If that's impossible, two simple solutions:

If you can't use latest version of X, wait for plugin A to support it and download previous version (we need the assetlib API to handle getting different versions, not just the latest one).
The plugin maker uploads it as a different plugin (because the API breaks compatibility), just like we had to when Godot 3 went out.
In the worst case you have to make it manually instead.

If not these, that would involve two versions with different path being installed (screw all references at each update), or core changes being added to the engine to alter how paths work (complicated).
I think if we can just get simple dependency management it will be quite nice already, since it would basically use the functionality and structure Godot already has currently, and do automatically things the user would have to do manually anyways.

@Zylann How are any of those options better than what I've suggested?  And there is nothing magical about searching for and appending text to a specific string.
Option 1 forces developers try and make sure their version updates don't break any other plugin that depends on it, which adds a lot of extra work on them. Who wants to develop plugins for something like that?
Option 2 forces users to not use the plugin they want because they have to either wait for the plugin to update, if it's even still being actively developed, or just not use it at all.
Option 3 once again makes things more difficult for the plugin developers.
Two versions with different paths isn't a hard problem.  If they ever update, check to see what install packages have them as a dependency, rerun the search and replace (taking into account the old version number) on all the scripts for those packages, then finally unpack the new version.
I feel like people are trying to make this out to be more complicated than it needs to be.

Proper dependency management and conflict avoidance is an important and hard issue. It's being discussed so much because of that

@Two-Tone What I meant is that you're thinking way too hard on the issue even for it being a hard issue. A lot of the conversation (including my replies, admittedly, so please excuse me as this is going to sound hypocritical) have been about what's wrong with the ideas presented, not so much about what can be done to change it and make it a better plan. The ideas being suggested can't only have downsides, can they? It seems like everyone wants their way and their way only.

Trying to just push this through without figuring out all the problems that dependency brings would be a very bad idea and just cause way more problems. This is a hard issue and it will take time.

I suggested coming up with a few simple ideas with the implication that we'd iron them out and then decide once we feel we've got a good system. I didn't think I had to explicitly point that out, but there you go.

I'm kinda annoyed at how long this conversation has gone on without any progress. Seriously.

+1 on everything in this thread - extending the plugin module and increase use of the asset lib is the way to go

After having trying out and learning Unity, I now have a clearer idea on what this should be like.
First is Unity now has a 'Package Manager' which they use to let users select which components of the engine they would like to install/uninstall. It looks like they may also use this for the Asset Store in the future.
Something like this would really help Godot to be able to "reduce bloat" while having "official addons". Unity is still super bloated even with their Package Manager while if Godot had one it could get even smaller. The "one exe" method of distribution might have to go though.
Also, normal "modules" maybe should be built as shared libraries (dll/.so files) so that they can be selected for install/uninstall in the package manager.
Projects and packages should have manifest files listing packages it depends on and Godot / the package manager would ensure that required packages are installed.

Not disagreeing with the overall merit of moving to the Package Manager approach, I have not made up my mind on that yet, but I want to point out that it has its weaknesses as well - one single executable can be way more integrated and well-tested, at least more easily.
With a Package manager, the testing surface grows exponentially for each plugin and version. This isn't that bad if the plugins are mostly small additions (like currently on the AssetLib) not hooking too deep into the engine core, but it quickly becomes so if you split up the core parts of the engine.
Unity already runs into issues, crashes and compatibility problems with different versions of their packages, their editors, or with having some specific package combinations installed (or not installed), or even with the order they were installed in.
This is a lot of additional complexity that needs to be managed well, lest we end up in dependency hell :)

Regarding dependency hell, why not take a page from something like AppImage and try to avoid dependency problems by making plugin distributors responsible for managing asset / file dependencies? Here's how I see this problem...
Plug-ins should (ideally) be self contained elements of either logic (custom editors, custom nodes, dynamically linked libraries) or assets (models, materials, fonts, textures, etc) that should have their own sets of dependencies. Trying to figure out shared dependencies is a conundrum that, while people might like it, it means a lot more effort in order to maintain it. If you were to actually allow shared dependencies, you would have to approach this problem similar to the steam runtime and maintain a bunch of different libraries, plugins and assets as their own unique packages. You would also need to store multiple of the same packages on a server for multiple versions -- it would be a maintenance nightmare and it's no surprise that modern linux distros are more or less moving away from package managers.
A plugin could essentially be a single tar archive of all of the elements needed to for that plugin to function in isolation. They should work natively on their own with all dependencies archived inside as well. When godot sees this file, it checks for a manifest file inside of the root of the archived folder. This manifest could contain basic info (name, description, godot-version, icon, authors) but should also contain things like: update-info (link to download the latest version, maintained by plugin developers) which godot can use to automatically update the plugin (with users permission), project_website (link to repository or website to let users quickly find the source of a package), and class_names (list of classes within this addon that can be extended, used for scripting.) It could also point to the addons it uses as dependencies using relative path to an internal plugin file.
If we allow plugins within plugins, then the dependency problems will be (more or less) solved. Differing version of the same library will be in different directory paths and could be treated as different 'paths' on a given fork. The problems with this system, of course, is that you end up having a lot of duplicate information on a given disk. However, when packaging a game, godot could ideally go through all the plugin directories and merge file references that match via checksum (this can included tar-represented file structures?). This way, if two files are identital, godot will optimize the eventual output binary by merging the reference to a single file.
I'm sure this is far too naive of a solution for a problem this complicated, but I was thinking about the plugin system more recently due to my custom font patch. I think the best user experience is one where they don't have to worry about dependencies, and the developers are the ones who should be concerned with managing their own dependencies. This makes it easier for the user to manage: a project can have their addons represented as a single file in the file system dock, you can right click and get additional options (extract files (for development), update plugin (for easy updates), go to website (for easy community organization)) and also reference classes within a given 'container' (extends "[path-to-addon-tar]://ClassName")
Just my 2 cents on the matter, I'm sure there are some flaws with the way I'm thinking of it as well, but it could be an interesting solution and also be a decent way to cut down on the maintenance necessity for the Godot team.

Just a 2 cents based on the previous 2 cents (does this make it 4 milles?):
I guess .tar archives would be bit unfamiliar to most users (ahem, windows users) so it might be better to use .zip archives, as they have somewhat wider support.
That being said, what about distributing plugins in Godot's own .pck format? I does have some cons (such as being unrecognized by most applications), but that might be a chance to make it more standard and used.

I guess .tar archives would be bit unfamiliar to most users (ahem, windows users) so it might be better to use .zip archives, as they have somewhat wider support.

That being said, what about distributing plugins in Godot's own .pck format? I does have some cons (such as being unrecognized by most applications), but that might be a chance to make it more standard and used.

Sure, I mostly meant .tar as an example of a generic packed archive type. It could and probably should use godot's .pck format.

Why not use ZIP since it offers compression and is a standard format?

@Calinou I agree. If we mandate that .pck files must be used, then people will have to use Godot Engine in order to pack things rather than just using whatever their preferred packing tool is to bundle up their plugin for usage. GitHub also downloads repositories as .zip files and it houses the largest collection of Godot plugins by far.

I agree here. I don't think another file type needs to be created. Zip files seem to work very well, and folks have the option of installing using the Godot interface thru the asset library, or they can download manually themselves. a .pck file does not allow that choice.

In my ideal world, I would like to have two download buttons at https://godotengine.org/download:
One sleek bare minimum download that meets reduz three requirements, and
One bloat download with all approved and and regularly maintained plugins which have surpassed some quality control, so I know their API works at least as well with the core as the API from the minimum version.
Even more ideal:
Select and mix. Upon downloading the bare minimum, I can select minimum version + Plugin "Themes". Which are collections of common plugins needed for common usecases. (so for instance: 2DPixelart, 3D FirstPersonShooter, 3D ThirdPerson, RTS, RPG, 2.5D game, 2DCutout, Rougelike, ManagementSim, Eventpackage, Shaderpackage, Scriptpackage, Assetpackace, ect). So when I start the engines essential plugins are already installed and ready to go and I know they work because they have been approved.
More advanced users could untick individual plugins within those Themes collections if they know they don't need them, or combine Plugin Themes, or add additional single plugins to Themes from the pool of approved plugins to make a custom package.
Ideally, the result would be that https://godotengine.org/download creates from those choices one download link on the fly. A download link I could copy and share with whoever, who then could with one click download the exact same Godot plugin configuration as me.
Why? Because this would be a massive time saver and quality assurance for people who work in teams or groups, for educators and teachers in schools who have to deploy on multiple computers, for people who follow tutorials and want the same config as the tutor, for individuals who are happy there is one official approved plugin by the dev team for something that is commonly used, but not by everyone, and so not have to search through Github and Reddit for days to find what they need.
A one-click bare minimum download should still exist for those who need the small size, as well as a max feature download who do not care about size but want maximum features to see what the engine can do.

A simpler solution would be to ask to install additional [official only?] plugins when running editor for first time. Currently we have option to open AssetLib if project list is empty; why not have a dialogue popup to install official plugins if running Godot for first time?

Closing in favor of https://github.com/godotengine/godot-proposals/issues/142, https://github.com/godotengine/godot-proposals/issues/577 and https://github.com/godotengine/godot-proposals/issues/607, as feature proposals are now tracked on the Godot proposals repository.


fix(katana): make sure value is within FieldElement range

Fix: #845
Doing mod by 0x9 caused some randomly generated values to be out of range. so mod by 0x8 to make sure any randomly generated value is within range.

(is there any place where we hard coded this private keys, because of this change we need to update those)

(is there any other place where we hard coded this private keys? because of this change we need to update those)

@kariy done, found few more places where old address was used (missed them because they started with 0x03e instead of 0x3e so do review that once again.

@kariy done, found few more places where old address was used (missed them because they started with 0x03e instead of 0x3e so do review that once again.

it seems the private keys changed slightly, the first account private key is now 0x18000000003000001800000000000300000000000030060018000066001
can you fix that as well? thanks

oops


Three shades of green on the homepage

summary:
We have three shades of green on the homepage that are very similar.
expected result:
Matching shades of green.
actual result:
environment (device, browser, mobile / web):
link:
screenshot (you can copy/paste images directly in):

which is the correct green? 5BCC85?

Yes, 5BCC85 is correct

Ok so I just took a look at this and it appears all the colors have changed since the above conversation happened. From what I can tell, this is how you want it to be ("correct" color pulled from logo):

The highlight color also applies to the "Save Lives" text on the front page, as well as active items in the nav bar.
If so, #490 should do the trick.


An error should be thrown when an undefined scene is started

Version

Phaser Version: v3.55.2

Operating system: MacOS

Browser: Chrome

Description

When starting a scene with an invalid key, the command just kills the current scene with no error shown.
https://github.com/photonstorm/phaser/blob/5c8ecbcf999e6f328d21884e877c9e5935d2d350/src/scene/SceneManager.js#L1154
Seems there is no handling of the negative case of that if.
Example Test Code
When no such scene was registered in the GameConfig:
this.scene.start("game-over");

Additional Information
Would a PR to console log/warn be considered?

Thanks for opening this issue, and for submitting a PR to fix it. We have merged your PR into the master branch and attributed the work to you in the Change Log. If you need to tweak the code for whatever reason please submit a new PR.


gpu support error(sampled_loss)

I'm now a little bit hesitated that I have found a similar issue before, however I can't find it now
in the file seq2seq_model.py in translate foloder, I use    with device("/gpu:0')     instead of cpu, there's an error like this :
InvalidArgumentError                      Traceback (most recent call last)
<ipython-input-10-84f5fbeb6a90> in <module>()
      1 sess = tf.InteractiveSession()
----> 2 nmt.train(sess, train_data, valid_data, EVALUATE_PER)

<ipython-input-6-ca1988aa7469> in train(self, sess, train_data, valid_data, evaluate_per)
    190
    191             _, cost = self.step(sess, train_encoder_batches[batch_i], train_decoder_batches[batch_i],
--> 192                                train_target_batches[batch_i], train_target_weight_batches[batch_i], self.train_flag)
    193
    194             # if the loss is equal to NaN(not a number), raise error

<ipython-input-6-ca1988aa7469> in step(self, sess, encoder_data, decoder_data, target_data, tw_data, train_flag)
    122                 output_feed.append(self.final_outputs[idx])
    123
--> 124         output = sess.run(output_feed, input_feed)
    125         return output
    126

/opt/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in run(self, fetches, feed_dict)
    313         `Tensor` that doesn't exist.
    314     """
--> 315     return self._run(None, fetches, feed_dict)
    316
    317   def partial_run(self, handle, fetches, feed_dict=None):

/opt/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _run(self, handle, fetches, feed_dict)
    509     # Run request and get response.
    510     results = self._do_run(handle, target_list, unique_fetches,
--> 511                            feed_dict_string)
    512
    513     # User may have fetched the same tensor multiple times, but we

/opt/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _do_run(self, handle, target_list, fetch_list, feed_dict)
    562     if handle is None:
    563       return self._do_call(_run_fn, self._session, feed_dict, fetch_list,
--> 564                            target_list)
    565     else:
    566       return self._do_call(_prun_fn, self._session, handle, feed_dict,

/opt/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _do_call(self, fn, *args)
    584         # pylint: disable=protected-access
    585         raise errors._make_specific_exception(node_def, op, error_message,
--> 586                                               e.code)
    587         # pylint: enable=protected-access
    588       six.reraise(e_type, e_value, e_traceback)

InvalidArgumentError: Cannot assign a device to node 'sequence_loss_by_example/sampled_softmax_loss_39/embedding_lookup_1': Could not satisfy explicit device specification '/device:GPU:0'
	 [[Node: sequence_loss_by_example/sampled_softmax_loss_39/embedding_lookup_1 = Gather[Tindices=DT_INT64, Tparams=DT_FLOAT, validate_indices=true, _device="/device:GPU:0"](proj_b/read, sequence_loss_by_example/sampled_softmax_loss_39/concat)]]
Caused by op u'sequence_loss_by_example/sampled_softmax_loss_39/embedding_lookup_1', defined at:
  File "/opt/anaconda2/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/opt/anaconda2/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/opt/anaconda2/lib/python2.7/site-packages/ipykernel/__main__.py", line 3, in <module>
    app.launch_new_instance()
  File "/opt/anaconda2/lib/python2.7/site-packages/traitlets/config/application.py", line 592, in launch_instance
    app.start()
  File "/opt/anaconda2/lib/python2.7/site-packages/ipykernel/kernelapp.py", line 403, in start
    ioloop.IOLoop.instance().start()
  File "/opt/anaconda2/lib/python2.7/site-packages/zmq/eventloop/ioloop.py", line 151, in start
    super(ZMQIOLoop, self).start()
  File "/opt/anaconda2/lib/python2.7/site-packages/tornado/ioloop.py", line 866, in start
    handler_func(fd_obj, events)
  File "/opt/anaconda2/lib/python2.7/site-packages/tornado/stack_context.py", line 275, in null_wrapper
    return fn(*args, **kwargs)
  File "/opt/anaconda2/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 433, in _handle_events
    self._handle_recv()
  File "/opt/anaconda2/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 465, in _handle_recv
    self._run_callback(callback, msg)
  File "/opt/anaconda2/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 407, in _run_callback
    callback(*args, **kwargs)
  File "/opt/anaconda2/lib/python2.7/site-packages/tornado/stack_context.py", line 275, in null_wrapper
    return fn(*args, **kwargs)
  File "/opt/anaconda2/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 260, in dispatcher
    return self.dispatch_shell(stream, msg)
  File "/opt/anaconda2/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 212, in dispatch_shell
    handler(stream, idents, msg)
  File "/opt/anaconda2/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 370, in execute_request
    user_expressions, allow_stdin)
  File "/opt/anaconda2/lib/python2.7/site-packages/ipykernel/ipkernel.py", line 175, in do_execute
    shell.run_cell(code, store_history=store_history, silent=silent)
  File "/opt/anaconda2/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2902, in run_cell
    interactivity=interactivity, compiler=compiler, result=result)
  File "/opt/anaconda2/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 3006, in run_ast_nodes
    if self.run_code(code, result):
  File "/opt/anaconda2/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 3066, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-9-bfcd832d993d>", line 2, in <module>
    BATCH_SIZE, NUM_SAMPLES, MAX_GRADIENT_NORM, LEARNING_RATE, decay_factor = LEARNING_RATE_DECAY_FACTOR)
  File "<ipython-input-6-ca1988aa7469>", line 88, in __init__
    self.loss_sequence = seq2seq.sequence_loss_by_example(self.outputs, self.targets, self.tw, self.target_voca, softmax_loss_function = softmax_loss_function)
  File "seq2seq_mod070_ver5.py", line 974, in sequence_loss_by_example
    crossent = softmax_loss_function(logit, target)
  File "<ipython-input-6-ca1988aa7469>", line 66, in sampled_loss
    self.target_voca)
  File "/opt/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/nn.py", line 865, in sampled_softmax_loss
    name=name)
  File "/opt/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/nn.py", line 661, in _compute_sampled_logits
    all_b = embedding_ops.embedding_lookup(biases, all_ids)
  File "/opt/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/embedding_ops.py", line 86, in embedding_lookup
    validate_indices=validate_indices)
  File "/opt/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 423, in gather
    validate_indices=validate_indices, name=name)
  File "/opt/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/op_def_library.py", line 655, in apply_op
    op_def=op_def)
  File "/opt/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2040, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/opt/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1087, in __init__
    self._traceback = _extract_stack()

what's the problem with this?
I heard that there are many device related things modified in release 0.8.0 just because of many of ops are coded with cuda(?)  then why not this part, I mean sampled_loss part in seq2seq_model.py

Could you please include exact instructions on how you are running this (command line usage) so I can reproduce the problem. Thank you!

Actually the problem occured when I run my own code which is very similar to translate.py + seq2seq_model.py. Im now using ver 0.7.1 . I found that there is no error when i change all of the "/cpu:0" to "/gpu:0" (please refer to below code) from seq2seq_model.py but my own code . with my code when they are not changed, code runs well, but when changed, outputs the error (error above)
if num_samples>0 and num_samples<self.target_vocab_size:
with tf.device("/cpu:0"):
w = tf.get_variable("proj_w", ..........
.......
def sampled_loss(inputs, labels):
with tf.device("/cpu:0"):
...

can you explain what does the error exactly mean?
and I have one more question is that : even it works with those tf.device("/gpu:0"), why there are still using tf.device("/cpu:0")  (files in the path tensorflow/python/ops ), any other reason?

At this point I'd recommend trying things on 0.9 since 0.7.1 is quite old at this point. If you want further help, a complete reproducible test case would be necessary.

Automatically closing due to lack of recent activity. Please reopen when further information becomes available.


Create data directory before writing to it

Fixes #3

Current dependencies on/for this PR:

main

PR #9   👈

This comment was auto-generated by Graphite.

Thanks :-)