How to ignore ports without skipping actual ones?

Prerequisites

[X ] Have you checked for an existing issue describing the issue?

I believe that this isn't possible yet, so seeking for advice or need to create a feature request.
I have a repo with custom ports which aren't and won't be in the FreeBSD ports tree. One port is not selfcontained:
root@deblndw013x3j:~/ports/dns/register-hostnames
# cat Makefile
PORTNAME=               register-hostnames
PORTVERSION=    20210622
CATEGORIES=             dns
DISTFILES=

MAINTAINER=             michael-o
COMMENT=                Dynamically register hostnames in Active Directory

NO_ARCH=                yes
NO_BUILD=               yes
NO_TEST=                yes

RUN_DEPENDS=    kinit:security/krb5 nsupdate:dns/bind-tools

WRKSRC=                 ${FILESDIR}

do-install:
        @${MKDIR} ${STAGEDIR}/bin
        @${INSTALL_SCRIPT} ${WRKSRC}/register-hostnames ${STAGEDIR}/bin

.include <bsd.port.mk>

My jails have a base set of ports which include both dependencies from RUN_DEPENDS. That repos is build with:
poudriere bulk -j 122-release-amd64 -z ldadw-base -f /usr/local/etc/poudriere.d/122-release-amd64-default-ldadw-base-ports

Now I want to build this custom tree:
poudriere bulk -j 122-release-amd64 -O ldadw-custom -z ldadw-custom -f /usr/local/etc/poudriere.d/122-release-amd64-default-ldadw-custom-ports

This of course will trigger the build of both runtime dependencies which I don't want since those are already in ldadw-base. So I have added both ports to a blacklist file and poudriere tells me:
[00:00:09] Skipped ports: dns/register-hostnames
[00:00:09] Ignored ports: security/krb5 dns/bind-tools
[122-release-amd64-default-ldadw-custom] [2021-06-22T12:25:16] [committing:] Queued: 3  Built: 0  Failed: 0  Skipped: 1  Ignored: 2  Tobuild: 0   Time: 00:00:07
[00:00:09] Logs: /var/poudriere/data/logs/bulk/122-release-amd64-default-ldadw-custom/2021-06-22T12:25:16
[00:00:09] Cleaning up
[00:00:09] Unmounting file systems

My port is skipped. My question: Is there anyway to tell poudriere to continue with the port and assume that the dependencies will be provided at runtime? This of course does not apply to build dependencies, but this could also be solved by compiling them, but not exposing them in the repo metadata when hosted for pkg.
Any help is appreciated. The only workaround I see is to remove the RUN_DEPENDS or move this port to ldadw-base, but other maybe upcoming custom ports will suffer from the same issue.

Why are you making 2 separate repositories? Poudriere won't support the pkg idea of multiple remote repositories. Any packages needed for runtime dependencies need to be present and built in the same repository.

Why are you making 2 separate repositories? Poudriere won't support the pkg idea of multiple remote repositories. Any packages needed for runtime dependencies need to be present and built in the same repository.

I will try to expand. While I could add this package into by base repository this would only solve the symptom not the problem. My idea is to have a base repo for all jails. Then, repos per jail (use case) type on top of base which may depend on packages in base or any other packages from the official ports tree. So to speak a layered design. I want to avoid recompiling ports which aren't necessary. My idea is very similar to one of @rootwyrn in #534.
Theoretically one could do the following:

Assume that repo base has been build with this id: 122-release-amd64-default-ldadw-base
I instruct poudriere to nullfs mount this local repo into the jail 122-release-amd64-default-ldadw-custom and poudriere will try to use these packages first and then build the rest from ports:

poudriere bulk -j 122-release-amd64 -O ldadw-custom -z ldadw-custom -M `122-release-amd64-default-ldadw-base` -f /usr/local/etc/poudriere.d/122-release-amd64-default-ldadw-custom-ports

where -M would mount previously prepared local repos in /usr/local/etc/pkg/repos in the jail. Alternatively, there would be a directory for this run which could contain repos to include.
With that one can avoid duplication and introduce a layered design. WDYT?

It seems like that https://github.com/freebsd/poudriere/pull/797 achieves a, simpler, but same goal though limited for some reason to official FreeBSD repo and the arbitrary one.

I was able to very uglily trick poudriere with a modified version of #797:

Modified common.sh for:

    cat >> "${MASTERMNT}/etc/pkg/poudriere.conf" <<-EOF
    FreeBSD: {
            url: ${packagesite},
            mirror_type: "NONE",
            pubkey: "NONE",
            signature_type: "NONE"
    }
    EOF

Started the build:

poudriere bulk -j 122-release-amd64 -O ldadw-custom -z ldadw-custom -f /usr/local/etc/poudriere.d/122-release-amd64-default-ldadw-custom-ports -b https://deblndw011x.ad001.siemens.net/FreeBSD/packages/FreeBSD:12:amd64/ldadw-base -c

but it still wanted to build too many packages, then I did the following:
root@deblndw011x:/usr/local/etc/poudriere.d
# cp -p 122-release-amd64-default-ldadw-base-make.conf 122-release-amd64-default-ldadw-custom-make.conf
# cp -rp 122-release-amd64-default-ldadw-base-options 122-release-amd64-default-ldadw-custom-options

It is not fully clear why I need to fake this overlay and why pkg simply doesn't install both RUN_DEPENDS from the repo, but insists on the default options config. I guess it does not take it uses its own tree.
Is this a starting point we can discuss for layering? WDYT?

In https://blog.shatow.net/posts/2013-07-21-managing-role-based-freebsd-servers-with-meta-packages-and-poudriere/ I cover how I do this. I just make a separate meta package for each jail and then can use 1 unified repository. Of course this only works if you do not have packages with conflicting options.

Thanks, I'll look into it after my vacation.

In https://blog.shatow.net/posts/2013-07-21-managing-role-based-freebsd-servers-with-meta-packages-and-poudriere/ I cover how I do this. I just make a separate meta package for each jail and then can use 1 unified repository. Of course this only works if you do not have packages with conflicting options.

I was caught up again by this. Read your post. I understand your approach, but a see a few issues:

Since you use metaports the ports you actually need are then automatic ports which means subject to deletion by pkg autoremove. This I don't want.
I have a base set of ports I deploy on all servers, jails, etc. Recompiling them means also merge base options and specific options. That is rather a waste.

For one usecase I have now tried -b URL:
[00:00:07] Package fetch: Looking for missing packages to fetch from https://deblndw011x.ad001.siemens.net/FreeBSD/packages/${ABI}/ldadw-base

This works, but since the repo config done in common.sh I need to apply: https://github.com/freebsd/poudriere/issues/883#issuecomment-866791715 since URL does not gurantee that it is a mirror of FreeBSD, but could also be a custom one. Maybe I should create a followup ticket where a URL should not expect mirror, signature, etc. like the canonical repo?

The build has now been completed. Well, packages have been downloaded, but many which are available at the supplied repo are still being rebuild. This I don't understand.

The build has now been completed. Well, packages have been downloaded, but many which are available at the supplied repo are still being rebuild. This I don't understand.

Found it. I still need to merge the port options from the repo to be used and the port options for this bulk. I will create an issue.

In https://blog.shatow.net/posts/2013-07-21-managing-role-based-freebsd-servers-with-meta-packages-and-poudriere/ I cover how I do this. I just make a separate meta package for each jail and then can use 1 unified repository. Of course this only works if you do not have packages with conflicting options.

I was caught up again by this. Read your post. I understand your approach, but a see a few issues:

Since you use metaports the ports you actually need are then automatic ports which means subject to deletion by pkg autoremove. This I don't want.

To clarify something the dependencies are automatic but do not show in pkg autoremove because the metaports are non-automatic and still installed. So pkg autoremove shows nothing typically. pkg noauto is quite clean and only shows pkg and the metaports.
The leaf packages only show up in pkg autoremove if I trim them from the metaport and upgrade it.
Here is an example from some of my jails. The matrix jail metaport used to depend on conduit but no longer does so it wants to delete it.
samba# pkg noauto
pkg-1.18.4
local-fileserver-meta-20210811_12
local-jail-meta-20220925_7
local-postfix-relay-meta-20210527_18,2
samba# pkg autoremove -n
Checking integrity... done (0 conflicting)
Nothing to do.

roundcube# pkg noauto
pkg-1.18.4
local-jail-meta-20220925_7
local-postfix-relay-meta-20210527_18,2
local-webmail-meta-20220802_3
roundcube# pkg autoremove -n
Checking integrity... done (0 conflicting)
Nothing to do.

prometheus# pkg noauto
pkg-1.18.4
prometheus-2.36.0_5
local-jail-meta-20220925_7
local-postfix-relay-meta-20210527_18,2
prometheus# pkg autoremove -n
Checking integrity... done (0 conflicting)
Nothing to do.

matrix# pkg noauto
pkg-1.18.4
local-jail-meta-20220925_7
local-matrix-meta-20221030_4
local-postfix-relay-meta-20210527_18,2
matrix# pkg autoremove -n
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 3 packages:

Installed packages to be REMOVED:
        matrix-conduit: 0.4.0_2
        py39-pip: 22.2.2
        py39-virtualenv: 16.7.5

Number of packages to be removed: 3

The operation will free 69 MiB.

For what it's worth I am trying to wrap up #822 this week. It will significantly reduce port rebuilds to only when a version or PORTREVISION change happens (and sane things like OPTIONS). It won't force rebuild everything so much anymore.

For one usecase I have now tried -b URL:

Yeah that makes sense as a workaround for what you want.

[00:00:07] Package fetch: Looking for missing packages to fetch from https://deblndw011x.ad001.siemens.net/FreeBSD/packages/${ABI}/ldadw-base
Bootstrapping pkg from https://deblndw011x.ad001.siemens.net/FreeBSD/packages/FreeBSD:12:amd64/ldadw-base, please wait...

This works, but since the repo config done in common.sh I need to apply: #883 (comment) since URL does not gurantee that it is a mirror of FreeBSD, but could also be a custom one. Maybe I should create a followup ticket where a URL should not expect mirror, signature, etc. like the canonical repo?

Definitely please open a separate issue so it can be fixed and tracked.

That's a make thing Poudriere can't do much about.

That's a make thing Poudriere can't do much about.

But it does work when I am in the jail or just doing the from a host. I will report anyway for others which might trip over. Hopefully, you can explain why BSD make (?) fails here in this setup.

That's a make thing Poudriere can't do much about.

But it does work when I am in the jail or just doing the from a host. I will report anyway for others which might trip over. Hopefully, you can explain why BSD make (?) fails here in this setup.

Forget that, this is a problem in the build scripts of the OpenJDK, bad use of printf. I have created a patch and will create a bug with the OpenJDK 8 port.

FTR: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=267690

In https://blog.shatow.net/posts/2013-07-21-managing-role-based-freebsd-servers-with-meta-packages-and-poudriere/ I cover how I do this. I just make a separate meta package for each jail and then can use 1 unified repository. Of course this only works if you do not have packages with conflicting options.

I was caught up again by this. Read your post. I understand your approach, but a see a few issues:

Since you use metaports the ports you actually need are then automatic ports which means subject to deletion by pkg autoremove. This I don't want.

To clarify something the dependencies are automatic but do not show in pkg autoremove because the metaports are non-automatic and still installed. So pkg autoremove shows nothing typically. pkg noauto is quite clean and only shows pkg and the metaports.
The leaf packages only show up in pkg autoremove if I trim them from the metaport and upgrade it.
Here is an example from some of my jails. The matrix jail metaport used to depend on conduit but no longer does so it wants to delete it.
samba# pkg noauto
pkg-1.18.4
local-fileserver-meta-20210811_12
local-jail-meta-20220925_7
local-postfix-relay-meta-20210527_18,2
samba# pkg autoremove -n
Checking integrity... done (0 conflicting)
Nothing to do.

roundcube# pkg noauto
pkg-1.18.4
local-jail-meta-20220925_7
local-postfix-relay-meta-20210527_18,2
local-webmail-meta-20220802_3
roundcube# pkg autoremove -n
Checking integrity... done (0 conflicting)
Nothing to do.

prometheus# pkg noauto
pkg-1.18.4
prometheus-2.36.0_5
local-jail-meta-20220925_7
local-postfix-relay-meta-20210527_18,2
prometheus# pkg autoremove -n
Checking integrity... done (0 conflicting)
Nothing to do.

matrix# pkg noauto
pkg-1.18.4
local-jail-meta-20220925_7
local-matrix-meta-20221030_4
local-postfix-relay-meta-20210527_18,2
matrix# pkg autoremove -n
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 3 packages:

Installed packages to be REMOVED:
        matrix-conduit: 0.4.0_2
        py39-pip: 22.2.2
        py39-virtualenv: 16.7.5

Number of packages to be removed: 3

The operation will free 69 MiB.

For what it's worth I am trying to wrap up #822 this week. It will significantly reduce port rebuilds to only when a version or PORTREVISION change happens (and sane things like OPTIONS). It won't force rebuild everything so much anymore.

This adds up. I have, indeed, a very similar approach, but without the meta ports, so called "packages classes". I will consider this in the next round of improvements. The downside is that you have an extra layer which simplifies installation, but the actually ports you use aren't direct anymore and these are which I document. In anycase, both approaches work and have pros and cons. Thanks for sharing.


Reintroduce Error checking for container creation

This PR reintroduces the error check merged in #337. It seems to have disappeared between then and the most recent commit.

Please sign the commit

@amohoste could you please sign the commit?

@amohoste


[CLI] wandb: ERROR Dropped streaming file chunk

Describe the bug
I'm running a sweep. I started 6 agents in parallel with that sweep id. All going fine. Then I started another 6 agents with the same sweep id in a different parallel process. Working fine at start, but after a while, I started seeing the following error on every log step:
wandb: ERROR Dropped streaming file chunk (see wandb/debug-internal.log) ERROR:root:dropped chunk 404 Client Error: Not Found for url: https://api.wandb.ai/files/[ENTITY]/[PROJECT]/c60hxx3a/file_stream NoneType: None
The file wandb/debug-internal.log is blank.
Any ideas on what might be going wrong?
Operating System

OS: Ubuntu 18.04.4 LT
Environment: Python
Python Version: 3.9.2

Hey @sophiaas , thanks for reporting the issue. Could you please share the wandb version you're using? This was a known issue in a previous wandb version, hence, my reason for asking.
Also, can you try syncing the runs, using wandb sync and check if this resolves the issue? As it seems like wandb tried to upload a file that wasn’t there/available. To get to the root cause as to why this is happening - we would need more logs, I see you mentioned that debug-internal.log was empty. If possible, could you share other log files - for instance: debug.log, etc.

@sophiaas when you start your agent be sure to specify the fully qualified sweep name, i.e. USERNAME/PROJECT/SWEEP_ID that's my best guess as to what might be going wrong.

I met the same ERROR. When I accidentally deleted the runs on webpage, I get the same ERROR in CLI "ERROR:root:dropped chunk 404 Client Error: Not Found for url", and I can't stop it by CTRL C. My wandb version is 0.11.2.

@vanpelt I think this was my issue. I'm making sure to provide all of those details now and am no longer running into the error. Thanks!


Sort with particular date without time

1.In Ersole, sorting logs by day without including hours is not functioning correctly.
2.There is also no option to sort loggers by two specific dates.

@krishna-celllabs With the updated version, the date and time are combined in the same select element. When a date is selected, the time will automatically be set to 00:00:00


[BUG]Why is the text displayed on a single line

In the latest version, the text can only be displayed on one line with a scroll bar underneath, may I ask why?

Please post the actual markdown file, as well as the version information:
Versions (please complete the following information):

OS: [e.g. iOS]
Obsidian version: [e.g. v0.12.4]
Plugin version: [e.g. v1.4.9]

The screenshot is from macos 12.5, but the ios system（17.2.1） is the same
obsidian version is the latest version
Plugin version: the latest version

It is also helpful if you post your markdown file that exhibits this problem.

==dodge== #noun
↓↓
to move quickly to avoid someone or something

Sorry, I should have been clearer... please attach your markdown file using:

Also, please send a screenshot of these settings:

try this
-  ==proofread== #verb
??
	to read through something that is written or printed in order to correct any mistakes in it
<audio controls src="https://ssl.gstatic.com/dictionary
/static/sounds/oxford/proofread--_gb_1.mp3" ></audio>
※ Proofread documents

The first step in investigating this issue is to reproduce it.
If you want me to look into this, you will need to do the two things I asked yesterday. I do not want to use the text that you pasted into your post:

I want the actual file as I asked yesterday, as well as the screenshot.

OK,That is it.
test.md

Hi
Thanks for that. A few points...

The solution is to remove the tab character at the start of "to read through something that" line

After I removed the tab character, the card renders as expected

The tab character had the unexpected effect you noticed, because that is a characteristic of Obsidian itself (i.e. nothing to do with this plugin). I didn't know about this until I started looking now, but it is in the Obsidian documentation:
https://jackiegeek.gitee.io/obsidian-docs/en/How to/Format your notes/#code-blocks

The change you noticed was probably due to a recent code change to fix #800. Prior to that change, leading "whitespace" (spaces, tabs etc) were stripped out, and therefore Obsidian didn't do its special formatting for tabs.

Cheers
Ronny

Thank you, Ronny.
So this is an Obsidian bug? But I have thousands of cards and I have to delete thousands of tabs to fix the problem. I'm backing out of 1.10.5 now.

Hi hicaicai
I would say that it's a "characteristic" of Obsidian rather than a bug... they intentionally treat a tab at the start of a line as an indicator of a code block.
Options

I'm sure it would be a pain, but to make the fixes manually, probably the easiest way would be during a review. Use the edit card feature...

Note that after clicking "save" there is no immediate change in how the card is displayed.

For an automated method you could try one of the "regular expression" plugins. This means you could remove all tabs that are at start of a line. But be careful and take a backup of the vault before making wholesale automated changes.

Wait for a code change to the plugin, but given the huge backlog and few maintainers, I wouldn't expect anything before 6 or 12 months.

Cheers
Ronny

If I need a format with tab characters, I need to wait for the plug-in to be upgraded, right?
Ok，I get it. Thank you again, Ronny.

Yes, that's correct unfortunately.

Yes, that's correct unfortunately.

I also noticed another issue, which is that the settings interface can be scrolled horizontally.

Hi, is this still an issue or can this be closed?


Question: Is there a way to update the device list?

I got some new echos on black friday and added them like normal to my alexa account.
But they are not showing up in home assistant. Is it possible to update the device list?

Oof, nevermind. A restart solved it.


API for adding documents to index with optional Id

Is your feature request related to a problem? Please describe.
I am trying to add document to index using method like СreateAsync, CreateDocumentAsync and etc, but all of them require Id as a mandatory parameter for query.
However Elasticsearch API has a way to index document using autogenerated ID (https://www.elastic.co/guide/en/elasticsearch/reference/7.11/docs-index_.html).
I don't need my own ID because data is just logs and I would not like to implement mechanisms to avoid ID duplication when hundreds of clients will add data to index simultaneously.
Describe the solution you'd like
I would like to have overloaded methods which will allow me to not specify Id. It should use Elastic API "POST /<target>/_doc/"
ElasticClient.Create
ElasticClient.CreateAsync
ElasticClient.CreateDocument
ElasticClient.CreateDocumentAsync
ElasticClient.Index
ElasticClient.IndexAsync
Describe alternatives you've considered
I have found a workaround using ElasticLowLevelClient, but it not convenient and looks like hack.

Hi, @stivx0.
The Create* methods map one-to-one with the create API on the server which explicitly requires an ID to be provided. Generally, you want to use the Index* methods which do not require an ID in the path.
If you've configured your client with a default index or a mapping for a type, you can use IndexDocumentAsync.
var client = new ElasticClient(new ConnectionSettings().DefaultMappingFor<Music>(m => m.IndexName("music")));
var doc = new Music { Name = "TEST" };
var indexResponse = await client.IndexDocumentAsync(doc);

If you prefer to provide the index name on the request directly you can use IndexAsync
var client = new ElasticClient();
var doc = new Music { Name = "TEST" };
var indexResponse = client.IndexAsync(doc, i => i.Index("music"));

Does that help?

Thanks a lot, that works!
I have seen PUT in description "PUT request to the index API, read more about this API online" and unfortunatelly even have not tried to test it.


JS-Capstone: Counting comments

Activities Done:

[x] Added a counter to the comment title to count number of comments.

Great. I'll make the adjustments. Amazing insight thanks.


Revert "zfs cannot be distributed. Disabling it in the isos."

ZFS's popularity is growing, and not including it by default is a
bit frustrating. On top of that, the base iso includes ZFS
anyway due to other packages depending upon it.
I think we're in the clear to do this on the basis that Oracle
probably doesn't care, it is probably fine (the SFLC agrees) and
we're a small fish. If a copyright holder asks us to, we can
definitely revert it again.
This reverts commit 33d07c7ea9f0d05d47dd4b41ded3b261380acfac.
Motivation for this change
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[ ] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Assured whether relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

@armijnhemel says:

"if we ship the ZFS source code, then for all practical purposes the GPL has been fullfilled in my opinion. This also seems to be the opinion of Ubuntu and SFLC."

and we do distribute the source code via binary cache, and the build instructions by virtue of having nixpkgs in the image.

I believe this is safe for 18.09, too.

Backported in a0c7949780b88d991e1b52e3c160f4233b7b5018


Does not work with React-Native: relies on http module

In React-Native, I keep getting the error:
"Unable to resolve module http
When I try to install https://www.npmjs.com/package/http via 'npm install http', I get an empty folder with package.json file.
Any idea how to resolve this?

This is caused by a deprecated feature which will be removed in next version

Removed depency of http


Public launch minimum requirements

An upcoming public launch has been discussed in recent group meetings. The codebase is open already, of course, so this would mean promoting the existence of Pangeo Forge on Twitter, Discourse, and other public forums, and encouraging new users to the try the platform. Here's a list of items culled from our various Issue and PR trackers which we either definitely need to, or might want to, complete before that happens.
Priorities reflected below are to ensure that new users will:

Understand how to write and submit a recipe
Be able to have that recipe run in a Columbia-owned Bakery
Can get data back out via an approachable and documented STAC interface
Don't hit any major friction points along the way

I welcome any and all feedback on these points: additions, subtractions, edits, etc.
Bug fixes

[ ] https://github.com/pangeo-forge/pangeo-forge-recipes/issues/200

Possibly interferes with our ability to register test runs of recipes that use subset_inputs feature?

Features

[ ] https://github.com/pangeo-forge/pangeo-forge-recipes/issues/79 ?

Prioritization depends on how frequently we expect users to need a solution for this.

Infrastructure

[ ] https://github.com/pangeo-forge/pangeo-forge-gcs-bakery/issues/19
[ ] https://github.com/pangeo-forge/pangeo-forge-catalog/issues/1
[ ] https://github.com/pangeo-forge/staged-recipes/issues/67

Some of the questions here are already resolved, but others are not, such as those relating to https://github.com/pangeo-forge/pangeo-forge-recipes/issues/36

[ ] https://github.com/pangeo-forge/roadmap/pull/31 ?

Docs

[ ] https://github.com/pangeo-forge/pangeo-forge-recipes/pull/71
[ ] https://github.com/pangeo-forge/pangeo-forge-recipes/issues/189
[ ] https://github.com/pangeo-forge/pangeo-forge-recipes/issues/188

cc @pangeo-forge/dev-team

Thanks for putting this together Charles! So helpful.
We already have a minimal working product. The bakeries can run recipes and store data. Yes, there are many bugs to fix and new features to add, but the base functionality is there.
So for me the, absolute "must haves" for the public launch are:

https://github.com/pangeo-forge/pangeo-forge-recipes/pull/71 - without this, people won't know how to participate.
https://github.com/pangeo-forge/pangeo-forge-gcs-bakery/issues/19 - we can't launch without a bakery in production!
https://github.com/pangeo-forge/pangeo-forge-catalog/issues/1 - without a catalog of some sort, it's impossible to know what data we have stored

So I would put laser focus on those three issues. Once we solve them, I think we can make the big announcement that Pangeo Forge is open for business (with the caveat that there are still many kinks to work out).

For my reference, from https://github.com/pangeo-forge/staged-recipes/pull/74#issuecomment-916411092

The manual validation of the recipe that @wildintellect and @sharkinsspatial did in https://github.com/sharkinsspatial/zarr_examples/blob/main/imerg_sample.ipynb was a very useful QC point. We need to make that part of the process! Once the test recipe has been run, a bot should post a link to the data and encourage the use to play with it. Maybe even provide a binder link to open it up directly.


HubSpot API Key Sunset: Private App support

HubSpot sunset the API key by 30 November 2022. It is replaced by a private key, where you can specify access on a fine-grained level.
https://developers.hubspot.com/changelog/upcoming-api-key-sunset
Migration Guide
Acceptance criteria:

HubSpot.NET supports Private Key for HubSpot API access

I have made the changes and they work with most tests for private key.
For some reason I cannot create the pull request, so I have attached the patch file (with TXT extension).
Anyone who can help?
Issue_18_HubSpot_private_key.txt

Sorry, this diff file has too many conflicts - every files clashes with the primary branch trunk.
I'm guessing you've made these changes against a branch that isn't the primary branch. Please try again using trunk as as your branch.

Sorry. Can see that I picked the wrong branch.


Ghost can only find one page template when several are available

Issue Summary
A summary of the issue and the browser/OS environment in which it occurs.
According to the documentation, Ghost will pick up a template of the form page-[whatever].hbs and use this instead of the default page for slug [whatever].  It does this for one of several such templates, but not for the others.  In other words, one slug gets its template from page-whatever.hbs, whereas the others get it from page.hbs.
To Reproduce
Create several identical templates, e.g., page-1.hbs, page-2.hbs, etc.
Create slugs in the Ghost navigation with names 1, 2, etc.
Create posts converted into pages through the Ghost editor, pointing at slugs 1,2,3.
Any other info e.g. Why do you consider this to be a bug?
Given that the documentation is opaque on this matter, it could be a 'feature' instead of a bug.  However, it would obviously be hugely helpful if it worked.
Ghost Version: 2.6.0
Node Version: 8.11.3
Browser/OS: Chrome 70.0.3538.102 (Official Build) (64-bit)
Database: default (presumably MySQL).

Following the steps you've given I couldn't replicate this issue unfortunately.

Create slugs in the Ghost navigation with names 1, 2, etc.

What do you mean by this? Slugs are created in the settings sidebar of the editor - do you think it's possible you've used tags, rather than changing the slugs?

I know the difference between tags and slugs, obviously.
What operating system are you using?

Cool :relaxed: I'm trying to replicate it on Linux! What are you running?
What is the contents of your page-1.hbs, page-2.hbs etc..? Have you run the theme through gscan?

Sorry, this should have been at the top.  I'm running Windows 10 Pro 64bit.
There are a number of apparently related issues, all to do with no finding files.  Is Ghost supposed to be able to run directly on Windows, or does it need the WSL?

You shouldn't need WSL, no.
As far as this issues goes, it can't be reproduced so I'd suggest going to the community forums and asking for help building themes there - if you do manage to find a reproducible bug, feel free to raise an issue with everything we need to replicate it!
I'm going to close this issue now - I hope you get your problem fixed through the forum :smile:

, it can't be reproduced
It can't be reproduced on Linux, but it can on Windows.


Updating Object Properties

Challenge Updating Object Properties has an issue.
User Agent is: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36.
Please describe how to reproduce this issue, and include links to screenshots if possible.
My code:

// Example
var ourDog = {
  "name": "Camper",
  "legs": 4,
  "tails": 1,
  "friends": ["everything!"]
};

ourDog.name = "Happy Camper";

// Setup
var myDog = {
  "name": "Coder",
  "legs": 4,
  "tails": 1,
  "friends": ["Free Code Camp Campers"]
};

// Only change code below this line.

myDog["name"] = "Happy Camper";

The instruction states:

Update the myDog object's name property. Let's change her name from "Coder" to "Happy Coder".

This tracker is fo bugs only, if you have problem with your code please consider asking for help on
Help Chatroom or Help JS, alternatively look on our official forum

Happy coding 👍

@Marmiz why the symbols being given,my code is correct,the output is fight!

@Marmiz why the symbols being given,my code is correct,the output is right!

@siguniang the editor we use gives JS suggestions on how to write your code. The symbol over your code is a suggestion to use dot notation rather than brackets because you don't need spaces in the property name.

Your code isn't correct according to the instructions. The use of "Happy Camper" is just an example.
This is a standard message notifying you that this issue seems to be a request for help. Instead of asking for help here, please click the "help" button on the challenge on Free Code Camp, which will take you to the help chatroom for that specific challenge. You can also view our full list of official chatrooms.
If you think I'm wrong in closing this issue, please reopen it and add further clarification. Thank you, and happy coding!

@Marmiz @erictleung has been thought to be quotation marks being given,did nont see the word is wrong,has now been resolved.thank you!


pkgs/README: recommend by-name in versioning section

Asked @SuperSandro2000 for a review because we discussed about this topic in a different context.
Description of changes
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

This doesn't work very well in general, see https://github.com/NixOS/nixpkgs/tree/master/pkgs/by-name#recommendation-for-new-packages-with-multiple-versions

This doesn't work very well in general, see master/pkgs/by-name/README.md#recommendation-for-new-packages-with-multiple-versions
Related: discourse.nixos.org/t/rfc-for-multiversioned-packages/50532

I understand there are multiple ways of adding versioned packages, but I don't understand what exactly is it that "this doesn't work"? Are you talking about sharing Nix expressions between the different versions? Which will not work if you also have to make directories refer to each other, and that is not allowed? I can modify the docs to mention that issue with pkgs/by-name...
In general, there is some duplicate content between the pkgs' README.md and pkgs/by-name README...

Which will not work if you also have to make directories refer to each other, and that is not allowed?

Yes, relative paths are not a stable interface across package dirs.

It's also generally problematic to repeat yourself, so we shouldn't recommend that either. We should generally not recommend bad practices.
Given that there isn't one best way to do this, I don't think this document should be setting rules on where files should be located. The rules on the attr name should remain though.

It's also generally problematic to repeat yourself, so we shouldn't recommend that either. We should generally not recommend bad practices.
Given that there isn't one best way to do this, I don't think this document should be setting rules on where files should be located. The rules on the attr name should remain though.

OK I understand. Indeed the best practices there are debatable... I still think a change to this paragraph would be beneficial. I pushed something.

The wording is a bit weird and confusing but this is better and necessary.

Thanks for the suggestion. Done!


Make it possible to install beta releases using this Brew tap

Currently, this Brew tap only contains the stable builds for 1Password CLI.
We should make it possible to opt-in to installing beta builds too:
brew install --cask 1password/tap/1password-cli-beta

You can achieve this by doing:
brew tap homebrew/cask-versions
brew install 1password-cli-beta


add ShardWidth to index info

Description
Client libraries sometimes need to know shard width for things like imports. It would be great instead of being hard-coded if the client's interrogated the Pilosa server to get this information.
Currently ShardWidth is a constant in Pilosa's code, but there's no reason this has to always be the case. One could imagine choosing different shard width's per index depending on the data and particular tradeoffs one is interested in.
I think that ShardWidth should be part of the response to a schema request - specifically in the IndexInfo so that client's are ready to consider ShardWidth as a per-index thing even if it's still a constant in Pilosa for now.
Success criteria (What criteria will consider this ticket closeable?)
ShardWidth can be changed server side and client's work without update or change to configuration.

I've pushed a fix for this, but this may be problematic when an Index instance is created on the client-side. syncSchema is optional to call when the schema is exist on the server-side, so it is possible that an Index wouldn't know what the slice width is before an import. A possible workaround is, the client-side import code may call the schema endpoint to get the slice width if it is not already known.


flow compatible libdefs found in flow-typed for the above untyped dependencies

Im getting this error when running yarn on a new project:
!! No<EMAIL_ADDRESS>libdefs found in flow-typed for the above untyped dependencies !!
Node version: 9.4
OS: Mac OS 10.12.6
Complete error:
yarn install v1.3.2
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
info<EMAIL_ADDRESS>The platform "darwin" is incompatible with this module.
info<EMAIL_ADDRESS>is an optional dependency and failed compatibility check. Excluding it from installation.
info<EMAIL_ADDRESS>The platform "darwin" is incompatible with this module.
info<EMAIL_ADDRESS>is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
$ node -r babel-register internals/scripts/CheckNativeDep.js && npm run flow-typed && npm run build-dll && electron-builder install-app-deps && node node_modules/fbjs-scripts/node/check-dev-engines.js package.json

><EMAIL_ADDRESS>flow-typed /Users/user/Developer/app
> rimraf flow-typed/npm && flow-typed install --overwrite || true

• Found 83 dependencies in package.json to install libdefs for. Searching...
• flow-typed cache not found, fetching from GitHub...
• Installing 12 libDefs...
  • react-router_v4.x.x.js
    └> ./flow-typed/npm/react-router_v4.x.x.js
react-router
/Users/user/Developer/app/flow-typed/npm
  • enzyme_v3.x.x.js
    └> ./flow-typed/npm/enzyme_v3.x.x.js
enzyme
/Users/user/Developer/app/flow-typed/npm
  • react-redux_v5.x.x.js
    └> ./flow-typed/npm/react-redux_v5.x.x.js
react-redux
/Users/user/Developer/app/flow-typed/npm
  • react-router-dom_v4.x.x.js
    └> ./flow-typed/npm/react-router-dom_v4.x.x.js
react-router-dom
/Users/user/Developer/app/flow-typed/npm
  • chalk_v2.x.x.js
    └> ./flow-typed/npm/chalk_v2.x.x.js
chalk
/Users/user/Developer/app/flow-typed/npm
  • redux_v3.x.x.js
    └> ./flow-typed/npm/redux_v3.x.x.js
redux
/Users/user/Developer/app/flow-typed/npm
  • express_v4.16.x.js
    └> ./flow-typed/npm/express_v4.16.x.js
express
/Users/user/Developer/app/flow-typed/npm
  • rimraf_v2.x.x.js
    └> ./flow-typed/npm/rimraf_v2.x.x.js
rimraf
/Users/user/Developer/app/flow-typed/npm
  • minimist_v1.x.x.js
    └> ./flow-typed/npm/minimist_v1.x.x.js
minimist
/Users/user/Developer/app/flow-typed/npm
  • flow-bin_v0.x.x.js
    └> ./flow-typed/npm/flow-bin_v0.x.x.js
flow-bin
/Users/user/Developer/app/flow-typed/npm
  • react-test-renderer_v16.x.x.js
    └> ./flow-typed/npm/react-test-renderer_v16.x.x.js
react-test-renderer
/Users/user/Developer/app/flow-typed/npm
  • jest_v22.x.x.js
    └> ./flow-typed/npm/jest_v22.x.x.js
jest
/Users/user/Developer/app/flow-typed/npm
• Generating stubs for untyped dependencies...
  •<EMAIL_ADDRESS>    └> flow-typed/npm/source-map-support_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/electron-debug_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/enzyme-to-json_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/babel-plugin-dev-expression_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/eslint-import-resolver-webpack_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/eslint-formatter-pretty_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/npm-logical-tree_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/stylelint-config-standard_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/url-loader_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/font-awesome_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/history_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/react-router-redux_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/redux-thunk_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/babel-plugin-add-module-exports_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/babel-jest_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/babel-plugin-dynamic-import-webpack_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/babel-preset-env_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/babel-preset-react-optimize_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/babel-plugin-transform-class-properties_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/babel-preset-react_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/babel-preset-react-hmre_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/babel-register_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/detect-port_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/babel-preset-stage-0_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/enzyme-adapter-react-16_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/electron-devtools-installer_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/eslint-config-airbnb_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/file-loader_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/redux-logger_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/spectron_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/sass-loader_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/style-loader_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/stylefmt_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/webpack-merge_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/babel-eslint_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/babel-plugin-transform-es2015-classes_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/babel-loader_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/cross-env_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/concurrently_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/css-loader_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/eslint-plugin-compat_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/electron-builder_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/eslint-plugin-promise_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/cross-spawn_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/eslint-plugin-import_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/eslint-plugin-react_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/extract-text-webpack-plugin_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/identity-obj-proxy_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/fbjs-scripts_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/webpack_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/uglifyjs-webpack-plugin_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/webpack-bundle-analyzer_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/webpack-dev-server_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/devtron_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/react-hot-loader_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/eslint-plugin-jest_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/electron-rebuild_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/eslint_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/flow-typed_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/eslint-plugin-flowtype_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/stylelint_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/eslint-plugin-jsx-a11y_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/babel-core_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/flow-runtime_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/sinon_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/jsdom_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/node-sass_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/babel-plugin-flow-runtime_vx.x.x.js
  •<EMAIL_ADDRESS>    └> flow-typed/npm/electron_vx.x.x.js

!! No<EMAIL_ADDRESS>libdefs found in flow-typed for the above untyped dependencies !!

I've generated `any`-typed stubs for these packages, but consider submitting
libdefs for them to https://github.com/flowtype/flow-typed/

><EMAIL_ADDRESS>build-dll /Users/user/Developer/app
> cross-env NODE_ENV=development node --trace-warnings -r babel-register ./node_modules/webpack/bin/webpack --config webpack.config.renderer.dev.dll.js --colors

Hash: fe8b8a4cf4de11660124
Version: webpack 3.10.0
Time: 1125ms
              Asset     Size  Chunks                    Chunk Names
renderer.dev.dll.js  1.16 MB       0  [emitted]  [big]  renderer
[./node_modules/electron-debug recursive] ./node_modules/electron-debug 160 bytes {0} [built]
[./node_modules/webpack/buildin/harmony-module.js] (webpack)/buildin/harmony-module.js 596 bytes {0} [built]
   [0] dll renderer 12 bytes {0} [built]
    + 148 hidden modules

WARNING in ./node_modules/electron-debug/index.js
81:45-58 Critical dependency: the request of a dependency is an expression
 @ ./node_modules/electron-debug/index.js
 @ dll renderer

WARNING in ./node_modules/electron-debug/index.js
84:85-106 Critical dependency: the request of a dependency is an expression
 @ ./node_modules/electron-debug/index.js
 @ dll renderer
  • electron-builder version=19.55.3
  • loaded configuration file=package.json ("build" field)
  • installing production dependencies platform=darwin arch=x64 appDir=/Users/user/Developer/app/app
✨  Done in 413.71s.

That's not an error, it's expected output. The missing libdefs have been stubbed out with any, and are located alongside the real libdefs.


Synchronize @date-io DateType types

Hey, I just made a Pull Request!
This pull request includes a small tweak to the opencost plugin so that the TS types will build correctly.
In short, the issue arose because the opencost plugin was using date utils from @date-io/date-fns, while all other plugins in the package used @date-io/luxon. date-fns and luxon each export a DateType type, and they were conflicting when importing from @date-io. This change switches the opencost plugin to use @date-io/luxon like the other plugins, and resolves the type error.
:heavy_check_mark: Checklist

[ ] A changeset describing the change and affected packages. (more info)
[ ] Added or updated documentation
[ ] Tests for new functionality and regression tests for bug fixes
[ ] Screenshots attached (for UI changes)
[ ] All your commits have a Signed-off-by line in the message. (more info)

Hey @mattray! This should solve the build issue. Opening a fork of my own seemed simpler than sending instructions or getting push access to your fork :)
Note that I haven't actually tested that the thing runs properly (not sure how). I've only resolved the build error. So we may not be completely out of the woods yet.

Thanks, I'm merging and testing, I'll report back.


Draw relationship line on scene tree dock if option is on

Fix #16855
I tried to find what commit caused it but could not.

Cherrypicked into 3.0.3


URGENT: [composer] GRPC randomly throws ServiceException("Socket closed")

@dwsupplee @jdpedrie this issue still randomly persists.
URGENT REQUEST. We're part of the Google Cloud Startup program and launching this year... a fix would be greatly appreciated so we can move to production.
Log output:
PHP Fatal error:  Uncaught Google\Cloud\Core\Exception\ServiceException: {
    "message": "Socket closed",
    "code": 14,
    "status": "UNAVAILABLE",
    "details": []
} in /[...]/Google/composer-google-cloud/vendor/google/cloud/Core/src/GrpcRequestWrapper.php:257
Stack trace:
#0 /[...]/Google/composer-google-cloud/vendor/google/cloud/Core/src/GrpcRequestWrapper.php(194): Google\Cloud\Core\GrpcRequestWrapper->convertToGoogleException(Object(Google\ApiCore\ApiException))
#1 [internal function]: Google\Cloud\Core\GrpcRequestWrapper->handleStream(Object(Google\ApiCore\ServerStream))
#2 /[...]/Google/composer-google-cloud/vendor/google/cloud/Firestore/src/SnapshotTrait.php(122): Generator->current()
#3 /[...]/Google/composer-google-clo in /[...]/Google/composer-google-cloud/vendor/google/cloud/Core/src/GrpcRequestWrapper.php on line 257

@dwsupplee firestore, not spanner

@jdpedrie
Package  Version State
grpc     1.23.1  stable
protobuf 3.10.0  stable

Thanks @lukasgit.
Does it seem to always be related to a particular service call, or does it happen randomly?
Would you be able to enable gRPC debugging and share the results relating to a failure? You can set the following environment variables to turn on debugging: GRPC_VERBOSITY=debug GRPC_TRACE=all.

@jdpedrie I'm not seeing any gRPC debugging info in the php logs:
putenv('GRPC_VERBOSITY=debug');
putenv('GRPC_TRACE=all');

@lukasgit I'm sorry I was unclear. Setting those variables will cause additional logging data to be written to stderr, not to the PHP log.

@jdpedrie not seeing any gRPC debugging info to stderr.
php.ini
display_errors = stderr

@jdpedrie ^^

Hey @lukasgit sorry for not responding sooner. I'm working on this though, hope to have more by the end of the day, or tomorrow at latest.

@jdpedrie great, thanks for the update.

Are you using PHP-FPM? Make sure your configuration has catch_workers_output=yes. This should cause the workers' stderr output to be written to the server error log.

@jdpedrie still nothing... here is what I have configured:
nginx-1.17.5 (dev environment compiled with --with-debug):
error_log logs/error.log debug;

php-fpm (php-7.3.11):
catch_workers_output = yes
env[GRPC_VERBOSITY] = debug
env[GRPC_TRACE] = all

php code:
putenv('GRPC_VERBOSITY=debug');
putenv('GRPC_TRACE=all');

Hi @lukasgit, I'm sorry again for the back-and-forth. I've been trying without much success to capture the gRPC debugging data from nginx/php-fpm.
Do you have control over the system which is managing the FPM daemon? supervisord and systemd for instance? Are you able to configure php-fpm to start with an explicit pipe of stderr to file?
php-fpm 2>/var/log/php-fpm.log

Additionally, I spoke with a contact on the gRPC team, and he suggested you toggle the debug and verbosity a bit differently than what I advised earlier:
GRPC_VERBOSITY=debug
GRPC_TRACE=api,call_error,channel,client_channel_call,connectivity_state,handshaker,http,subchannel,tcp

I've opened an issue on gRPC to improve the utilities for capturing gRPC debugging information in PHP.

Hi @jdpedrie, no worries on the back-and-forth. Whatever it takes for us to resolve this issue.
I do have full control over the development system. Following your instructions, still nothing related to gRPC in /var/log/php-fpm.log :
[08-Nov-2019 21:24:18] NOTICE: fpm is running, pid 1104
[08-Nov-2019 21:24:18] NOTICE: ready to handle connections

nginx-1.17.5 (dev environment compiled with --with-debug):
error_log logs/error.log debug;

php-fpm (php-7.3.11):
catch_workers_output = yes
env[GRPC_VERBOSITY] = debug
env[GRPC_TRACE] = api,call_error,channel,client_channel_call,connectivity_state,handshaker,http,subchannel,tcp

php code:
putenv('GRPC_VERBOSITY=debug');
putenv('GRPC_TRACE=api,call_error,channel,client_channel_call,connectivity_state,handshaker,http,subchannel,tcp');

Hi @lukasgit,
@stanley-cheung, the person I've been talking to on the gRPC team, mentioned that in his tests using Apache, he found that setting the environment variables with putenv and even in the server configuration was too late and they were ineffective. In his test using docker, he set them in the Dockerfile and had better luck. Could you try setting them at the highest level possible? Perhaps etc/environment or similar.

Hi @lukasgit I am one of the maintainers of the grpc php extension.
The only thing that seems to work for me (in terms of the php-fpm + nginx setup), is to do the combination of these 2:

Set the environment variable at the top of my Dockerfile ( I am using FROM php:7.2-fpm-stretch as a base for example)
Run php-fpm 2>/var/log/php-fpm.log to start the service (instead of relying on the standard CMD) to redirect stderr away to my desired log file

I have tried all those catch_workers_output = yes, env[GRPC_VERBOSITY] = debug in the fpm www.conf file but none of those work. Doing putenv() in a .php script seems definitely too late.
But even with the php-fpm 2>mylog trick, the log is being outputed kind of ugly with " [pool www] child 12 said into stdout: " and lines got broken up.
So I am currently working on adding a php.ini option for the grpc extension that we can divert all those grpc logs into a separate log file, tracked in here. Will keep you updated.
But also just so I know, for the initial error, how rare / often does that happen? Are we talking about 1 every 10 request, or 1 every 10000 request? Just want to see what the scale of things is.

Hi @stanley-cheung thanks for jumping in on this issue.
I will hang tight until the php.ini option for the grpc extension is available.
As for how many times the error occurs per x requests, it varies widely.  Sometimes it happens 7 times out of 50 requests.  Sometimes 0 times out of 1000 requests.  We haven't ran a stress test of 10,000 requests on grpc yet.

I started this PR: https://github.com/grpc/grpc/pull/20991. This works for the most part but may need some more polish. I am slightly concerned about the lack of file rotations / capping the size of the log file - as it stands, the log file will grow unbounded. There might be a need for a separate cronjob or something to monitor and regularly truncating this log file.

@stanley-cheung any update on this issue?

@lukasgit Sorry for the late reply. We are very close to cutting a 1.26.0RC1 release candidate, which will contain that fix. Once that's done, will you be able to install that in your environment and enable the grpc log to a separate log file? You might need to monitor the file size growth if the error was not happening frequently.

@stanley-cheung since our last chat, we upgraded the composer version and only had the error happen one time (yesterday).  So yeah, it's still very random.  I can definitely run your next RC, just provide me with instructions when you're ready. Thanks!

@lukasgit 1.26.0RC1 should be available on PECL now. If you just do [sudo] pecl install grpc-beta, that should be the version installed.
Please add the following lines to your php.ini file:
grpc.grpc_verbosity=debug
grpc.grpc_trace=all,-timer_check
grpc.log_filename=/var/log/grpc.log

You can add more traces to ignore by adding more minus '-' sign to the grpc.grpc_trace option. See this doc for all the possible values for that option. There are potentially a few you can add to trim down the volume of the log.

i got same issue on my local
but on production not working with same version

Have a similar issue with our local env. Every connection to Firestore from PHP fails with "unavailable" and an empty message. Code 14.
PHP 7.2.10
grpc extension v1.26.0
google/protobuf ^v3.3.0
Pinning grpc/grpc to anything between v1.19 & v1.25 works. 1.26 & 1.26.1 breaks.

@digarahayu @stakable could you try enabling gRPC tracing as @stanley-cheung described above and sharing some logs from the error cases?

I think this issue is related to mine #2539.
Fatal error: Uncaught Google\Cloud\Core\Exception\ServiceException: { "message": "Empty update", "code": 14, "status": "UNAVAILABLE", "details": [] }

Same here. I recently started getting quite a few of these ... like once every 10 minutes.
    "message": "Socket closed",
    "code": 14,
    "status": "UNAVAILABLE",
    "details": []
} in /lib/firestore/vendor/google/cloud-core/src/GrpcRequestWrapper.php:257```

An update, as I can't seem to be able to edit the previous message.
This never happened on a server from another ISP (Hetzner).
On the other hand, on the servers at DigitalOcean, it happens quite often, sometimes multiple times per hour, sometimes once per day, with random methods, which don't necessarily have big payloads.

@edi are you able to try capturing some of the tracing data as described by @stanley-cheung above?

@jdpedrie @stanley-cheung we've noticed the issue persists but something of interest might be happening.
The issue seems to suddenly stop when we update the composer package google/cloud.
Is there possibly a correlation between these random "Socket closed" errors due to changes on your end that might not be compatible with a prior version of google/cloud composer package?

Do you know what the previous version of google/cloud was where the issue appeared? We'll have to look at what has changed between then and now. But do I understand correctly that the latest version of the client seems to no longer have the issue? If so, that's great news!

On the server with the issue, I have "google/cloud-core": "^1.31", which is very weird becase I've installed it myself via composer require about two weeks ago.
Regarding grpc, I have 1.27.0 installed, and I'm now upgrading to 1.28.1, hopefully it sorts it out.
This is how many times it happened for me so far, in the past 48 hours.
I've also re-installed cloud-core to 1.36.1 (given the OP's success with it), will be back tomorrow with an update.
Regards.

Thanks @edi, please do let us know.

Well gents, so far so good after updating to<EMAIL_ADDRESS>👍
As a sidenote, the server which didn't get a<EMAIL_ADDRESS>update but only a gRPC one, still has the issues.
Another server running the same<EMAIL_ADDRESS>as the one having issues, has<EMAIL_ADDRESS>installed. That didn't have this kind of issues either.
So, the issue is obviously not with gRPC (1.27 or 1.28.1), but with the older version of cloud-core.
1st server -> cloud-core 1.31 gRPC 1.28.1  -> BAD
2nd server -> cloud-core 1.35  gRPC 1.27.0 -> OK
3rd server -> cloud-core 1.36.1 gRPC 1.28.1 -> OK

I may have spoken too soon. Still happens on<EMAIL_ADDRESS>and<EMAIL_ADDRESS>Times when it happened today
Will try implement the logs mentioned above and come back with some more info ... it's so weird as the same exact setup on another server at another ISP and I have never had this issue.

I have decided to move on to a new Hetzner server. Just set up a new droplet on DO, basic nginx / php  setup. (debian 10 x64), nothing special.
Then you'll start randomly getting those socket closed errors, with the latest binaries and SDKs.
Back on Hetzner for 3 days now, no error whatsoever.

Back on Hetzner for 3 days now, no error whatsoever.

@edi other than your setup being on a different hosting provider, was there a change in your server configuration or code ?

This is an issue we get totally randomly as well, and have been unable to trace, following this if the rest of you manage to find some fix.
https://sentry.io/share/issue/5ede539ebefb4e18a09cf9415c9a7b9b/

I'm fighting with this issue for months. I'm running my apps in App Engine standard (7.3), and this issue basically forced us to remove firestore from various services.

@joshuaoliver @sl0wik we are hunting down the problem as this is clearly not an isolated issue.

@stanley-cheung Updated the following components. Will monitor over the next couple of days and post result.
php
$ php -v
PHP 7.4.7 (cli) (built: Jun 12 2020 04:55:45)

grpc
$ php --re grpc | head -1
Extension [ <persistent> extension #34 grpc version 1.29.1 ] {

protobuf
$ php --re protobuf | head -1
Extension [ <persistent> extension #35 protobuf version 3.12.2 ] {

Using version ^0.133.1 for google/cloud
Package operations: 25 installs, 0 updates, 0 removals
  - Installing google/crc32 (v0.1.0): Loading from cache
  - Installing psr/cache (1.0.1): Loading from cache
  - Installing psr/http-message (1.0.1): Loading from cache
  - Installing ralouphie/getallheaders (3.0.3): Loading from cache
  - Installing guzzlehttp/psr7 (1.6.1): Loading from cache
  - Installing guzzlehttp/promises (v1.3.1): Loading from cache
  - Installing symfony/polyfill-php72 (v1.17.0): Downloading (100%)
  - Installing symfony/polyfill-mbstring (v1.17.0): Downloading (100%)
  - Installing symfony/polyfill-intl-idn (v1.17.0): Downloading (100%)
  - Installing guzzlehttp/guzzle (6.5.4): Downloading (100%)
  - Installing firebase/php-jwt (v5.2.0): Loading from cache
  - Installing google/auth (v1.9.0): Downloading (100%)
  - Installing google/protobuf (v3.12.2): Downloading (100%)
  - Installing google/common-protos (1.2): Loading from cache
  - Installing grpc/grpc (1.27.0): Loading from cache
  - Installing google/grpc-gcp (0.1.5): Downloading (100%)
  - Installing google/gax (1.3.1): Downloading (100%)
  - Installing symfony/polyfill-ctype (v1.17.0): Downloading (100%)
  - Installing ramsey/collection (1.0.1): Loading from cache
  - Installing brick/math (0.8.15): Loading from cache
  - Installing ramsey/uuid (4.0.1): Loading from cache
  - Installing psr/log (1.1.3): Loading from cache
  - Installing monolog/monolog (2.1.0): Downloading (100%)
  - Installing rize/uri-template (0.3.2): Loading from cache
  - Installing google/cloud (v0.133.1): Downloading (100%)

@stanley-cheung problem still persists:
[12-Jun-2020 11:02:52 UTC] PHP Fatal error:  Uncaught Google\Cloud\Core\Exception\ServiceException: {
    "message": "Socket closed",
    "code": 14,
    "status": "UNAVAILABLE",
    "details": []
} in /opt/composer-google-cloud/composer-google-cloud-20200610/vendor/google/cloud/Core/src/GrpcRequestWrapper.php:257
Stack trace:
#0 /opt/composer-google-cloud/composer-google-cloud-20200610/vendor/google/cloud/Core/src/GrpcRequestWrapper.php(194): Google\Cloud\Core\GrpcRequestWrapper->convertToGoogleException()
#1 [internal function]: Google\Cloud\Core\GrpcRequestWrapper->handleStream()
#2 /opt/composer-google-cloud/composer-google-cloud-20200610/vendor/google/cloud/Firestore/src/SnapshotTrait.php(122): Generator->current()
#3 /opt/composer-google-cloud/composer-google-cloud-20200610/vendor/google/cloud/Firestore/src/SnapshotTrait.php(61): G in /opt/composer-google-cloud/composer-google-cloud-20200610/vendor/google/cloud/Core/src/GrpcRequestWrapper.php on line 257

@stanley-cheung Google is losing customers because this issue (see comment above from @joshuaoliver):

I'm fighting with this issue for months. I'm running my apps in App Engine standard (7.3), and this issue basically forced us to remove firestore from various services.

/cc @schmidt-sebastian have we seen similar issues in other client libraries per chance? This seems like it may be network related, but we're having a hard time pin pointing it.

We've been tracking this issue since last year and had connectivity issues with various versions / combinations post grpc v1.19. We've been running the following both locally and in production (PHP72 Flex) for most of this year without issue:
"grpc/grpc": "v1.19", "google/protobuf": "^v3.3.0",
Rolling back is not ideal but you may at the very least get a stable service running until a fix is discovered. I haven't looked at this for months but I'm pretty certain things break for us if we update grpc beyond v1.19.

@dwsupplee @stanley-cheung

Hello, I have a python grpc client and a C++ server. There is a exception appearing occasionally which tell me "Socket closed".

https://gitter.im/grpc/grpc?at=5ee236d924a3382d5d5e66cb

"error": "14 UNAVAILABLE: failed to connect to all addresses"

https://gitter.im/grpc/grpc?at=5ecea678b101510b201efa12

@lukasgit did you by any chance be able to add these entries to your php.ini to enable grpc level logs? https://github.com/googleapis/google-cloud-php/issues/2427#issuecomment-562334183.
Once you saw an UNAVAILABLE error like you mentioned, can you go clip out perhaps the logs since a few minutes before that point and send it to us?
What is the Firestore operation that triggers this? Can you give us some profile as to how this error is triggered? Something like this would help: "we are calling  once every X seconds/minutes/hours. This error happens once every Y such operations."

@stanley-cheung yes however no grpc log is being created; still getting this error:
[12-Jun-2020 21:22:21 UTC] PHP Fatal error:  Uncaught Google\Cloud\Core\Exception\ServiceException: {
    "message": "Socket closed",
    "code": 14,
    "status": "UNAVAILABLE",
    "details": []
}

@stanley-cheung we isolated where we think we can capture Firestore operations that triggers this Socket closed issue. I'll have something for you within an hour or less.

While we are at it, you mentioned that you are using "GCE Debian instance located in Zone us-east1-b", can you point us the exact image name, size of the instances, etc, so that when we try to replicate this on our end, to get as close to your setup?

@stanley-cheung reply to my email and I'll send you those details

@lukasgit - I'm also interested in the solution. I'm happy to share all my AppEngine instance information.

TL;DR: I had been looking into this in the last day or so, but I still couldn't reproduce the Socket Closed | 14 | Unavailable error. Here are the details:

I created a GCP VM and enabled the Firestore API. There are some troubles with access and permissions and service accounts and whatnot, but eventually I am able to create a fresh GCP project. The GCP VM is in us-east1-b with Debian 10 (which makes it easier to install PHP 7.3), and the Firestore database is in nam5 (us-central).

I installed the grpc pecl extension version 1.29.1 via pecl install grpc. PHP version is PHP 7.3.14-1~deb10u1 (cli) (built: Feb 16 2020 15:07:23) ( NTS ). I added extension=grpc.so to /etc/php/7.3/cli/php.ini and /etc/php/7.3/fpm/php.ini. My composer.json is extremely simple:

{
    "require": {
	"google/cloud-firestore": "v1.13.0"
    }
}

These are all the other composer packages that got pulled in (I scrubbed the composer.lock file a bit).
"name": "brick/math",    "version": "0.8.15",
"name": "firebase/php-jwt",    "version": "v5.2.0",
"name": "google/auth",    "version": "v1.9.0",
"name": "google/cloud-core",    "version": "v1.37.1",
"name": "google/cloud-firestore",    "version": "v1.13.0",
"name": "google/common-protos",    "version": "1.2",
"name": "google/gax",    "version": "1.3.1",
"name": "google/grpc-gcp",    "version": "0.1.5",
"name": "google/protobuf",    "version": "v3.12.2",
"name": "grpc/grpc",    "version": "1.27.0",
"name": "guzzlehttp/guzzle",    "version": "6.5.4",
"name": "guzzlehttp/promises",    "version": "v1.3.1",
"name": "guzzlehttp/psr7",    "version": "1.6.1",
"name": "monolog/monolog",    "version": "2.1.0",
"name": "psr/cache",    "version": "1.0.1",
"name": "psr/http-message",    "version": "1.0.1",
"name": "psr/log",    "version": "1.1.3",
"name": "ralouphie/getallheaders",    "version": "3.0.3",
"name": "ramsey/collection",    "version": "1.0.1",
"name": "ramsey/uuid",    "version": "4.0.1",
"name": "rize/uri-template",    "version": "0.3.2",
"name": "symfony/polyfill-ctype",    "version": "v1.17.0",
"name": "symfony/polyfill-intl-idn",    "version": "v1.17.0",
"name": "symfony/polyfill-mbstring",    "version": "v1.17.0",
"name": "symfony/polyfill-php72",    "version": "v1.17.0",

First attempt is to just create a CLI script to continuously ping my Firestore database. The script goes something like this (see below). It basically just keeps querying a document in Firestore every 10 seconds. (I later modified my script to also update the Firestore document via update-document($db); every minute just to make sure things change in a regular interval.) This script has been running for 2+ hours now with no error.

<?php
require 'vendor/autoload.php';
use Google\Cloud\Firestore\FirestoreClient;
$db = new FirestoreClient([
  'projectId' => '<my gcp project>'
]);
function update_document($db) {
  $data = [
    'name' => sprintf('document data: Los Angeles %s', date('H:i')),
    'state' => 'CA',
    'country' => 'USA'
  ];
  $db->collection('cities')->document('LA')->set($data);
}
while (true) {
  // call update_document($db) once every 6 loops
  $docRef = $db->collection('cities')->document('LA');
  $snapshot = $docRef->snapshot();
  if ($snapshot->exists()) {
    echo sprintf("%s \"%s\"", date('Y-m-d H:i:s'), $snapshot['name']);
  } else {
    echo "Failed\n";
  }
  sleep(10);
}

Next, I tried to have the script run by Nginx + PHP-FPM. I have a very simple /etc/nginx/sites-available/mysite set up. And made sure I ran something like sudo systemctl restart nginx and sudo systemctl restart php7.3-fpm.

server {
    listen 8080;
    root /var/www/html;
    index index.php index.html index.htm;
    server_name localhost;
    location / {
        try_files $uri $uri/ =404;
    }
    location ~ \.php$ {
        include snippets/fastcgi-php.conf;
        fastcgi_pass unix:/run/php/php7.3-fpm.sock;
    }
}

And a /var/www/html/firestore.php script (which is extremely similar to my CLI script above) to be executed by php-fpm when that's requested by Nginx:
<?php
require '/home/stanleycheung/firestore-test/vendor/autoload.php';
use Google\Cloud\Firestore\FirestoreClient;
$db = new FirestoreClient([
  'projectId' => '<my gcp project>',
  'keyFile' => json_decode(file_get_contents("<path to credentials.json>"), true),
]);
$docRef = $db->collection('cities')->document('LA');
$snapshot = $docRef->snapshot();
if ($snapshot->exists()) {
  echo sprintf("From Nginx/FPM: %s \"%s\"\n",
               date('Y-m-d H:i:s'), $snapshot['name']);
} else {
  echo "Failed\n";
}

Reason I need to add the keyFile to the constructor is just for some auth thing that because nginx / php-fpm are running as the system user so there are some complication there.

I then set up a bash shell script to continuously ping Nginx:

#!/bin/bash
while :
do
    curl -s localhost:8080/firestore.php
    sleep 10
done

So far I haven't received a single error after running the script for almost 3 hours now. I was monitoring my nginx /var/log/nginx/access.log and /var/log/nginx/error.log and there was no error.
From Nginx/FPM: 2020-06-14 06:34:55 "document data: Los Angeles 06:34"
From Nginx/FPM: 2020-06-14 06:35:05 "document data: Los Angeles 06:34"
From Nginx/FPM: 2020-06-14 06:35:16 "document data: Los Angeles 06:34"
From Nginx/FPM: 2020-06-14 06:35:26 "document data: Los Angeles 06:35"
From Nginx/FPM: 2020-06-14 06:35:36 "document data: Los Angeles 06:35"
From Nginx/FPM: 2020-06-14 06:35:46 "document data: Los Angeles 06:35"
From Nginx/FPM: 2020-06-14 06:35:56 "document data: Los Angeles 06:35"
From Nginx/FPM: 2020-06-14 06:36:06 "document data: Los Angeles 06:35"
From Nginx/FPM: 2020-06-14 06:36:16 "document data: Los Angeles 06:35"
From Nginx/FPM: 2020-06-14 06:36:27 "document data: Los Angeles 06:36"
From Nginx/FPM: 2020-06-14 06:36:37 "document data: Los Angeles 06:36"

Unless stated above - I didn't touch anything else in my GCP project, my GCP VM, my nginx / php-fpm setup, my PHP installation, my grpc extension, and the client libraries. I didn't tweak any config parameters like timeouts and stuff. I got almost everything from the standard GCP / google cloud libraries documentations.

So that's where I am now. Current setup:

A main PHP CLI script to

query a Firestore document every 10 seconds, and

update the document every 60 seconds.

A bash script to run curl localhost:8080/firestore.php every 10 seconds to query the same Firestore document.
Regularly do tail -f on access.log, error.log, the CLI log, and the curl log to see if there's any error.
I am also able to go to the Cloud Console, navigate to Firestore on the left, and actually see the content of that Firestore document I keep updating.

I really want to be able to see the Socket Closed | 14 | Unavailable error myself, but so far I wasn't able to.
I am going to leave to script to run overnight to see if there's any error.

So looks like we could rule out the basic stuff. I am already using a similar set up in GCP, doing a similar set of operations on the Firestore API, using the latest version of the extension and the composer packages, and using a similar (albeit a bit too simple) Nginx + PHP-FPM setup.
So there are something else that we haven't been able to identify, that the common scenarios above are missing. Anyone participating in this thread can spot something in my setup and can say "hey I am doing something different , can you try  instead?"

Are there some config parameters somewhere that you had in your php-fpm config, or nginx config that I should try?
 I wonder if there are weird interactions if I enable some extensions that I don't know about
Have you installed any other composer packages that I should try adding?
Are there any setup at the Network Interface in your GCP that could make a difference? I am using whatever the default is, which shows up as nic0, IP forwarding: Off, etc, in my VM now.

Or coming from the other side - can someone else try to start from scratch, like I did, in your own setup / environment? Start with a clean VM/base image, a clean composer installation, a clean PECL extension installation and see if you can reproduce the issue that way?

Alright - we are finally getting somewhere. I am able to get in touch with @lukasgit and asked him to enable the grpc log once again using the instructions back here and we are able to get the relevant log file for this "Socket closed" issue.
So for anyone else monitoring this thread - let me repeat the instructions here:

Please add these 3 lines to your php.ini file. Could be /etc/php/7.3/fpm/php.ini if your PHP script is being executed by php-fpm. Add these after the extension=grpc.so line.

grpc.grpc_verbosity=debug
grpc.grpc_trace=all,-timer_check
grpc.log_filename=/var/log/grpc.log

Make sure the log file /var/log/grpc.log is writable.

$ sudo touch /var/log/grpc.log
$ sudo chmod 666 /var/log/grpc.log

Restart your php-fpm sudo systemctl restart php7.3-fpm.

Monitor /var/log/grpc.log, possibly by running tail -f /var/log/grpc.log | grep -i 'Socket closed'. Whenever you receive a "Socket closed" error in your Nginx error.log, there should be something corresponding being spit out here in the grpc.log.

Yes the log file will grow pretty quickly. You may want to zero it out every X minutes depending on how fast the file grows: cp /dev/null /var/log/grpc.log if no error happens. This can be run even when the tail -f command is still running - you don't have to restart that.

Once you received a 14 UNAVAILABLE Socket Closed error, clip out the relevant section by timestamp. Each log line starts with something like I1592187671.018904062. The first part<PHONE_NUMBER> is the current unix timestamp. You can try to match it to the exact time when your "Socket Closed" error happen, then go back 10 seconds and then go forward 10 seconds, or something like that. That should provide us a pretty good picture of the error.

In this case, I am seeing logs like these:
I1592187671.017156449 chttp2_transport.cc:1082]              transport 0x55b37dbe5910 got goaway with last stream id 31
I1592187671.017174678 connectivity_state.cc:151]             ConnectivityStateTracker client_transport[0x55b37dbe5c40]: READY -> TRANSIENT_FAILURE (got_goaway)

I1592187794.834571065 chttp2_transport.cc:2653]              ipv4:<IP_ADDRESS>:443: Complete BDP ping err={"created":"@1592187794.833724507","description":"Endpoint read failed","file":"/tmp/pear/temp/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc","file_line":2514,"occurred_during_write":0,"referenced_errors":[{"created":"@1592187794.832794013","description":"Secure read failed","file":"/tmp/pear/temp/grpc/src/core/lib/security/transport/secure_endpoint.cc","file_line":183,"referenced_errors":[{"created":"@1592187794.832736514","description":"Socket closed","fd":11,"file":"/tmp/pear/temp/grpc/src/core/lib/iomgr/tcp_posix.cc","file_line":781,"grpc_status":14,"target_address":"ipv4:<IP_ADDRESS>:443"}]},{"created":"@1592187794.832620171","description":"GOAWAY received","file":"/tmp/pear/temp/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc","file_line":1075,"grpc_status":14,"http2_error":0,"raw_bytes":"session_timed_out"}]}

This leads me a similar error with Node + Spanner: https://github.com/googleapis/nodejs-spanner/issues/620. And from there, it looks like this "GOAWAY received" error is a transient error that's retryable.
@jdpedrie @dwsupplee Can you perhaps look into this to see if this can be added to Firestore client library to handle this case?

Sorry, I was OOO on Friday.
We just submitted a change that should sync to the PHP client by tomorrow which will retry CommitRequests that fail with UNAVAILABLE. This might help here.
FWIW, we also just started retrying RunQuery requests in Node: https://github.com/googleapis/nodejs-firestore/pull/1116
From my experience, it is more fruitful to spend time on retry logic rather than trying to eliminate all network related issues.

@schmidt-sebastian Thanks for the update! I was also just pointed to an internal bug b/144734355 that's very similar to this issue.
And it eventually leads to some fixes like this in Go and this in Python. Would the PHP fix be similar to that?
Thanks!

Yes. PHP will need retry behavior on Watch RPCs, that includes setting the last resume token from the previous session. Note that even under normal operations, a Watch stream closes at least once an hour due to Auth token expiry, and as such retry is critical for these long lived stream.

Great details, thanks @schmidt-sebastian and @stanley-cheung for all your research thus far. @crwilcox does adding this retry behavior seem like a reasonable thing for us to get slated?

I think it’s equally important for the end user/developer to wrap client
calls with retry behavior in their code, and have the Dev documentation and
API reference docs suggest this approach. It’s best to have it
logged/handled and retried x times rather than immediate halt due to socket
exception errors.
On Mon, Jun 15, 2020 at 2:26 PM Sebastian Schmidt<EMAIL_ADDRESS>wrote:

Yes. PHP will need retry behavior on Watch RPCs, that includes setting the
last resume token from the previous session. Note that even under normal
operations, a Watch stream closes at least once an hour due to Auth token
expiry, and as such retry is critical for these long lived stream.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/googleapis/google-cloud-php/issues/2427#issuecomment-644331289,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACOQUD5EK2CMGVUTWHVCKM3RWZYVPANCNFSM4JFK7SRA
.

@lukasgit End users will not be able to retry Watch streams without incurring additional charges for document reads. Only the SDK has the ability to resume Watch streams and start where it last left off.

I understand, and absolutely the SDK must resume watch streams. However, to
account for future issues that may arise as we’ve discovered, wrapping
client calls in a retry block ensure the end user’s experience isn’t
interrupted by backend network issues. The cost is negligible, versus the
alternative of having upset users.
On Mon, Jun 15, 2020 at 3:23 PM Sebastian Schmidt<EMAIL_ADDRESS>wrote:

@lukasgit https://github.com/lukasgit End users will not be able to
retry Watch streams without incurring additional charges for document
reads. Only the SDK has the ability to resume Watch streams and start where
it last left off.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/googleapis/google-cloud-php/issues/2427#issuecomment-644363946,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACOQUD55O63BJGAMLFY3MADRWZ7M3ANCNFSM4JFK7SRA
.

@stanley-cheung @dwsupplee @schmidt-sebastian any update yet? Seeing "Transport closed" and "Socket closed" daily again.

@lukasgit Hi, I am running C++ helloworld with client and server on different terminal and see "14: Socket closed" I see previously many folks faced this but did not see a root cause or solution, can someone help?  This happens continuously for me. This is the first time am running grpc.

Why is the ball getting dropped here?  I've provided ample data and we've identified the problem.  Now we're waiting for a fix on your end.  Numerous people across many threads and community platforms are complaining about this issue. @stanley-cheung @dwsupplee @schmidt-sebastian

I am seeing the same issue with python client running grpcio Version: 1.32.0

I think you may have to set environments for authentication. Below is a sample code to access the firestore
 'your-app-id',
    ]);

	$collectionReference = $firestore->collection('users');
	$documentReference = $collectionReference->document($Userid);
	$snapshot = $documentReference->snapshot();

	echo "Hello " . $snapshot['name'];

?>

putenv(“GCLOUD_PROJECT=app-id”);
putenv(“GOOGLE_APPLICATION_CREDENTIALS=path/to/service-auth-key-file.json”);
use Google\Cloud\Firestore\FirestoreClient;
$firestore = new FirestoreClient([
‘projectId’ => ‘your-app-id’,
]);
$collectionReference = $firestore->collection(‘users’);
$documentReference = $collectionReference->document($id);
$snapshot = $documentReference->snapshot();
echo “Hello ” . $snapshot[‘name’];

Same problem here:

For anyone else reading this thread the line grpc.grpc_trace=all,-timer_check, actually broke my grpc. Something about the minus syntax. This made the issue I was originally trying to resolve 1000% worse and I just happened this time around while debugging trying the other syntax offered by some: grpc.grpc_trace=api,call_error,channel,client_channel_call,connectivity_state,handshaker,http,subchannel,tcp.
Also don't forget me make your log file writeable. sudo chmod 666 /var/log/grpc.log

Why is the ball getting dropped here? I've provided ample data and we've identified the problem. Now we're waiting for a fix on your end. Numerous people across many threads and community platforms are complaining about this issue. @stanley-cheung @dwsupplee @schmidt-sebastian

@stanley-cheung @dwsupplee @schmidt-sebastian getting close to a year later.  This is a pretty serious bug that affects production.  Do you have a fix or at the very least an update?

This issue was mis-categorized as a "question" and thus slipped through our SLO filters. I've adjusted it. cc: @dwsupplee

Thank you @meredithslota

Santa clou(d)s please help us...

Hi @lukasgit
Thanks a lot for having the patience with respect to this issue. I understand it's a bit frustrating to still getting the errors after such a long time.
Update on this:
I had created a setup based on this comment(https://github.com/googleapis/google-cloud-php/issues/2427#issuecomment-643890450) and it's been running for almost 10 days. So far I have had 1 occurrence  of "UNAVAILABLE" and I had application logs and GRPC logs enabled.
I will respond back here once I have a sense of what improvements can be made from the library's side at this point of time.

Hi @saranshdhingra
Huge thanks for taking initiative on resolving this issue.
As a workaround we run a function that attempts to execute interactions with Firestore.  If an error occurs, the function will log the error and try again up to five times before giving up.  This seems to work as the first attempt fails (randomly), while the second attempt works.  NOTE: In the last six months we've noticed a substantial decrease in "message": "Socket closed" / "status": "UNAVAILABLE errors.
Unfortunately we're now receiving these new error messages since November 9, 2021 (note these show the same "code":14):
[09-Nov-2021 19:13:39 UTC] FIRESTORE error:  Google\Cloud\Core\Exception\ServiceException: {
    "message": "GOAWAY received",
    "code": 14,
    "status": "UNAVAILABLE",
    "details": []
} in (redacted)/google/cloud/Core/src/GrpcRequestWrapper.php:257
Stack trace:
#0 (redacted)/google/cloud/Core/src/GrpcRequestWrapper.php(194): Google\Cloud\Core\GrpcRequestWrapper->convertToGoogleException()
#1 [internal function]: Google\Cloud\Core\GrpcRequestWrapper->handleStream()
#2 (redacted)/google/cloud/Firestore/src/SnapshotTrait.php(122): Generator->current()
#3 (redacted)/google/cloud/Firestore/src/SnapshotTrait.php(61): Google\Cloud\Firestore\DocumentReference->getSnapshot()
#4 (redacted)/google/cloud/Firestore/src/DocumentReference.php(347): Google\Cloud\Firestore\DocumentReference->createSnapshot()
#5 (redacted): Google\Cloud\Firestore\DocumentReference->snapshot()
#6 (redacted): require('(redacted)...')
#7 (redacted): require('(redacted)...')
#8 (redacted): require('(redacted)...')
#9 (redacted): (redacted)
#10 {main} in (redacted)
[09-Nov-2021 19:46:01 UTC] FIRESTORE error:  Google\Cloud\Core\Exception\ServiceException: {
    "message": "GOAWAY received",
    "code": 14,
    "status": "UNAVAILABLE",
    "details": []
} in (redacted)/google/cloud/Core/src/GrpcRequestWrapper.php:257
Stack trace:
#0 (redacted)/google/cloud/Core/src/GrpcRequestWrapper.php(194): Google\Cloud\Core\GrpcRequestWrapper->convertToGoogleException()
#1 [internal function]: Google\Cloud\Core\GrpcRequestWrapper->handleStream()
#2 (redacted)/google/cloud/Firestore/src/SnapshotTrait.php(122): Generator->current()
#3 (redacted)/google/cloud/Firestore/src/SnapshotTrait.php(61): Google\Cloud\Firestore\DocumentReference->getSnapshot()
#4 (redacted)/google/cloud/Firestore/src/DocumentReference.php(347): Google\Cloud\Firestore\DocumentReference->createSnapshot()
#5 (redacted): Google\Cloud\Firestore\DocumentReference->snapshot()
#6 (redacted): require('(redacted)...')
#7 (redacted): require('(redacted)...')
#8 (redacted): require('(redacted)...')
#9 (redacted): (redacted)
#10 {main} in (redacted)
[21-Dec-2021 05:24:05 UTC] FIRESTORE error:  Google\Cloud\Core\Exception\ServiceException: {
    "message": "GOAWAY received",
    "code": 14,
    "status": "UNAVAILABLE",
    "details": []
} in (redacted)/google/cloud/Core/src/GrpcRequestWrapper.php:257
Stack trace:
#0 (redacted)/google/cloud/Core/src/GrpcRequestWrapper.php(194): Google\Cloud\Core\GrpcRequestWrapper->convertToGoogleException()
#1 [internal function]: Google\Cloud\Core\GrpcRequestWrapper->handleStream()
#2 (redacted)/google/cloud/Firestore/src/SnapshotTrait.php(122): Generator->current()
#3 (redacted)/google/cloud/Firestore/src/SnapshotTrait.php(61): Google\Cloud\Firestore\DocumentReference->getSnapshot()
#4 (redacted)/google/cloud/Firestore/src/DocumentReference.php(347): Google\Cloud\Firestore\DocumentReference->createSnapshot()
#5 (redacted): Google\Cloud\Firestore\DocumentReference->snapshot()
#6 (redacted): require('(redacted)...')
#7 (redacted): require('(redacted)...')
#8 (redacted): require('(redacted)...')
#9 (redacted): (redacted)
#10 {main} in (redacted)

We are seeing high numbers of these errors in production when talking to Datastore in Firestore mode, from App Engine 2nd Generation runtimes.

We are seeing high numbers of these errors in production when talking to Datastore in Firestore mode, from App Engine 2nd Generation runtimes.

Hi @iamacarpet
Thanks for the info.
Currently, I have been running the test script constantly for 6 weeks, and it's extremely hard to reproduce this.
Can you tell me how frequently are you experiencing the errors(once a week, once a day etc)?
It would be great if you could send the error message(either paste here or in an online paste), making sure no sensitive information is present(if it's exactly like the error on the above comment, then you need not do it).
Meanwhile, I'll try to setup a check in App Engine as well.

 As my setup hasn't detected any errors in a while.
It will be great if you could also paste in your workaround code(retry mechanism), I wanna understand if there is some difference with the in built retry mechanism because this should be happening automatically upto 5 times, unless the option is overwritten(which I am sure isn't the case here).

@saranshdhingra I'll share some info from our new project that has been in testing, here is the traffic level (started getting real visitor traffic 5 days ago):

And the error we saw in Error Reporting:

The error has dropped off now, as we were already using the latest version of google-cloud-php, but changed our models to all use ExponentialBackoff for reads and writes, plus updated this function to better map to this error:
https://github.com/a1comms/GaeSupportLaravel/commit/303ae7ce919ec7122e1729d7cdee40bb259d1fa9
We used to see a lot of "Connection reset by peer" before it switched from Datastore to Firestore in Datastore mode, so it appears this error is the equivalent.
Example usage:
https://github.com/a1comms/GaeSupportLaravel/blob/php7.4-laravel6.0/src/A1comms/GaeSupportLaravel/Session/DatastoreSessionHandler.php#L145
https://github.com/a1comms/GaeSupportLaravel/blob/php7.4-laravel6.0/src/A1comms/GaeSupportLaravel/Session/DatastoreSessionHandler.php#L179
Since deploying this, the errors appear to have gone away.
We are currently in the process of deploying to our other projects, one of them has an error rate that looks like this:

Thanks a lot for the detailed info @iamacarpet
Although "UNAVAILABLE" errors are retry-able, not sure why that's not happening here.
Will try to dig in and update.

@saranshdhingra thanks, if it helps, the error thrown is weird.
It’s the only exception we’ve seen thrown without a stack trace, and from what I can tell, it’s something to do with being thrown from within the binary gRPC module.
Could be unrelated but in case it helps

@saranshdhingra thanks, if it helps, the error thrown is weird.
It’s the only exception we’ve seen thrown without a stack trace, and from what I can tell, it’s something to do with being thrown from within the binary gRPC module.
Could be unrelated but in case it helps

My thoughts exactly with @iamacarpet
FYI, we have not experienced any further "UNAVAILABLE" errors since December 21, 2021 using the latest versions:
google/grpc-gcp (v0.2.0)
google/protobuf (v3.19.4)
grpc/grpc (1.42.0)

I haven't seen any errors in the project I created to replicate this error.
So, I will be closing this for now.
But, I want to let you know that we have efforts planned out for improving the error messages across the library for all products.
I will try to come back to this and see if our efforts make such situations better.
Thanks a lot for the patience you folks showed for this particular error :)

Hey there @saranshdhingra could you share what OS you're running on and what version of the gRPC library you're using where you do not see the "socket closed" issue?  I'm inclined to think this is an issue in the network stack below gRPC or perhaps how gRPC is interacting with the native network stack.
We are running on Ubuntu 18.04 with gRPC v1.28.1 and see this issue with some frequency (on the order of 10s of minutes during a long-lived bi-directional streaming connection).
I'm curious if this issue was solved or if it just did not repro in your environment and hence was closed?  Thanks!


Updated option-provider.md

Updated option-provider.md with the provided string literal

I have read the CLA Document and I hereby sign the CLA

recheck

I have read the CLA Document and I hereby sign the CLA


SystemExit: 2

Issue Template
First of all, thank you for finding the bugs. Please help us improve by providing the bug report as much as possible in the following format.
Describe the bug
/usr/local/lib/python3.6/dist-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.
import pandas.util.testing as tm
usage: ipykernel_launcher.py [-h] [-th TRAINING_THRESHOLD] [-nc NCLUSTER]
[-crp CRP_FACTOR] [-stb STEP_BEFORE]
[-wn WEIGHT_NORM] [-func FUNCTION] [-cmax CMAX]
[-cmin CMIN] [-fmd FEATURE_MAP_DROPOUT]
[-idt INPUT_DROPOUT] [-hdt HIDDEN_DROPOUT]
[-lbs LABEL_SMOOTHING]
[-fsize FILTER_SIZES [FILTER_SIZES ...]]
[-fnum NUM_FILTERS] [-al ALPHA] [-lmda LMBDA]
[-b BATCH_TRAINING] [-mg MARGIN] [-opt OPTIMIZER]
[-s SAMPLING] [-ngr NEGRATE] [-l EPOCHS]
[-lr LEARNING_RATE] [-k HIDDEN_SIZE]
[-km ENT_HIDDEN_SIZE] [-kr REL_HIDDEN_SIZE]
[-k2 HIDDEN_SIZE_1] [-l1 L1_FLAG] [-gp GPU_FRAC]
[-npg NUM_PROCESS_GEN] [-mn MODEL_NAME]
[-db DEBUG] [-exp EXP] [-ds DATASET_NAME]
[-dsp DATASET_PATH] [-ld LOAD_FROM_DATA]
[-sv SAVE_MODEL] [-tn TEST_NUM] [-ts TEST_STEP]
[-t TMP] [-r RESULT] [-fig FIGURES]
[-plote PLOT_EMBEDDING] [-plot PLOT_ENTITY_ONLY]
ipykernel_launcher.py: error: ambiguous option: -f could match -func, -fmd, -fsize, -fnum, -fig
An exception has occurred, use %tb to see the full traceback.
SystemExit: 2
To Reproduce
Steps to reproduce the behavior:
`# Author: Sujit Rokka Chhetri
License: MIT
import sys
from pykg2vec.utils.kgcontroller import KnowledgeGraph
from pykg2vec.config.config import Importer, KGEArgParser
from pykg2vec.utils.trainer import Trainer
def main():
# getting the customized configurations from the command-line arguments.
args = KGEArgParser().get_args(sys.argv[1:])
# Preparing data and cache the data for later usage
knowledge_graph = KnowledgeGraph(dataset=args.dataset_name, negative_sample=args.sampling)
knowledge_graph.prepare_data()

# Extracting the corresponding model config and definition from Importer().
config_def, model_def = Importer().import_model_config(args.model_name.lower())
config = config_def(args=args)
model = model_def(config)

# Create, Compile and Train the model. While training, several evaluation will be performed.
trainer = Trainer(model=model, debug=args.debug)
trainer.build_model()
trainer.train_model()

if name == "main":
main()`
Expected behavior
Flawless run
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

OS: [Linux
SDK: jupyter notebook

Dependency (please complete the following information):

Current version of the dependencies

Additional context
Is anyone still monitoring this repo?

Yes there is someone still monitor this repo.
But from this report, I can't tell what's happening in your environment. Can you report more information so that I can help?

Thanks for your reply @louisccc , the jupyter notebook snippet is not running as expected. Perhaps the code is deprecated.

related to #144 .


Is this needed - TX end of transmission fix

Additionally, gFlagEndTransmission == true, which was being set when
a serial config was in progress or when TX timeout (but not always...(?))
path was switching to foreground function w/o stopping the radio, and
some RF pulses were emited after the radio was kicked out from TX mode
upon, for instance, the beginning of a serial configuration process.

Wondering if this fix should be imported: https://github.com/joaquimorg/uv-k5-firmware-custom/commit/3b96b2ad32518b8464bd2d2a9c3d853c50b0813d
cc @ErikS-web

Any idea how to test old situation, as I never noticed it ?

Wondering if this means that radio transmits when not wanted? Can you do couple of tests like CHIRP programming and receive on the active TX frequency to make sure it doesn't transmit?
Also can you check it transmit timeout works properly?

I am still wondering how to detect a tinx TX. Does it turns Red LED on ?
I will keep my powermeter attatched for some while. But if analog meter is fast enough ...

I'd test while setting 2 radios on the same channel. And do bunch of tests to see if the radio transmits when it shouldn't.

Closing for now.


Typescript declaration does not work well

Expected behavior and actual behavior.
Exports valid typescript types.
Steps to reproduce the problem.

generate a typescript react project
Install this package using yarn yarn add canvas-datagrid
Import the library using import DataGride from "canvas-datagrid";
go the error below

Specifications like the version of the project, operating system, or hardware.
<EMAIL_ADDRESS>version "0.3.4"
resolved "https://registry.npmjs.org/canvas-datagrid/-/canvas-datagrid-0.3.4.tgz#b5c6ecc281ec70f474a6fe35d2625ba8d67e7d31"
integrity sha512-oTXeqtbIMrV6LL+bo8qPmem5TMgvtY9ewwMkJgBNjlhjUjeSuFDl3+0bMSsQCmuIkET4c9QlzNDDRgFOpHfMiw==
dependencies:
is-printable-key-event "1.0.0"

@dinfer Did you find any workarounds? I'm also struggling with the same problem

I did some fumbling around and turns out the problem was the way I imported the library.
So instead of
import canvasDatagrid from  'canvas-datagrid'

Just use
import 'canvas-datagrid'

And it will work. Alternatively you can just add canvas-datagrid to your tsconfig.json types array


.NET Core Versioning fails if the version field is not present and TargetFrameworks is used

VSTS Extension you are using
VersionDotNetCoreAssembliesTask
Where are you running it?
VSTS
Version of Extension/Task
1.19.2
Expected behavior and actual behavior
Expected: The version (e.g. AssemblyVersion) element is correctly added to the .csproj when it was not present before.
Actual: Although the output claims otherwise, the .csproj was not modified.
Steps to reproduce the problem
These steps assume that the "Version field to update" setting is left empty.

Create a new .csproj in the new format.
Instead of setting the <TargetFramework> property, set <TargetFrameworks> (e.g. <TargetFrameworks>net46;netstandard2.0</TargetFrameworks>).
Make sure to not include an <AssemblyVersion> element in the .csproj.
Execute the task on the .csproj.
Inspect the .csproj.

This is most likely caused because the regex used to insert the AssemblyVersion element is looking for </TargetFramework>, which is not present due <TargetFrameworks> being used.
A workaround is to simply add the field with a dummy value (e.g. <AssemblyVersion>1.0.0</AssemblyVersion> to the .csproj. In that case, the normal replacement takes effect.

I think we could do things a little differently with that to handle both situations.
The first easy option that occurs to me is to use a regex to check for  and if it exists to use that in the replacement instead of just .
I'll see what I can do with this, might be a little while before I can put together a PR but I'll try to get it done this month.

I've updated the task to now check for TargetFramework or TargetFrameworks and update correctly. I'll have a release out later today hopefully.


Added a second multitenancy policyset for hub only policies

Some of the multitenancy policies are for the hub and some are for all managed clusters.  I tried to separate them. Be aware I did not modify the placement settings which currently have all policies deployed only to the hub, but this change will help anyone looking at this understand what they need to do.
Refs:

https://github.com/stolostron/backlog/issues/25956

Signed-off-by: Gus Parvin<EMAIL_ADDRESS>

I think this will need to be rebased after the community branch merges.  I'll just close this for now.