Dexalot - EUROC and sAVAX balances are not recognized

Describe the bug
EUROC and sAVAX are not being recognized on Dexalot exchange when used. Tried to add these to the gateway connector-token command but still no go.
On the screenshot below, ran balance command and observed only few of the added tokens are showing available yet on exchange it has balance. The tokens sAVAX and EUROC are also updated on avalanche_tokens.json

When starting the strategy using DEX connector, it is expected to display the balances for the market pair but no avail

This causes the exchange to only submit order on buy which at the time it was running, it has enough balance to open a sell order

Steps To Reproduce

Setup client and gateway
Setup dexalot wallet then check balances by:

Using gateway balance command
Using gateway connector-tokens
Setup PMM then start, observe the prompt for balances

Release version
dev-1.22.0
logs_dexalot-eurocusdc.log

My guess is this check in https://github.com/hummingbot/gateway/blob/main/src/connectors/dexalot/dexalot.ts#L136
is the issue:
    const balances = await Promise.all(
      req.tokenSymbols.map((symbol) => {
        return this._portfolioContract.getBalance(
          req.address,
          fromUtf8(symbol)
        );
      })
    );

Fix applied to development branch.


ARROW-7013: [C++] arrow-dataset pkgconfig is incomplete

I've verified that this now does the right thing working on R bindings on https://github.com/apache/arrow/pull/5454.

Thanks @kou, revised, PTAL


OG-Core developer meeting, Nov. 8, 2021

Agenda for OG-Core developer meeting: 11/8/21
noon-1pm ET
Zoom Meeting ID:<PHONE_NUMBER>
Zoom Meeting Password: ogusa
Zoom Meeting URL: https://riceuniversity.zoom.us/j/4073958475
This meeting is open to the public.
Agenda

OG-Core theory additions

Government capital progress
Stochastic income, and more flexible tax functions. Have to use root finder with EGM, and non-monotonicity introduces multiple solutions. @rickecon has some suggestions for solutions (JCT uses grid, multiple starting points in intuitive bins associated with shape of tax function).
For World Bank, @rickecon likes the simple monotonic cubic spline option on total income as an intermediate solution

OG-UK progress

Next meeting: Monday, Nov. 22, 2021, noon-1pm ET
@jdebacker @MaxGhenis @jpycroft @nikhilwoodruff

Notes:

Government capital PR #722 is just waiting for some test updates. Then can be merged.
Documentation update in PR #722 is significant. In particular, first time in a long time we have had TPI equilibrium algorithm fully documented.
Decided short run approach (for March 2022) to more robust tax functions is to just add a monotonic cubic spline function of total income. Save idiosyncratic income shocks and nonmonotonic functions for later.
OG-UK should have demographics PR done by the end of this week.
@rickecon is finishing a debt and deficits piece that uses OG-USA for a few simulations.


Rhel84 add missing packages

The packages in the last imagefactory nightly differ from ours. The
following packages are now added:
oddjob
oddjob-mkhomedir
psmisc
authselect-compat
rng-tools
dbxtool
Also, the rngd and nfs-convert services are enabled.
This PR addresses issues listed in #1176
This pull request includes:

[ ] adequate testing for the new functionality or fixed issue
[ ] adequate documentation informing people about the change

Very nice! Are these all the ones you want to add, or a partial list? Will there also be follow-ups to remove some? I see in the ticket there are plenty more, but I guess many/most/all will be pulled in as deps of these?

@teg These are all the packages that I want to add. I will create separate PRs for enabling/disabling services and for removing packages (aka as adding them to our list of excluded packages)

Very nice! Are these all the ones you want to add, or a partial list? Will there also be follow-ups to remove some? I see in the ticket there are plenty more, but I guess many/most/all will be pulled in as deps of these?

@teg These are all the packages that I want to add. I will create separate PRs for enabling/disabling services and for removing packages (aka as adding them to our list of excluded packages)

Also, is it okay to merge this since the test failures are unrelated to the changes?

Also, is it okay to merge this since the test failures are unrelated to the changes?

Yeah, if the failures are caused by flaky infrastructure, leave a brief comment about what's going on and why it is unrelated, then we can merge.

Yeah, if the failures are caused by flaky infrastructure, leave a brief comment about what's going on and why it is unrelated, then we can merge.

The tests are failing due to issues with openstack being overloaded. This is not related to this rhel 8.4 specific PR.

The tests are failing due to issues with openstack being overloaded. This is not related to this rhel 8.4 specific PR.


Poland reversed coordinates

As far as we saw the coordinates of all countries are in lon/lat format, except for Poland which are in lat/lon format.
For example pl/dolnoslaskie-addresses-state.geojson:
{"type":"Feature","properties":{"hash":"1606d482d1df2e9b","number":"2b","street":"Wałowa","unit":"","city":"Bolesławiec","district":"bolesławiecki","region":"dolnośląskie","postcode":"59-700","id":"16569eef-59a6-435c-a85e-f71bb1e29078"},"geometry":{"type":"Point","coordinates":[51.2780451,15.5654271]}}
VS cz/countrywide-addresses-country.geojson
{"type":"Feature","properties":{"hash":"c53a5e83a10ae58e","number":"9/10","street":"Jiřská","unit":"","city":"Praha 1","district":"","region":"","postcode":"11900","id":""},"geometry":{"type":"Point","coordinates":[14.40381,50.091628]}}

Fixed: https://batch.openaddresses.io/job/210638

Thanks!!!

@peterschinkel80 No problem, sorry it took so long, the holidays took up a bunch of my time this year. All the best!


React Material -UI server side Rendering

Hello Folks,
To brief about the problem which I am facing with respect to Pre-Render of React application which uses Material UI for styling aspects.
I have implemented React server side rendering of React-Material UI application using express node server.
I have followed below mentioned Razzle implementation approach.. and the Pre-Rendering is happening as expected....
https://github.com/kireerik/razzle-material-ui-styled-example
The more information about the implementation is mentioned in Razzle blog as below.
https://github.com/jaredpalmer/razzle#razzleconfigjs
As per the blog the Pre Render sample solution includes the Razzle node module which has got the webpack.config within it and in the solution we don't see any webpack.config.js included.

If at all if we want to have the webpack.config , we can have the extension for the same that we can create in our root folder ex: Razzle.config.js , we can put the our configurations within this file.**

Configuration can be to load the .less ,.css and any other .jsx files in general....
The example template of Razzle.config.js is given as below in the Razzle blog
// razzle.config.js
module.exports = {
modify: (config, { target, dev }, webpack) => {
// do something and return config
return config
}
}
In my Sample example I am trying to load the .less files by updating Razzle.config.js as below.
and the problem is its not accepting the modules loaders configuration inside the Razzle.config.js file
Ex:
'use strict';
console.log('From Razzle Config!!');
module.exports = {
modify: (config, {target, dev}, webpack) => {
// do something to config
resolve: {
extensions: ['', '.js', '.jsx']

},
module: {
loaders: [

    {

        test: /\.jsx?$/,
        loader: 'babel-loader',
        exclude: /node_modules/,
        query: {

        presets: ['es2015']

        }
    }
]

}
return config
} }

In the above example its throwing error at resolve and module keyword saying 'unused label'

I am bit confused about how to update the razzle.config to load the .less files and the .jsx files.

Help would be appreciated if some one correct the way of updating Razzle.config.js to load the .less and .jsx files.
Regards,
Goutham

Please submit support requests and questions to StackOverflow using the tag material-ui. StackOverflow is better suited for this kind of support. The issue tracker is for bug reports and feature discussions. See also our CONTRIBUTING guidelines.


When exporting pdf, the title is displayed incorrectly

I tried many files to export pdf, and this error occurred, and I tried to export html file, but no such error occurred

抱歉，回复晚了。
我注意到 Typora 的新版（<IP_ADDRESS>.2）更新说明有提到修复第三方主题导出PDF时出现的相关问题，可以尝试升级Typora 到最新 beta 版再确认一下问题是否得到修复。
如果问题依旧存在，欢迎继续回帖，并附上你所使用的版本号等有助定位问题的信息。

Sorry for my late reply.
I found that "Fix issues with PDF export on 3rd party themes."  is mentioned in release notes of Typora Beta <IP_ADDRESS>.2. You may update your Typora to see whether this issue is addressed in the latest update.
If this issue continues to occur, feel free to repost here. The version number you're using and other related information could be helpful to identify the issue.

I'm closing this issue. If any problem persists,  you're welcome to comment here or open a new issue.

I have the same issue when exporting pdfs, the version is typora 0.9.95-1, on manjaro linux.

I have the same issue when exporting pdfs, the version is typora 0.9.95-1, on manjaro linux.

Hi there! I've tested pdf exporting on Ubuntu 20.04 LTS, Typora 0.9.96 beta(I cannot install manjaro at the moment):

What it renders in application window(Misty Light Windows):

What the exported pdf look like(no noticeable misrendering could be seen):

As I failed to reproduce this issue on Ubuntu and have no concrete idea about what's going on, I recommend that you update Typora to latest version available and upload some screenshots(such as Typora main editing window, exported pdf & html in Misty theme & default built-in Github theme respectively).
I hope you could provide some detailed information so that I may spot where the problem lies.

Hi,
Upgrading to 0.9.96 beta solves the problem. It's only released yesterday on AUR.

或者您可以回滚过去的版本，我回滚到了

解决了所有的问题


New service for boiler seasonal control analysis

hi @ldodds this is ready for a first review.  I need to double check the tests against another school's data before merging tho.


Rebase of #182

See #182

Some of the most common renamings:
- global_declarations  ->  global_env
- global_context  ->  global_context_ext
- universe_context  ->  universes_decl
- consistent_universe_context_instance  ->  consistent_instance_ext
- snd Σ  ->  global_ext_constraints Σ
- on_global_decls_impl  ->  on_global_env_impl

To decide wether you should use a global_env or a global_env_ext:

global_env_ext changes only a few universes
reduction depends only on global_env
globalenv extension depends only on global_env
but typing, cumulativity, ... need global_env_ext


Support timeout for DNS

Not sure if possible given the current synchronous API and our Dns.SYSTEM implementation, but seems relevant for getting acceptable client behaviour.

Dupe of #95!


Remove references to T(n)

Pre-Submission Checklist

[x] Your pull request targets the staging branch of FreeCodeCamp.
[x] Branch starts with either fix/, feature/, or translate/ (e.g. fix/signin-issue)
[x] You have only one commit (if not, squash them into one commit).
[ ] All new and existing tests pass the command npm run test-challenges. Use git commit --amend to amend any fixes.

Type of Change

[x] Small bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds new functionality)
[ ] Breaking change (fix or feature that would change existing functionality)
[ ] Add new translation (feature adding new translations)

Checklist:

[ ] Tested changes locally.
[ ] Closes currently open issue (replace XXXX with an issue no): Closes #XXXX

Description

Removed text references to T(n) to make consistent with the video, which only discusses O(n).

Thanks for the fix @michaeltnoronha
✨ Congratulations on your first contribution to FreeCodeCamp! ✨


chore(deps): update terraform-module

This PR contains the following updates:

Package
Type
Update
Change

terraform-ibm-modules/icd-postgresql/ibm (source)
module
patch
3.9.3 -> 3.9.5

terraform-ibm-modules/landing-zone-vpc/ibm (source)
module
minor
7.15.0 -> 7.16.2

terraform-ibm-modules/security-group/ibm (source)
module
minor
2.3.1 -> 2.6.0

Release Notes

terraform-ibm-modules/terraform-ibm-icd-postgresql (terraform-ibm-modules/icd-postgresql/ibm)
v3.9.5
Compare Source
Bug Fixes

deps: update terraform-module (#336) (049883b)

v3.9.4
Compare Source
Bug Fixes

deps: updated required IBM provider version to >= 1.61.0, <2.0.0 to pickup fix for this provider issue (#346) (f371efb)

terraform-ibm-modules/terraform-ibm-landing-zone-vpc (terraform-ibm-modules/landing-zone-vpc/ibm)
v7.16.2
Compare Source
Bug Fixes

updates required provider version constraints to prevent major updates (#733) (da75636)

v7.16.1
Compare Source
Bug Fixes

change acl for_each conditional to fix bug (#725) (60867f4)

v7.16.0
Compare Source
Features

moved the management-vpc and workload-vpc submodule from the submodules directory to the modules directory to confirm with Hashicorp practises. See updated usage blocks on how to consume. (#694) (f583bbb)

terraform-ibm-modules/terraform-ibm-security-group (terraform-ibm-modules/security-group/ibm)
v2.6.0
Compare Source
Features

Adding a new output for Security Group ID, security_group_id_for_ref, which now gives two options for output of ID:1. security_group_id, original output, and will not be available until all rules have been applied for this group. Continue to use this value when implicit dependencies depend on a fully deployed security group, including the rules.2. security_group_id_for_ref, new output, will be available as soon as group is created and before all rules are deployed. Use this value when the rules you are supplying to the module block need a self-reference to the group being created in the same block. (#243) (b322a7b)

v2.5.1
Compare Source
Bug Fixes

updated required provider version constraints to prevent major version updates  (#240) (cc42f92)

v2.5.0
Compare Source
Features

added the ability to attach tags and/or access_tags to security groups (#239) (50c0b2e)

v2.4.0
Compare Source
Features

Allow to pass an existing security group id through the existing_security_group_id input  (#234) (fc0d147)

v2.3.2
Compare Source
Bug Fixes

direction is now optional in the security_group_rules input variable. If not specified, it defaults to "inbound" (#230) (ae11e20)

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

/run pipeline


Region limiter, some tests.

Some tests for good measure.

Simplify use of known API Limits
User can now use RiotAPI.setRateLimit(RateLimit.PRODUCTION_RATE_LIMITS); and has no reason to look up or know the production limits before that.

Allow users to set region specific rating withRiotAPI.setRateLimitingType(RateLimitingType.REGIONAL);

Rate limits are mostly region specific.
https://developer.riotgames.com/docs/rate-limiting

User rate limits are enforced per region. Every call made to any Riot Games API endpoint in a given region counts against the user rate limit for that key in that region, except where noted on the API Reference page.

Closing all the old PRs which won't be relevant to v3. My bad on never getting around to them.


Remove SQLAlchemy models

Remove SQLalchemy models from list_files
Add integration tests
Speed improvements: count(*) should be faster now
Bump up requirements

(still WIP)

@hellais updated after removing most of models.py and rerunning tests


fetching on demand problem

In MST I have used next code to do fetching on demand (what is planned to implement with types.lazy)
this example works as expected
@model("Test")
export class Test extends Model({
  test: tProp(types.maybe(types.number))
}) {
  @computed({requiresReaction: true})
  get $test() {
    const test = this.test;
    if (test) {
      return test;
    } else {
      setTimeout(() => this.testAction(), 0);
    }
  }

  @modelFlow
  testAction = _async(function * (this: Test) {
    this.test = yield* _await(Promise.resolve(1);
  })
}

const t = new Test({});
autorun(() => console.log("autorun ", t.$test));

this example worked in MST, but not in Keystone (it gets in loop action -> computed -> action -> computed and etc.)
@model("Test")
export class Test extends Model({
  test: tProp(types.maybe(types.number))
}) {
  @computed({requiresReaction: true})
  get $test() {
    const test = this.test;
    if (test) {
      return test;
    } else {
      setTimeout(() => this.testAction(), 10000); // to make the loop slow =)
    }
  }

  @modelFlow
  testAction = _async(function * (this: Test) {
    const response = yield* _await(Promise.resolve('' as any); // we become something with wrong type
    this.test = response; // thrown type checking excepton -> it's ok, but it also forces $test recalculation? why? this.test still not changed, still undefined
  })
}

const t = new Test({});
autorun(() => console.log("autorun ", t.$test));

Or is there some other recommended way to do fetching on demand in keystone? =)
thank you

I will prepare an example, an reopen it =)

found the next solution, it works (may be because it is not so nasty, as previous one =))
import {onBecomeObserved, onBecomeUnobserved, reaction} from 'mobx';

export function callOnDemand<V>(value: V, property: keyof V, fn: () => any): () => void {

  const disposers: (() => void)[] = [];
  let reactionDisposer;
  disposers.push(onBecomeObserved(value, property, () => {

    reactionDisposer = reaction(
      () => value[property],
      newValue => newValue === void 0 ? fn() : void 0,
      {fireImmediately: true, name: 'on demand reaction for ' + property}
    );

    disposers.push(reactionDisposer);
  }));

  disposers.push(onBecomeUnobserved(value, property, () => {

    if (reactionDisposer) {
      reactionDisposer();
      reactionDisposer = void 0;
    }
  }));

  return () => disposers.forEach((disposer: () => void) => disposer());
}

and then use it for example so
  onAttachedToRootStore(root) {

    callOnDemand(this.$, 'test', () => this.testAction());
  }


Straggler detection and mitigation

Currently we don't have a mechanism that detects and deals with stragglers.  Here we define a straggler as:
A process which is still working on its share of the parallelized computation when most or all of the other processes have completed their shares.

Straggler mitigation is a well-researched topic.  Here I will just post how the original MapReduce framework deals with stragglers, which can work on our system as well:
When a MapReduce operation is close
to completion, the master schedules backup executions
of the remaining in-progress tasks. The task is marked
as completed whenever either the primary or the backup
execution completes.

This is definitely a good idea.
I think major thing we'd need to figure out is how to blow away the output from the stragglers. So that we could restart them. In theory this wouldn't be too hard handles already do almost what we want.
A few random thoughts:
We could conceivably have a more aggressive definition of a straggler. We start jobs with approximately the same amount of data per worker and we can use the % of the data they've requested as a proxy for the jobs progress. This could allow us to detect workers that are straggling a lot sooner.
Backup executions of stragglers is great but what if we went a step further and broke the stragglers up into more shards so we could increase the parallelization factor and decrease the straggling.

@jdoliner I like the idea of monitoring the amount of downloaded data in order to detect stragglers.
I had a similar thought about breaking up input into more shards.  Basically we'd update the FilterShard structure such that it can contain a pointer to another FilterShard.  Let's call that a "FilterShard chain."
So, let's say that we want to break a container's input into two equal parts.  We'd simply generate two FilterShard chains by chaining the original filter shard with 1) a shard with modulus 2 and number 0, and 2) a shard with modulus 2 and number 1.  And we can trivially update BlockInShard and FilterInShard to accommodate for the chain.

An update: we have migrated PPS to a model where pods request "chunks" to process.  This model naturally lends itself to straggler mitigation: if a chunk takes too long to process, it can be split into multiple smaller chunks so the other pods can share the work.

Now we have finer grain processing of data via 'datums', so it should get more evenly distributed in general.
While a single datum could take much longer than others to process, it seems like 1.4 services this concern pretty well.
@derekchiang - are there other todo's here?

It might make sense in the new model to run multiple copies of the same datum if we get to the end of a job and a few datums are straggling.


Consider swapping out Gremlin back-end

We currently use sqlg as our Gremlin Server "back-end" which has worked well for prototyping so far. However, the version of sqlg (2.1.6) we are using depends on an older version of Tinkerpop (3.5.1) which is slowly becoming out of date. I tried upgrading to sqlg (3.0.0) yesterday, but it appears the sqlg is slowly moving away from supporting Gremlin Server in favor of having people import sqlg in a JVM application and use it as a library (that is at least its primary use case). sqlg has also upgraded to Java 17, but Tinkerpop does not fully support Java 17, and it seems that they are a ways from being able to upgrade to it.
I think it could be useful to explore alternate options for the Gremlin back-end to better future-proof nixpkgs-graph-explorer. Since we only interface with Gremlin Server itself this can be pretty freely swapped out without requiring code changes.
One alternative could be to use Janusgraph with one of its open source back-ends, e.g. ScyllaDB (docs). The downside to this is that by switching databases to something more specialized we narrow the available options for managed services.

We've faced performance issues using sqlg, which is another big motivation to swap the gremlin backend.

Another idea I had would be to remove the graph database all together in favor of a document-based approach.
My reasoning is as follows: our matter is serving the graph of derivations of a derivation. We could build the whole closure of each derivation and store it as a document: each derivation has its own document.
The reason this works is because extracting Nix data is idempotent/reproducible so data structures are immutable.
Thus even if we need to update the data model (e.g. we add new fields), we can re-run and erase overwrite.
Serving the closure of a derivation would then be straightforward: we return exactly the document of the derivation, no query resolution needed.
The overhead is the creation of documents, which should be manageable.
We could implement that quickly as JSON files that we just store on some storage (thanks to fsspec we could have the storage basically anywhere, on the server HDD or in the cloud).


Cannot load avro data into column-partitioned bigquery table

Client
BigQuery
Expected Behavior
I expect to be able to load avro data into a bigquery table that is partitioned by a column.
Actual Behavior
Different errors depending on if I use a table decorator or not:
With decorator:

Some rows belong to different partitions rather than destination partition 20180322

Without decorator:

Partitioning column ts must be of timestamp or date type, has type int64

Given the non-decorator error, I have a feeling it has something to do with the fact that I can only make the avro schema for the timestamp be a long type and since BigQuery ignores the logicalType (which would be timestamp-micro) then it just assumes that it's an int64. What I don't understand is why it's not realizing it's a timestamp when I create the table ahead of time. It shouldn't be trying to use the avro schema to create/modify the table.
Here is the shortest reproduction file I could make:
package main

import (
	"context"
	"fmt"
	"os"
	"strconv"
	"time"

	"google.golang.org/api/googleapi"

	"cloud.google.com/go/bigquery"
	"github.com/linkedin/goavro"
	"google.golang.org/api/option"
)

func isErrAlreadyExists(err error) bool {
	if err == nil {
		return false
	}
	if gerr, ok := err.(*googleapi.Error); ok && gerr.Code == 409 {
		return true
	}
	return false
}

func main() {
	ctx := context.Background()
	dataset := "test_analytics"
	bq, err := bigquery.NewClient(ctx, "admiral-test-183215")
	if err != nil {
		panic(err)
	}
	ds := bq.Dataset(dataset)
	if err := ds.Create(ctx, nil); err != nil && !isErrAlreadyExists(err) {
		panic(err)
	}
	schema := bigquery.Schema{
		&bigquery.FieldSchema{
			Name: "ts",
			Type: bigquery.TimestampFieldType,
		},
	}
	tblMeta := &bigquery.TableMetadata{
		Schema: schema,
		TimePartitioning: &bigquery.TimePartitioning{
			Field: "ts",
		},
	}
	tableName := "submissions_" + strconv.FormatInt(time.Now().UnixNano(), 10)
	tbl := ds.Table(tableName)
	if err := tbl.Create(ctx, tblMeta); err != nil && !isErrAlreadyExists(err) {
		panic(err)
	}

	fileName := "./data.avro"
	fw, err := os.Create(fileName)
	if err != nil {
		panic(err)
	}
	aw, err := goavro.NewOCFWriter(goavro.OCFConfig{
		W: fw,
		Schema: `{
			"type": "record",
			"namespace": "analytics",
			"name": "submission",
			"fields": [
				{"name": "ts", "type": "long", "default": 0}
			]
		}`,
		// biquery supports DEFLATE and Snappy
		CompressionName: goavro.CompressionSnappyLabel,
	})
	if err != nil {
		panic(err)
	}
	t := time.Now()
	m := map[string]interface{}{
		"ts": t.UnixNano() / 1000, // convert to microseconds
	}
	if err := aw.Append([]interface{}{m}); err != nil {
		panic(err)
	}
	fw.Close()

	fr, err := os.Open(fileName)
	if err != nil {
		panic(err)
	}

	day := time.Date(t.Year(), t.Month(), t.Day(), 0, 0, 0, 0, time.Local)
	tbl = ds.Table(fmt.Sprintf("%s$%s", tableName, day.Format("20060102")))

	bqr := bigquery.NewReaderSource(fr)
	bqr.SourceFormat = bigquery.Avro
	bqr.AllowJaggedRows = true
	bqr.MaxBadRecords = 0
	loader := tbl.LoaderFrom(bqr)
	loader.CreateDisposition = bigquery.CreateNever
	loader.WriteDisposition = bigquery.WriteAppend
	job, err := loader.Run(ctx)
	if err != nil {
		panic(err)
	}

	// docs say we have to check both errors, and neglect to say we need to
	// _also_ check the array of errors.
	status, err := job.Wait(ctx)
	if err != nil {
		panic(err)
	} else if err = status.Err(); err != nil {
		panic(err)
	} else if len(status.Errors) > 0 {
		panic(status.Errors[0])
	}
	fr.Close()
}

This is a known issue with BigQuery. There is an internal bug (33195959) tracking it and it seems that it is work in progress.

There has been progress on this issue and the feature should be available shortly. Unfortunately we can't provide an exact ETA.

@jba any update on this?

Awesome!

Email me your project ID(s). I am<EMAIL_ADDRESS>

Hi Jonathan,
Seems like I was in fact creating a bigquery loading job with a wrong
partition decorator. it was inconsistent with the value contained in the
Json key I defined as the partition timestamp.
Thx for your help
Le ven. 21 sept. 2018 à 23:15, Jonathan Amsterdam<EMAIL_ADDRESS>a écrit :

Email me your project ID(s). I am<EMAIL_ADDRESS>—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/google-cloud-go/issues/942#issuecomment-423672724,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABXMkrgUtuyM8DI8vAKGt0HHn0tqNkyOks5udVbxgaJpZM4S5a0B
.

Hi Jonathan,
We are using apache beam using java and created a datafow job which stream live data into BigQuery. We have used avro and made timetsamp column as
{"name":"timestamp", 		"type":{"type": "long", "logicalType":"timestamp-millis"}}
and in BigQuery we have made day wise partitioning on this timestamp field.
After few insertions, it started throwing exceptions
exception:  "ava.lang.RuntimeException: java.io.IOException: Insert failed: [{"errors":[{"debugInfo":"","location":"","message":"Value<PHONE_NUMBER>000000 for field timestamp of the destination table indus-tower:tower.device_telemetry is outside the allowed bounds. You can only stream to date range within 7 days in the past and 3 days in the future relative to the current date.","reason":"invalid"}],"index":0}
Athough, this data packet is residing in allowed time (with last 7 days). It is throwing same error even on same days data packet.
GCP Project Id - indus-tower
Please suggest.


Make FB Protein Complex file for Complex Portal

Outline: provide a file with discrete protein complexes for complex portal.
Complex Portal have offered make 'skelton' records for Protein complexes curated by FlyBase as part of gene groups.
Sandra Orchard has requested a file (tsv/xls) with the following information:
Name
Short name/symbol
Description
Interactions (ideally as UniProt IDs) *need GCRP mapping and phys_int file.
FlyBase xref
Publications (just as a PMID)
Google docs relating to complexes project

To generate this file:
Elements required:
-FBgn:GCRP mapping file
-Review of cv terms - how do we disguish discrete complexes from parent groups and subcomplexes : breakdown of FB2021_01 gene groups
-MITAB interactions file
-gene groups file

Review of cv terms:
In some cases we'd want the parent complex to be the "complex"

MCV
MITOCHONDRIAL COMPLEX V               SUBUNITS
FBgg0000491
protein complex group ;               FBCV:0003016
OXPHOS

MCV-CS
MITOCHONDRIAL COMPLEX V - CENTRAL STALK
FBgg0001686
protein complex group ; FBCV:0003016
MCV-F1

MCV-F0
MITOCHONDRIAL COMPLEX V - F0 SUBUNITS
FBgg0000489
protein complex group ; FBCV:0003016
MCV

MCV-F1
MITOCHONDRIAL COMPLEX V - F1 SUBUNITS
FBgg0000483
protein complex group ; FBCV:0003016
MCV

Need to fix issue: https://flybase.atlassian.net/browse/PDEV-149 to change gene group cv terms

From meeting, decided that this first load should only be for complexes that satisfy criteria as a discrete complex.
Tasks from meeting:

[ ] Make decision on ?? complex cases in spreadsheet
[ ] exclude all complex groups with alternative subunits e.g. testis specific isoforms (ribosomes and OXPHOS are prime examples of this)
[ ] Exand files to include: GO terms, synonyms
[ ]  Check with Sandra that Interaction file can be separate and how interactions should be formatted (pair, reciprocal, self-interactions?
[ ] Solve how to capture alternative SUs/isoforms in gene groups

Small tasks to do to:

[ ] Update definition for TFIIH groups
[ ] PMID for FBgg0001341
[ ] RNAcentral IDs for FBgn0019661, FBgn0019660, and FBgn0046696.

@gantonazzo I think that the definitions for TFIIH and CAK are fine as they are. Just update that for TFIIH-C (in the file and can you update the gene group def via proforma next epicycle)
Hopefully the below scans ok:
TFIIH-C
The general transcription factor complex Transcription factor II H (TFIIH) core complex, is a subcomplex of the TFIIH holoenzyme required in RNA pol II transcription with an ATP-dependent DNA helicase activity required for promoter DNA opening. TFIIH also functions in nucleotide excision repair (NER). After complex loading and the verification step of NER, the CAK subcomplex dissociates and the TFIIH core complex functions independently. (Adapted from PMID:25693126, PMID:21592869 and PMID:30798933).

@gantonazzo I think that the definitions for TFIIH and CAK are fine as they are. Just update that for TFIIH-C (in the file and can you update the gene group def via proforma next epicycle) Hopefully the below scans ok: TFIIH-C The general transcription factor complex Transcription factor II H (TFIIH) core complex, is a subcomplex of the TFIIH holoenzyme required in RNA pol II transcription with an ATP-dependent DNA helicase activity required for promoter DNA opening. TFIIH also functions in nucleotide excision repair (NER). After complex loading and the verification step of NER, the CAK subcomplex dissociates and the TFIIH core complex functions independently. (Adapted from PMID:25693126, PMID:21592869 and PMID:30798933).

Thanks, it reads well! I would just suggest to spell out CAK, since it's not mentioned before in the description:
"The general transcription factor complex Transcription factor II H (TFIIH) core complex, is a subcomplex of the TFIIH holoenzyme required in RNA pol II transcription with an ATP-dependent DNA helicase activity required for promoter DNA opening. TFIIH also functions in nucleotide excision repair (NER). After complex loading and the verification step of NER, the cyclin-dependent protein kinase-activating kinase (CAK) subcomplex dissociates and the TFIIH core complex functions independently. (Adapted from PMID:25693126, PMID:21592869 and PMID:30798933)."

Good for me - I flip-flopped over spelling it out - but you are right.

The files for the Complex Portal import have been generated and sent to Sandra.

Just to keep track of everything here: the record to update the description for TFIIH-C in FB has been loaded today.

Close for now. Open new ticket for fixing isoform issues.


Include JSch into standard DBeaver distribution

JSch UI is required by some 3rd party plugins.

Most likely this will be included in EE only. There is a bunch of extra E4 plugins which seems to be too heave and expensive for CE.


Inherit zod schema description by default

schema.openapi({ description: 'Some string' });

I am aware that the extension method .openapi() can be used to set the description, as shown above.
However, currently, it appears that descriptions of schema other than the root schema that is the target of register(), i.e., child or grandchild schema, are not automatically inherited. (See #95. My apologies if I am wrong.)
I think it is great that the .openapi() method provides a way to override the description, but it would be even better if the original description could be inherited by default.
Currently, I am handling this by calling my own recursive functions such as the following in advance, but it would be very helpful if these could be done inside functions such as register().
import { ZodTuple, ZodType } from 'zod';

const ZOD_TYPE = 'ZodType';
const zodDefKeyTypes = Object.setPrototypeOf({
    type        : ZOD_TYPE,
    options     : 'Array|Object',
    left        : ZOD_TYPE,
    right       : ZOD_TYPE,
    items       : 'Array',
    rest        : ZOD_TYPE,
    keyType     : ZOD_TYPE,
    valueType   : ZOD_TYPE,
    args        : ZOD_TYPE,
    returns     : ZOD_TYPE,
    schema      : ZOD_TYPE,
    innerType   : ZOD_TYPE,
    in          : ZOD_TYPE,
    out         : ZOD_TYPE,
}, null);

function addOpenApiMetadata(target) {
    if (!(target instanceof ZodType)) {
        throw new TypeError('The passed value is not a `ZodType`.');
    }
    if (target instanceof ZodTuple) {
        target = zodTupleToArray(target);
    }
    const { _def, description, shape } = target;
    _def.openapi || Object.assign(_def, {
        openapi: {
            _internal: undefined,
            metadata: { description, param: undefined },
        },
    });
    if (typeof shape === 'object') {
        for (const [key, value] of Object.entries(shape)) {
            shape[key] = addOpenApiMetadata(value);
        }
    } else {
        for (const [key, value] of Object.entries(_def)) {
            if (!(key in zodDefKeyTypes)) {
                continue;
            }
            if (zodDefKeyTypes[key] === ZOD_TYPE) {
                _def[key] = addOpenApiMetadata(value);
                continue;
            }
            for (const [k, v] of Object.entries(value)) {
                value[k] = addOpenApiMetadata(v);
            }
        }
    }
    return target;
}

Note: See #103 for an implementation of zodTupleToArray.

For example, suppose the following schema is defined.

schema.mjs
import { z } from "zod";

export const errorResponse = z.object({
    type: z.string().default('about:blank').describe('A URI reference [RFC3986](https://datatracker.ietf.org/doc/html/rfc3986) that identifies the problem type'),
    title: z.string().describe('A short, human-readable summary of the problem type'),
    status: z.number().int().min(400).describe('The HTTP status code ([RFC9110, Section 15](https://datatracker.ietf.org/doc/html/rfc9110#section-15)) generated by the origin server for this occurrence of the problem'),
    instance: z.string().describe('A URI reference that identifies the specific occurrence of the problem'),
    detail: z.string().describe('A human-readable explanation specific to this occurrence of the problem'),
}).partial().describe('[RFC 7807 - Problem Details for HTTP APIs](https://datatracker.ietf.org/doc/html/rfc7807)');

For this schema, call register() and then generateDocument() as follows.

api.mjs
import { z } from 'zod';
import {
    extendZodWithOpenApi,
    OpenAPIGenerator,
    OpenAPIRegistry,
} from '@asteasolutions/zod-to-openapi';

import { errorResponse } from './schema.mjs';

extendZodWithOpenApi(z);

const apiRegistry = new OpenAPIRegistry();
apiRegistry.register('ErrorResponse', errorResponse);

const apiGenerator = new OpenAPIGenerator(apiRegistry.definitions, '3.1.0');
export const apiDocument = apiGenerator.generateDocument();

The generated apiDocument will output YAML like this. (notice the absence of the description field.)

spec.yml
openapi: 3.1.0
components:
  schemas:
    ErrorResponse:
      type: object
      properties:
        type:
          type: string
          default: about:blank
        title:
          type: string
        status:
          type: integer
          minimum: 400
        instance:
          type: string
        detail:
          type: string
  parameters: {}
paths: {}

On the other hand, if you call register() on the schema through addOpenApiMetadata() above, the output YAML will change as follows.
+import { addOpenApiMetadata } from './util.mjs';
 import { errorResponse } from './schema.mjs';

 extendZodWithOpenApi(z);

 const apiRegistry = new OpenAPIRegistry();

-apiRegistry.register('ErrorResponse', errorResponse);
+apiRegistry.register('ErrorResponse', addOpenApiMetadata(errorResponse));

spec-new.yml
openapi: 3.1.0
components:
  schemas:
    ErrorResponse:
      type: object
      properties:
        type:
          type: string
          default: about:blank
          description: A URI reference [RFC3986](https://datatracker.ietf.org/doc/html/rfc3986) that identifies the problem type
        title:
          type: string
          description: A short, human-readable summary of the problem type
        status:
          type: integer
          minimum: 400
          description: The HTTP status code ([RFC9110, Section 15](https://datatracker.ietf.org/doc/html/rfc9110#section-15)) generated by the origin server for this occurrence of the problem
        instance:
          type: string
          description: A URI reference that identifies the specific occurrence of the problem
        detail:
          type: string
          description: A human-readable explanation specific to this occurrence of the problem
      description: "[RFC 7807 - Problem Details for HTTP APIs](https://datatracker.ietf.org/doc/html/rfc7807)"
  parameters: {}
paths: {}

@j-murata I get it now. Great catch. I'll check where the issue is as this is definitely not the expected behavior.

I made the following changes to schema.mjs and confirmed that the description fields are included in the output YAML, just like you said! Thanks.

schema.mjs
import { z } from "zod";
import { extendZodWithOpenApi } from '@asteasolutions/zod-to-openapi';

extendZodWithOpenApi(z);

export const errorResponse = z.object({
    ...
}).partial().describe('...');

// console.dir(errorResponse, { depth: null });

@AGalabov
Overriding zod's .describe() method inside extendZodWithOpenApi() may indeed be a reasonable solution.
However, from my use case, having to call this function when defining the schema seems a bit unusable.
This is because, in general, schema definitions using zod are not only for API specifications, but also for generating input validators, generating TypeScript type definitions, and for various other uses.
Of course it would be natural to call extendZodWithOpenApi() when creating an API specification, i.e. in api.mjs in the example above. But as far as schema.mjs is concerned, importing zod-to-openapi and calling this function seems unnatural to me.
flowchart LR
    S([Zod schema])
    A([API spec])
    I([Input validator])
    T([*.d.ts files])
    S --> A
    S --> I
    S --> T

Therefore, I would be happy if you could consider a different solution to this in the future, i.e. not overriding .describe(), but referring to and inheriting the value of description later.

@j-murata I get your point. And that is what we are doing in our project as well. Basically we have multiple schemas spread across files that we use for documentation (through the registry), validation and type coverage. And I do get your point about the description specifically, however with your current approach you wouldn't be able to use .openapi in that file at all - that means that you cannot "register" a function either.
I can share some insight on how we have setup our project:

We have a utils/zod file that calls extendZodWithOpenApi and reexports z.
We use that version everywhere in our project (we have an eslint rule not to import from 'zod')

this should also be achievable if you have an entrypoint where zod is extended and everything else is created afterwards

We've exported a registry from a separate file that can be used where schemas are defined. That's just useful for us so that we can just use that exported registry where needed instead of importing all schemas that need to be registered.

All that being said if it is really the description that you want, then that shouldn't be a problem. It would be a slightly less elegant solution but it can work. Just let me know if the explanation above changes your perspective on the matter. If not I'll see how the implementation should change to account for it.

I can share some insight on how we have setup our project:
1. We have a `utils/zod` file that calls `extendZodWithOpenApi` and reexports `z`.

2. We use that version everywhere in our project (we have an eslint rule not to import from `'zod'`)

* this should also be achievable if you have an entrypoint where zod is extended and everything else is created afterwards

3. We've exported a registry from a separate file that can be used where schemas are defined. That's just useful for us so that we can just use that exported registry where needed instead of importing all schemas that need to be registered.

Hey @AGalabov, thanks for detailing how you approach this in your own projects. I feel like these patterns are valuable information to share as part of the README.md , perhaps with an example and even the eslint rule

@j-murata I see your point now, thanks for the explanation. With that information provided I can agree that logic that depends on inner zod workings (length/description/regex, ...) should in fact be retrieved from the schemas themselves as opposed to relying on the overrides and the call of .openapi.
I'll make the necessary changes asap.

@AGalabov Thank you for your understanding.

I'll make the necessary changes asap.

I myself am not in such an urgent need, so there is no need for you to respond so quickly.
I hope zod-to-openapi will become a more usable library!:wink:

@AGalabov Any updates on this?
I found a workaround that intentionally delays the loading of the schemas.
This is not very smart, but it works for now.
import { z } from 'zod';
import { extendZodWithOpenApi } from '@asteasolutions/zod-to-openapi';
// import { errorResponse } from './schema.mjs';
extendZodWithOpenApi(z);
const { errorResponse } = await import('./schema.mjs');

Hey @j-murata sorry for the delay. I am waiting on a Code review. We were kind of busy at work, but I'll ping around today.

@j-murata you can give it a go using the latest v4.5.2 release.

@AGalabov
Unfortunately, it does not seem to output the descriptions of nested child and grandchild objects.
In the example above, the description of the errorResponse itself is output, but the description of each property under it is not.
Do you have any ideas on how to resolve this?

@j-murata sorry, I kind of missed your message since the issue was preemptively closed by me :/ I saw it randomly in my mail. You are correct I've missed a place in the code where we were relying on the description from .openapi() and were not using the one from .describe(). I opened a new PR that should resolve it.

@j-murata this should hopefully be fully fixed in the v4.6.0 release. Please let me know - Github closed the issue on its own since the PR was linked

@AGalabov
I tried my own codes using v4.6.0 and had no problems at all. Thank you for your long support!


Fix testing deps

What was wrong?
Test dependencies have gotten stale.
How was it fixed?
Update and pin versions.
Cute Animal Picture

don't review this, just review #13

going to merge via #13


Upload code coverage results

My fork testing https://github.com/prince-chrismc/pyrsia-sandbox/pull/1

You can see it here
https://codecov.io/github/pyrsia/pyrsia

I ran this locally, but it consumes a few GB's diskspace and it takes a long time.
I believe this is very useful, but maybe some parameters can be tweaked so that it doesn't fill your disk everytime you run sh test_code_coverage.sh ?
If this is not intended to be run on a developers machine, this is less of an issue.

That's an excellent point. 🤔
Tarpaulin is massive, but there are precompiled binaries that might work on your system (depending on rust version and libc) which are fairly small. That could be a solution for most.
Given the local output is annoying to look at I personally don't think I would use it locally. The results are posted on PR which pretty graphs.
I expect we will eventually write a better docker image to make a smaller image and something  reusable but that's out of scope for me


Add @Override annotations to implementations

We have been received similar kinds of issues which points out that there's an implementation but interface is missing.
Maybe we can prevent it by adding @Override annotations to all of implementations as JedisCluster does it now.

:+1:

I'll have this by tonight hopefully :pray:


⚠ Spryr has degraded performance

In 357700a, Spryr (https://spryr.com) experienced degraded performance:

HTTP code: 200
Response time: 5917 ms

Resolved: Spryr performance has improved in 7981bc8 after 21 minutes.


Pull correct variable for etcd initial variable

This shouldn't use the inventory_hostname variable, as that will just yield the same variable, but rather use the host which we're looping over.

Thanks alot.
it was a type from previous PR.

I was wondering if this was a long standing error, since it pretty much makes nothing work.

Yes in fact everything failed : https://travis-ci.org/kubespray/kargo/builds/125556416
We should have revert the merge.
your build is in progress:
https://travis-ci.org/kubespray/kargo/builds/126643599


Battery reporting issues

ecowitt2mqtt version
2022.01.0
Describe the bug
Using a gw1100 and have a WH32M connected to channel one. The battery is reported to home assistant but has a state of unknown.
The value passed is on/off
Not sure if this is a HA issue or not.
Another issue, again could be HA is my weather station (ws-5000) reports battery as a voltage but in HA it thinks 3.16v is low whej really thats 100%. This trips my low battery alerts.
To Reproduce
Steps to reproduce the behavior:
Same as above
Expected behavior
Issue with battery on channel 1 should be reporting on/off
The other issue I'm not sure how it should be handled.

Additional context

Had some life stuff come up, but haven't lost sight of this; will get to it as soon as I can!

Not a problem, totally understand!

@w1ll1am23 I'm back! Would love your help in diagnosing further.
Unfortunately, these devices are all over the place when it comes to representing battery values – as such, I have to have this (rather gross) helper to determine what I'm looking at:
https://github.com/bachya/ecowitt2mqtt/blob/155b11696effb81f7d15e2fe5f22f99d4ad1d665/ecowitt2mqtt/util/battery.py#L8-L21
Would you mind providing a HASS diagnostics dump of this device?

https://www.toptal.com/developers/hastebin/mivoguneca.json

Thanks! To confirm, we're investigating 2 issues:

batt1 is showing an unknown state.
wh80batt is showing a 3.16 state and HASS is interpreting that as a low battery (when it shouldn't).

Is that correct?

That is correct.

Issue 1: batt1 showing an unknown state
This issue was fixed in #109, but I haven't cut a new release since 2022.01.0 (what you're using). You can use the dev docker image or install the dev branch directly, but if you'd prefer a pinned version, hang in there: I'll cut a new one soon.
(Background info, in case you're interested: https://github.com/home-assistant/core/issues/65640#issuecomment-1031302211)
Issue 2: wh80batt showing a 3.16 state,, and HASS is interpreting that as a low battery (when it shouldn't)
Your diagnostics show your device is sending a value of 3.16, so that value (and unit) is correct. There's nothing in ecowitt2mqtt that tags that as a "low battery" – I wonder if that's something in your automation? Can you share that?

Nice, glad to hear the batt1 will start working.
The other makes sense figured it was an issue with HA.
I'm using this blue print.
https://gist.github.com/sbyx/1f6f434f0903b872b84c4302637d0890
I think it probably needs to consider units and is assuming the 3.16 in my cases is a % not v

That looks right. 👍🏻
Given that this issue is addressed, going to close. I will cut a new version today!


[carbonplan] Investigate spawn failures

Context
Reported via https://2i2c.freshdesk.com/a/tickets/493
User got TimeoutError on spawning on prod hub in carbonplan cluster

I tried to spawn the smaller image and was able to start it:
2023-02-27T18:33:21Z [Normal] Pulling image "carbonplan/trace-python-notebook:latest"

Event log
Server requested
2023-02-27T18:31:55.868160Z [Warning] 0/2 nodes are available: 2 node(s) didn't match Pod's node affinity/selector. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.
2023-02-27T18:31:57Z [Normal] pod triggered scale-up: [{eksctl-carbonplanhub-nodegroup-nb-r5-large-NodeGroup-4564L5LAHRC2 0->1 (max: 500)}]
2023-02-27T18:33:14.409319Z [Normal] Successfully assigned prod/jupyter-damianavila to ip-192-168-7-170.us-west-2.compute.internal
2023-02-27T18:33:15Z [Normal] Pulling image "busybox"
2023-02-27T18:33:16Z [Normal] Cancelling deletion of Pod prod/jupyter-damianavila
2023-02-27T18:33:17Z [Normal] Successfully pulled image "busybox" in 1.901398403s
2023-02-27T18:33:17Z [Normal] Created container volume-mount-ownership-fix
2023-02-27T18:33:17Z [Normal] Started container volume-mount-ownership-fix
2023-02-27T18:33:21Z [Normal] Pulling image "carbonplan/trace-python-notebook:latest"
```

It looks like the autoscale groups should be setup looking at the AWS EC2 console, and it looks like a large node just got started
i-07f7d403cebd75e21 (carbonplanhub-nb-r5-large-Node)

kubectl get nodes -A
'NAME                                           STATUS   ROLES    AGE     VERSION
ip-192-168-22-56.us-west-2.compute.internal    Ready    <none>   34d     v1.24.7-eks-fb459a0
ip-192-168-28-180.us-west-2.compute.internal   Ready    <none>   34d     v1.24.7-eks-fb459a0
ip-192-168-7-170.us-west-2.compute.internal    Ready    <none>   8m39s   v1.24.7-eks-fb459a0

hmm create by arn:aws:iam::631969445205:user/erik

Hey! I've been away climbing since 2hours 45min back.
What was created by my AWS account?

@damianavila I would imagine a 2 minute timeout for node startup may not always be sufficient in slow provisioning cases

A node in carbonplan cluster although it might be from automation

The timeout for spawn to ready should at least be 10 min, and i expect 5mins to startup a fresh server and download an image.

err="0/2 nodes are available: 2 node(s) didn't match Pod's node affinity/selector.

That is the core issue i guess. Is a node pool missing tags about the relevant labels or missing labels? I'm on mobile, if this is AWS, getting "tags" about labels is relevant for the autoscaler to be aware of things, and labes are relevant once the node is provisioned and made available in k8s.

ok looking at this the autoscheduler loop did complete
I0227 16:21:50.124722       1 schedule_one.go:826] "Unable to schedule pod; no fit; waiting" pod="prod/jupyter-norlandrhagen" err="0/3 nodes are available: 3 node(s) didn't match Pod's node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling."
I0227 16:21:50.124800       1 schedule_one.go:900] "Updating pod condition" pod="prod/jupyter-norlandrhagen" conditionType=PodScheduled conditionStatus=False conditionReason="Unschedulable"
I0227 16:23:30.619212       1 scheduling_queue.go:957] "About to try and schedule pod" pod="prod/jupyter-norlandrhagen"
I0227 16:23:30.619240       1 schedule_one.go:85] "Attempting to schedule pod" pod="prod/jupyter-norlandrhagen"
I0227 16:23:30.619421       1 binder.go:808] "All bound volumes for pod match with node" pod="prod/jupyter-norlandrhagen" node="ip-192-168-10-49.us-west-2.compute.internal"
I0227 16:23:30.619500       1 binder.go:365] "AssumePodVolumes" pod="prod/jupyter-norlandrhagen" node="ip-192-168-10-49.us-west-2.compute.internal"
I0227 16:23:30.619511       1 binder.go:373] "AssumePodVolumes: all PVCs bound and nothing to do" pod="prod/jupyter-norlandrhagen" node="ip-192-168-10-49.us-west-2.compute.internal"
I0227 16:23:30.619545       1 default_binder.go:52] "Attempting to bind pod to node" pod="prod/jupyter-norlandrhagen" node="ip-192-168-10-49.us-west-2.compute.internal"
I0227 16:23:30.645890       1 eventhandlers.go:159] "Delete event for unscheduled pod" pod="prod/jupyter-norlandrhagen"
I0227 16:23:30.645906       1 eventhandlers.go:184] "Add event for scheduled pod" pod="prod/jupyter-norlandrhagen"
I0227 16:23:30.648864       1 schedule_one.go:266] "Successfully bound pod to node" pod="prod/jupyter-norlandrhagen" node="ip-192-168-10-49.us-west-2.compute.internal" evaluatedNodes=4 feasibleNodes=1

In response to https://github.com/2i2c-org/infrastructure/issues/2271#issuecomment-1446877720, I think that's a pretty normal message. The k8s autoscaler waits for the scheduler to declare 'nope, can not do this!' before kicking in.

Discussion in slack points to a few things, I started a small node and pull time isn't that slow
Events:
  Type     Reason            Age    From                 Message
  ----     ------            ----   ----                 -------
  Warning  FailedScheduling  3m5s   prod-user-scheduler  0/3 nodes are available: 3 node(s) didn't match Pod's node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.
  Normal   Scheduled         88s    prod-user-scheduler  Successfully assigned prod/jupyter-pnasrat to ip-192-168-27-73.us-west-2.compute.internal
  Normal   TriggeredScaleUp  2m53s  cluster-autoscaler   pod triggered scale-up: [{eksctl-carbonplanhub-nodegroup-nb-r5-large-NodeGroup-4564L5LAHRC2 0->1 (max: 500)}]
  Normal   Pulling           87s    kubelet              Pulling image "busybox"
  Normal   Pulled            85s    kubelet              Successfully pulled image "busybox" in 1.92192449s
  Normal   Created           85s    kubelet              Created container volume-mount-ownership-fix
  Normal   Started           85s    kubelet              Started container volume-mount-ownership-fix
  Normal   Pulling           81s    kubelet              Pulling image "carbonplan/trace-python-notebook:latest"
  Normal   Pulled            7s     kubelet              Successfully pulled image "carbonplan/trace-python-notebook:latest" in 1m14.113566996s
  Normal   Created           7s     kubelet              Created container notebook
  Normal   Started           7s     kubelet              Started container notebook

Looking a user created a larger image just after with a different image with similar timings
 date; k  get events --sort-by=.metadata.creationTimestamp   -n prod
Mon Feb 27 03:42:17 PM EST 2023
LAST SEEN   TYPE      REASON                 OBJECT                           MESSAGE
10m         Warning   FailedScheduling       pod/jupyter-pnasrat              0/3 nodes are available: 3 node(s) didn't match Pod's node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.
10m         Normal    TriggeredScaleUp       pod/jupyter-pnasrat              pod triggered scale-up: [{eksctl-carbonplanhub-nodegroup-nb-r5-large-NodeGroup-4564L5LAHRC2 0->1 (max: 500)}]
8m55s       Normal    Scheduled              pod/jupyter-pnasrat              Successfully assigned prod/jupyter-pnasrat to ip-192-168-27-73.us-west-2.compute.internal
8m54s       Normal    Pulling                pod/jupyter-pnasrat              Pulling image "busybox"
8m54s       Normal    TaintManagerEviction   pod/jupyter-pnasrat              Cancelling deletion of Pod prod/jupyter-pnasrat
8m52s       Normal    Created                pod/jupyter-pnasrat              Created container volume-mount-ownership-fix
8m52s       Normal    Started                pod/jupyter-pnasrat              Started container volume-mount-ownership-fix
8m52s       Normal    Pulled                 pod/jupyter-pnasrat              Successfully pulled image "busybox" in 1.92192449s
8m48s       Normal    Pulling                pod/jupyter-pnasrat              Pulling image "carbonplan/trace-python-notebook:latest"
7m34s       Normal    Started                pod/jupyter-pnasrat              Started container notebook
7m34s       Normal    Created                pod/jupyter-pnasrat              Created container notebook
7m34s       Normal    Pulled                 pod/jupyter-pnasrat              Successfully pulled image "carbonplan/trace-python-notebook:latest" in 1m14.113566996s
5m53s       Normal    Killing                pod/jupyter-pnasrat              Stopping container notebook
2m45s       Warning   FailedScheduling       pod/jupyter-maxrjones--science   0/4 nodes are available: 4 node(s) didn't match Pod's node affinity/selector. preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling.
2m37s       Normal    TriggeredScaleUp       pod/jupyter-maxrjones--science   pod triggered scale-up: [{eksctl-carbonplanhub-nodegroup-nb-r5-xlarge-NodeGroup-6SC77IU7K0OZ 0->1 (max: 500)}]
75s         Normal    Scheduled              pod/jupyter-maxrjones--science   Successfully assigned prod/jupyter-maxrjones--science to ip-192-168-6-76.us-west-2.compute.internal
74s         Normal    TaintManagerEviction   pod/jupyter-maxrjones--science   Cancelling deletion of Pod prod/jupyter-maxrjones--science
74s         Normal    Pulling                pod/jupyter-maxrjones--science   Pulling image "busybox"
72s         Normal    Created                pod/jupyter-maxrjones--science   Created container volume-mount-ownership-fix
72s         Normal    Pulled                 pod/jupyter-maxrjones--science   Successfully pulled image "busybox" in 1.738588057s
72s         Normal    Started                pod/jupyter-maxrjones--science   Started container volume-mount-ownership-fix
69s         Normal    Pulling                pod/jupyter-maxrjones--science   Pulling image "carbonplan/main-single-user:latest"
6s          Normal    Started                pod/jupyter-maxrjones--science   Started container notebook
6s          Normal    Created                pod/jupyter-maxrjones--science   Created container notebook
6s          Normal    Pulled                 pod/jupyter-maxrjones--science   Successfully pulled image "carbonplan/main-single-user:latest" in 1m2.515621941s

Reporting user
14m         Warning   FailedScheduling       pod/jupyter-norlandrhagen        0/5 nodes are available: 1 Insufficient memory, 4 node(s) didn't match Pod's node affinity/selector. preemption: 0/5 nodes are available: 1 No preemption victims found for incoming pod, 4 Preemption is not helpful for scheduling.
14m         Normal    TriggeredScaleUp       pod/jupyter-norlandrhagen        pod triggered scale-up: [{eksctl-carbonplanhub-nodegroup-nb-r5-xlarge-NodeGroup-6SC77IU7K0OZ 1->2 (max: 500)}]
13m         Normal    Killing                pod/jupyter-maxrjones--science   Stopping container notebook
13m         Normal    Scheduled              pod/jupyter-norlandrhagen        Successfully assigned prod/jupyter-norlandrhagen to ip-192-168-6-76.us-west-2.compute.internal
13m         Normal    Started                pod/jupyter-norlandrhagen        Started container volume-mount-ownership-fix
13m         Normal    Created                pod/jupyter-norlandrhagen        Created container volume-mount-ownership-fix
13m         Normal    Pulled                 pod/jupyter-norlandrhagen        Successfully pulled image "busybox" in 539.748351ms
13m         Normal    Pulling                pod/jupyter-norlandrhagen        Pulling image "busybox"
12m         Normal    Pulling                pod/jupyter-norlandrhagen        Pulling image "carbonplan/main-single-user:latest"
12m         Normal    Started                pod/jupyter-norlandrhagen        Started container notebook
13m         Normal    Pulled                 pod/jupyter-norlandrhagen        Successfully pulled image "carbonplan/main-single-user:latest" in 559.612548ms
12m         Normal    Created                pod/jupyter-norlandrhagen        Created container notebook
13m         Normal    Pulled                 pod/jupyter-norlandrhagen        Successfully pulled image "carbonplan/main-single-user:latest" in 554.802121ms
12m         Warning   BackOff                pod/jupyter-norlandrhagen        Back-off restarting failed container
13m         Normal    Pulled                 pod/jupyter-norlandrhagen        Successfully pulled image "carbonplan/main-single-user:latest" in 507.098424ms
12m         Normal    Pulled                 pod/jupyter-norlandrhagen        Successfully pulled image "carbonplan/main-single-user:latest" in 530.297722ms

Back-off restarting failed container

Is a good sign i guess, the question becomes how that user failed to start. Maybe user specific config in home folder broke the jupyter server startup somehow?
A way to debug would be to visit /hub/admin and start a server for that user, then inspect logs from the containers failing startup attempt.

Definitely see pending pod for this user

kubectl debug node/ip-192-168-28-180.us-west-2.compute.internal -it --image ubuntu
Looking at users .bashrc and .bash_profile
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/srv/conda/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/srv/conda/etc/profile.d/conda.sh" ]; then
        . "/srv/conda/etc/profile.d/conda.sh"
    else
        export PATH="/srv/conda/bin:$PATH"
    fi
fi
unset __conda_setup

if [ -f "/srv/conda/etc/profile.d/mamba.sh" ]; then
    . "/srv/conda/etc/profile.d/mamba.sh"
fi
# <<< conda initialize <<<

cat .bash_profile

export PREFECT__FLOWS__CHECKPOINTING="true"

source ~/.bashrc

I removed the bashrc renaming it .bashrc_old and it looks like it started
kubectl logs jupyter-norlandrhagen   -c notebook -n prod
[I 2023-02-28 18:57:44.415 SingleUserNotebookApp manager:91] [nb_conda_kernels] enabled, 1 kernels found
[I 2023-02-28 18:57:44.451 SingleUserNotebookApp notebookapp:1593] Authentication of /metrics is OFF, since other authentication is disabled.

kubectl get pod -n prod jupyter-norlandrhagen
NAME                    READY   STATUS    RESTARTS   AGE
jupyter-norlandrhagen   1/1     Running   0          4m27s

Having AWS or Loki logs would have helped speed the time to resolution in this case. That is already tracked by #2272
Rolling back timeout changes IN #2276
Before closing this out I'll try figure a better way to access the nfs homedirs and try improve the docs

Before closing this out I'll try figure a better way to access the nfs homedirs and try improve the docs

The deployer has a command for this i recall, to start a pod with nfs mount. Check deployer --help

Sure, but still room for docs improvements as the first place I looked was the infrastructure guide and all the stuff on nfs was talking about ssh ing into a node and I can't see exec_homes_shell mentioned in the docs
https://infrastructure.2i2c.org/en/latest/search.html?q=+exec_homes_shell

hmm it's here https://infrastructure.2i2c.org/en/latest/sre-guide/support/home-dir.html maybe search is not as robust as I'd hope - I've been relying on it as I haven't internalized the doc layout/structure

Given docs are already present, and the user can spawn theiir server I'm closing this investigation issue

So maybe something is happening in his user space that is causing the timeout?

I still have that spider sense!!! Ja Ja


Conditional select menus

These changes to script/data-harmonizer/index.js and script/data-harmonizer/field_rules.js provide hooks that allow a client application to customize certain column aspects. The client passes the configuration to a DataHarmonizer instance via a new property called field_settings.
This will be useful to NMDC because it will enable our code which sets up a DataHarmonizer instance to customizer the behavior columns that represent GOLD Ecosystem Classification Levels and let them interact with each other. But I think these hooks are generic enough to be useful to other clients of DataHarmonizer as well.
Fixes #302

@ddooley @turbomam I think this is ready for review now

As mentioned in commit, "Field function hooks are now possible by assigning a dictionary of field names as keys in  dh.field_settings{} to an object containing functions such as onChange() which are merged with the template's "field" data structure.  Validation rules trigger onChange() etc. functions when value of field changes.  onChange() can then trigger effects on other field picklist content etc."


label-maker-dask.ipynb example tutorial fails

The https://github.com/microsoft/PlanetaryComputerExamples/blob/main/tutorials/label-maker-dask.ipynb tutorial fails for me.
When running lmj.execute_job() I get many repetition of the following warning:
distributed.worker - WARNING - Compute Failed
Function:  execute_task
args:      ((<function tile_to_label at 0x7f6dd4e45c10>, Tile(x=15550, y=12548, z=15), 'segmentation', [(<class 'dict'>, [['name', 'Roads'], ['filter', ['has', 'highway']]]), (<class 'dict'>, [['name', 'Buildings'], ['filter', ['has', 'building']]])], '[https://qa-tiles-server-dev.ds.io/services/z17/tiles/{z}/{x}/{y}.pbf](https://qa-tiles-server-dev.ds.io/services/z17/tiles/%7Bz%7D/%7Bx%7D/%7By%7D.pbf)'))
kwargs:    {}
Exception: 'SSLError(MaxRetryError(\'HTTPSConnectionPool(host=\\\'qa-tiles-server-dev.ds.io\\\', port=443): Max retries exceeded with url: /services/z17/tiles/15/15550/12548.pbf (Caused by SSLError(CertificateError("hostname \\\'qa-tiles-server-dev.ds.io\\\' doesn\\\'t match either of \\\'*.azure-api.net\\\', \\\'*.portal.azure-api.net\\\', \\\'*.management.azure-api.net\\\', \\\'*.scm.azure-api.net\\\', \\\'*.configuration.azure-api.net\\\', \\\'*.regional.azure-api.net\\\', \\\'*.developer.azure-api.net\\\', \\\'*.data.azure-api.net\\\'")))\'))'

Followed by an SSL error
---------------------------------------------------------------------------
SSLError                                  Traceback (most recent call last)
Input In [7], in <cell line: 1>()
----> 1 lmj.execute_job()

File /srv/conda/envs/notebook/lib/python3.8/site-packages/label_maker_dask/main.py:108, in LabelMakerJob.execute_job(self)
    106 def execute_job(self):
    107     """compute the labels and images"""
--> 108     self.results = dask.compute(*self.tasks)

File /srv/conda/envs/notebook/lib/python3.8/site-packages/dask/base.py:573, in compute(traverse, optimize_graph, scheduler, get, *args, **kwargs)
    570     keys.append(x.__dask_keys__())
    571     postcomputes.append(x.__dask_postcompute__())
--> 573 results = schedule(dsk, keys, **kwargs)
    574 return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])

File /srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/client.py:3010, in Client.get(self, dsk, keys, workers, allow_other_workers, resources, sync, asynchronous, direct, retries, priority, fifo_timeout, actors, **kwargs)
   3008         should_rejoin = False
   3009 try:
-> 3010     results = self.gather(packed, asynchronous=asynchronous, direct=direct)
   3011 finally:
   3012     for f in futures.values():

File /srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/client.py:2162, in Client.gather(self, futures, errors, direct, asynchronous)
   2160 else:
   2161     local_worker = None
-> 2162 return self.sync(
   2163     self._gather,
   2164     futures,
   2165     errors=errors,
   2166     direct=direct,
   2167     local_worker=local_worker,
   2168     asynchronous=asynchronous,
   2169 )

File /srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/utils.py:311, in SyncMethodMixin.sync(self, func, asynchronous, callback_timeout, *args, **kwargs)
    309     return future
    310 else:
--> 311     return sync(
    312         self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
    313     )

File /srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/utils.py:378, in sync(loop, func, callback_timeout, *args, **kwargs)
    376 if error:
    377     typ, exc, tb = error
--> 378     raise exc.with_traceback(tb)
    379 else:
    380     return result

File /srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/utils.py:351, in sync.<locals>.f()
    349         future = asyncio.wait_for(future, callback_timeout)
    350     future = asyncio.ensure_future(future)
--> 351     result = yield future
    352 except Exception:
    353     error = sys.exc_info()

File /srv/conda/envs/notebook/lib/python3.8/site-packages/tornado/gen.py:762, in Runner.run(self)
    759 exc_info = None
    761 try:
--> 762     value = future.result()
    763 except Exception:
    764     exc_info = sys.exc_info()

File /srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/client.py:2025, in Client._gather(self, futures, errors, direct, local_worker)
   2023         exc = CancelledError(key)
   2024     else:
-> 2025         raise exception.with_traceback(traceback)
   2026     raise exc
   2027 if errors == "skip":

File /srv/conda/envs/notebook/lib/python3.8/site-packages/label_maker_dask/main.py:38, in tile_to_label()
     22 """
     23 Parameters
     24 ------------
   (...)
     34     representing the label of the tile
     35 """
     37 url = label_source.format(x=tile.x, y=tile.y, z=tile.z)
---> 38 r = requests.get(url)
     39 r.raise_for_status()
     41 tile_data = mapbox_vector_tile.decode(r.content)

File /srv/conda/envs/notebook/lib/python3.8/site-packages/requests/api.py:75, in get()
     64 def get(url, params=None, **kwargs):
     65     r"""Sends a GET request.
     66
     67     :param url: URL for the new :class:`Request` object.
   (...)
     72     :rtype: requests.Response
     73     """
---> 75     return request('get', url, params=params, **kwargs)

File /srv/conda/envs/notebook/lib/python3.8/site-packages/requests/api.py:61, in request()
     57 # By using the 'with' statement we are sure the session is closed, thus we
     58 # avoid leaving sockets open which can trigger a ResourceWarning in some
     59 # cases, and look like a memory leak in others.
     60 with sessions.Session() as session:
---> 61     return session.request(method=method, url=url, **kwargs)

File /srv/conda/envs/notebook/lib/python3.8/site-packages/requests/sessions.py:529, in request()
    524 send_kwargs = {
    525     'timeout': timeout,
    526     'allow_redirects': allow_redirects,
    527 }
    528 send_kwargs.update(settings)
--> 529 resp = self.send(prep, **send_kwargs)
    531 return resp

File /srv/conda/envs/notebook/lib/python3.8/site-packages/requests/sessions.py:645, in send()
    642 start = preferred_clock()
    644 # Send the request
--> 645 r = adapter.send(request, **kwargs)
    647 # Total elapsed time of the request (approximately)
    648 elapsed = preferred_clock() - start

File /srv/conda/envs/notebook/lib/python3.8/site-packages/requests/adapters.py:517, in send()
    513         raise ProxyError(e, request=request)
    515     if isinstance(e.reason, _SSLError):
    516         # This branch is for urllib3 v1.22 and later.
--> 517         raise SSLError(e, request=request)
    519     raise ConnectionError(e, request=request)
    521 except ClosedPoolError as e:

SSLError: None: Max retries exceeded with url: /services/z17/tiles/15/15550/12548.pbf (Caused by None)

Looks like the  label_source does not work as suggested in the notebook.

label_source: str. A template string for a tile server providing OpenStreetMap QA tiles. Planetary Computer hosts a tile server supporting this format at https://qa-tiles-server-dev.ds.io/services/z17/tiles/{z}/{x}/{y}.pbf

cc @drewbo. Do you have any insight into the qa-tiles-server-dev.ds.io tile server?

Hi @TomAugspurger, sorry I don't have insight into the state of that server (I've transitioned to a new role). cc: @geohacker @batpad for more info

Thanks for the ping @drewbo! @TomAugspurger I'm not sure what the state of that server is, sorry! We noticed this on PEARL as well and I assumed that you guys must have cleaned it up those dev stacks. I think I may have also lost access to the PC devops account.
If the stack wasn't deleted, is it possible for you to paste the logs here? In the meantime, let me try and get access to my devops account.


Change time procces func to admit sec,milli,micro and nano in command queries

What changed?
The time processor is modified to allow queries in which search queries are entered with precision of seconds, milliseconds, microseconds and nanoseconds.

Why?
Currently, the queries made to obtain a list of flows forced to introduce the complete format of the date in nano type, for example:
cadence --domain samples-domain wf list --query "CloseTime >1683388191918908000"
If queries were made with less precision, for example micro<PHONE_NUMBER>918908), milli<PHONE_NUMBER>918) or sec<PHONE_NUMBER>) the query did not return information when comparing with a different type. To avoid this problem, padding is performed by completing the searched dates with zero until the types are equal and all formats can be consulted.

How did you test it?

Potential risks

Release notes
Running various tests via command with different operators (><=) and different date formats
cadence --domain samples-domain wf list --query "CloseTime >1683388191918908000"

Documentation Changes

Pull Request Test Coverage Report for Build 01882696-0db3-475f-9489-ff1d25929905

2 of 6 (33.33%)  changed or added relevant lines in 1 file are covered.
146 unchanged lines in 21 files lost coverage.
Overall coverage decreased (-0.04%) to 57.279%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

common/persistence/elasticsearch/esVisibilityStore.go
2
6
33.33%

Files with Coverage Reduction
New Missed Lines
%

client/history/client.go
2
38.1%

client/history/metricClient.go
2
45.3%

common/membership/hashring.go
2
83.54%

common/persistence/sql/sqlplugin/postgres/db.go
2
85.0%

common/task/weightedRoundRobinTaskScheduler.go
2
89.12%

common/util.go
2
53.44%

service/history/handler.go
2
47.15%

service/history/task/transfer_standby_task_executor.go
2
86.51%

service/matching/matcher.go
2
90.29%

service/history/task/fetcher.go
3
91.75%

Totals

Change from base Build 0188213d-51c6-49cd-b12b-eb555da0ec0b:
-0.04%

Covered Lines:
86984

Relevant Lines:
151861

💛  - Coveralls

Closing the PR as part of old PR cleanup effort. @neil-xie FYI this sounds like good improvement to tackle at some point


Split linux and mac rust jobs

Currently the builds take 14min and 27min, is should be slash.
Ubuntu
Ideal this should be changed to a "build installer" job

MacOS
This should be a basic sanity check just to ensure local development is possible

Blocked by #388 and #379


Global headers

Is there a way to initialize a client with global headers so that you don't have to pass the same one for each request?

Sure! https://github.com/anttiviljami/openapi-client-axios/blob/master/DOCS.md#parameter-optsaxiosconfigdefaults

Also, the client instance is just an axios instance, so you can directly modify client.defaults.headers

Thanks!

This is no longer present in the documentation.
For those looking for it, see from the previous commit: https://github.com/openapistack/openapi-client-axios/blob/235fc7a578f11257c36c09ff27b0e90af68c5291/DOCS.md#authentication


Encourage Posting Issues instead of Reviews

Many reviews on the Marketplace for extensions end up being feature requests, issues, or general requests for help. It would be great if the Marketplace encouraged users to post issues or use the correct forums by showing something like the following on the review screen.

The Issue URL would be the bugs.url property, while the Support URL would be a new support.url property.
  "bugs": {
    "url": "https://github.com/prettier/prettier-vscode/issues"
  },
  "support": {
    "url": "https://stackoverflow.com/questions/tagged/prettier"
  },

This makes sense. Assigning to Backlog.
We plan to do this,


New iOS11 warnings in device page

Has anyone  else seen these warnings, or have I done something wrong ?
The accessory was created after the update to iOS11.
https://photos.app.goo.gl/eWXVSeCjlX4CWYfL2

The device wouldn't show up in the home app, I had to add it by first giving the app the pass code then I was able to add it. But its showing that error .

It is causing any issues with functionality? I haven't updated yet and am wary of doing so. I do recall I got that message when adding the bridge in iOS10, but it doesn't appear on the bridge screen within the Home app like in your picture. Since it's not a true HomeKit device, the warning is expected and as long as it's not causing you any functional issues you definitely don't need to worry about it.

no its not causing any issues. just wanted to mention it

I don't have the photos to back it up, but something similar was written in iOS 10 as well

it also happened in my ipad in ios10,after taken homebridge ,the icon became grey,and the homebride didn't work

I am using a Wink hub in my installation and any of my devices is HomeKit certified. I do get those messages but I am able to control everything without any issues.


Changing Horizon HAL base path

Hi, I am running a Horizon instance that sits behind a load balancer. My problem is that the base path inside the _links attribute gives me the instance URL, which is different than the one I am using (load balancer's).
Does anyone have any suggestions to deal with this kind of architecture?

Probably changing https://github.com/stellar/go/blob/49a0d85c4217dc38f7576c058e31a8653b2f20b2/services/horizon/internal/httpx/base_url.go#L29 to r.Header.Get("Host") should fix this.

Actually, i realized that the actual implementation already works. Using r.Host checks for the Host header in the request. I just had to include it on all requests by default.
Thanks a lot @bartekn!


V2 OIDC userinfo endpoint returns sub claim only, for personal Microsoft accounts.

According to Fetch the OpenID Connect metadata document, the oidc userinfo endpoint is https://graph.microsoft.com/oidc/userinfo. However, this endpoint returns different results for the work accounts and personal Microsoft accounts.
An application has been created in App registrations (Preview). Its supported account types are accounts in any organizational directory and personal Microsoft accounts. The scope is set to openid, profile, email, https://graph.microsoft.com/user.read.
For a work account, the userinfo endpoint returns something like:
{
    "sub": "xxxxx",
    "name": "xxxxx",
    "family_name": "xxxxx",
    "given_name": "xxxxx",
    "picture": "xxxxx",
    "email": "xxxxx"
}

For a personal Microsoft account, only the sub claim gets returned:
{
    "sub": "xxx"
}

I also noticed that the access token of a work account is quite different from the access token of a personal account. The former is a JWT, but the latter is not.
No explanation for above in any Azure AD docs.
Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 628db02d-345a-5dc4-17f0-3d6ff76d7fe5
Version Independent ID: 524b460d-47b7-72c9-e994-ea94b3a9dd2b
Content: Azure Active Directory v2.0 and the OpenID Connect protocol
Content Source: articles/active-directory/develop/v2-protocols-oidc.md
Service: active-directory
Sub-service: develop
GitHub Login: @CelesteDG
Microsoft Alias: celested

@danielhcx
Thanks for your feedback! We will investigate and update as appropriate.

Hi @danielhcx,
For personal Microsoft account like outlook.com, the authentication request is processed by the Identity Provider like live.com. The Idp like Live.com does not return first name, last name and multiple other attributes and that is why you do not have this information automatically populated on the guest account with your azure AD. So if you need this information within your directory, you would have to manually fill these details for the guest accounts or create a script which will update all the Guest user's basic information.
To check all the users information, login with admin/global administrator account and get the jwt token and use that jwt token to hit the below url
https://graph.microsoft.com/v1.0/users.
In the below screenshot, for guest user the surname, givenname are null. After that I updated the jobTitle, so it is populating.

I hope this solves your issue.
Thank you.

Thanks, @MohitDhingra-MSFT .
Creating guest accounts can't be a solution for my case, because I can't predict which live.com account will be used for the "Connect With Microsoft" in my app. Anyway, according to the OIDC spec, the sub claim is the only one required in a userInfo response. https://graph.microsoft.com/oidc/userinfo doesn't break it.
With a live.com user's consent, my app can get more attributes about the user from the endpoint https://graph.microsoft.com/v1.0/me/. However, it's not a standard OIDC userInfo endpoint. To integrate with Azure AD OIDC, it will definitely make our life easier if the /oidc/userinfo endpoint can return the same information from /v1.0/me for a personal account. Do you know if there are any plans for that?

@CelesteDG Could you please help get this checked with engineering teams if we have any plans so that  /oidc/userinfo  endpoint returns the same information as the /v1.0/me for a personal account?

Hi y'all! So is there an update regarding this issue? Like danielhcx said, the user info endpoint (https://graph.microsoft.com/oidc/userinfo) returns this {"sub":"AAAAAAAAAAAAAAAAAAAAA..."} instead of the full user info. I used a personal 'msn'  account for testing. How does one obtain a valid user info payload (similar to the gmail one: {"sub", "name", "given_name", "family_name", "picture", "email", "email_verified", "hd"})? Thanks!

For now, you can get most of the information in the id_token JWT returned from the /token endpoint.

Awesome, thanks!

Comparing the common (https://login.microsoftonline.com/common/v2.0/.well-known/openid-configuration) and a tenant specific (https://login.microsoftonline.com/{tenant}/v2.0/.well-known/openid-configuration) jwks_uri, it looks like the same set of private keys is used to generate an id_token.
Isn't this a security risk?
I understand the JWT validator should verify the "aud" but you would think Azure maintains different key sets for different tenants (if not for the apps)?

My sincerest for just now getting to this issue.
#reassign:@hpsin - Hi @hpsin - Can you please review and advise? Thank you very much.

#pri2

Is there any update on this?
I can see 2 very simple solutions:

add full $select support for oidc/userinfo endpoint
add support for sub on the /me endpoint

Either of these would allow to get details of the users
the full point of having the sub, is that it ensures that it's the correct user
so if i have to implement another call to get more details of the user, where this validation is not in place, than i cannot guarantee that the information relates to the same user

Hey folks - a couple points on /me and /userinfo.

/me returns a set of data that is both beyond what userinfo is allowed to return per spec, and not in the correct format.  So we can't just proxy /userinfo to /me, which would be simplest.

The team working on getting /userinfo support for personal accounts (MSA) is still working to get the userinfo endpoint hooked up.  Once complete, it will just return the subset of information found in the userinfo response for AAD users.

Adding sub to /me is a great idea - I'll suggest it internally.  CC @dkershaw10

@arun-a-nayagam - we may switch to tenant specific keys in the future, but have no plans at this time.  It would create significant strain on large applications to have to download and possibly cache tens of thousands of jwks responses as they got tokens from multiple tenants.  It's not clear the security risk though - all the keys are held in the same (extremely secure) place.  If your concern is cross-tenant attacks allowed via the same signature, you're already assuming that the application is ignoring the actual tenant information inside the token - which means they're already vulnerable to such an issue.

The access token for MSA users is an internal, encrypted format that MSA services understand.  It's expected (and documented in many of the v2 docs).

Any progress on
Adding sub to /me is a great idea - I'll suggest it internally. CC @dkershaw10
?

Not at this time I'm afraid.

As a note - we're in the final stretches of fixing this.  Currently, Userinfo now returns the following for MSA:
"sub": "AAAAAAAAAAAAAAAAAAAAAK0rc3MZ7ZKKpELMYTircG4",
"Name": "Hirsch Singhal",
"Given_name": "Hirsch",
"Family_name": "Singhal",
"Preferred_username"<EMAIL_ADDRESS>"Email"<EMAIL_ADDRESS>"Email_verified": true,
"picture": "https://graph.microsoft.com/v1.0/me/photo/$value"
This sub claim is incorrect, the capitalization is incorrect, and we should not return the verified claim until we support it across the platform.  We will update here when it is fully correct.

Great news :) Thanks.
pon., 27 lip 2020, 18:59 użytkownik Hirsch Singhal<EMAIL_ADDRESS>napisał:

As a note - we're in the final stretches of fixing this. Currently,
Userinfo now returns the following for MSA:
"sub": "AAAAAAAAAAAAAAAAAAAAAK0rc3MZ7ZKKpELMYTircG4",
"Name": "Hirsch Singhal",
"Given_name": "Hirsch",
"Family_name": "Singhal",
"Preferred_username"<EMAIL_ADDRESS>"Email"<EMAIL_ADDRESS>"Email_verified": true,
"picture": "https://graph.microsoft.com/v1.0/me/photo/$value"
This sub claim is incorrect, the capitalization is incorrect, and we
should not return the verified claim until we support it across the
platform. We will update here when it is fully correct.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/MicrosoftDocs/azure-docs/issues/28317#issuecomment-664516994,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFLV5FDV4IPUCTHPKLR44IDR5WW5FANCNFSM4HCIBWKQ
.

And what about adding "sub" to "/me" ?

@dkershaw10 - what's a good place to accrue this feature request on /me?

You mean a reason why I need sub in "/me" response?

@hpsin Changes published for this API? . Still unable to get the user claims for personal account
https://graph.microsoft.com/oidc/userinfo

@VQuery - are you requesting a token on v2 using scope=openid+profile+email? Without the latter two you will not get information about the user per spec.

@hpsin yes I'm using all these scope in my call.
https://login.microsoftonline.com/common/oauth2/v2.0/authorize?client_id=a535eb28-e87a-4565-946d-ebb298df75ac&scope=**openid** **profile** **email**&response_type=code&redirect_uri=https%3A%2F%2Flocalhost%3A44332%2Faccount%2Fsignin-microsoft&state=CfDJ8MNAoixKWeRFra-nk5VtvSaGesm26UAfk2JShOqoC4QJ0pmrBf1U6OLnNnZ16I26isYgzG1B4PBGpw5R1AAvjHAaf37vLRjAh9XQBpfFPsDbvmWElBuTRmIjZlFxBrujbD_Ptc9oDpnpjfxrbC_r5hNt0cf9-YUx7nYOGXv2pJwVEFG740n_T6Jydp-HG6N5vrKvOqq09qUtWQbnMANOwpbYUksjdGHDznT7Z9guJJgWF4u2d7VwQmtRUgHX87L0hh5SjUByGq-mwOZCaePx_gG8Q65ZCAB-1AEb4HGKcF0awS4sYS85s9hgbdbC9ptpdPDDKtWY5q84AANEoZ6BGnS99A4cHrqLvbYJcKLeaFoP8cOoUT7yXdoTyIErkKYszFkD5p9Wopf3UIuFl6LCcbTC33dLVyAwLDnIbKJO4fQuxjidmJ-Mnc5G4YqCzUUqBhe-uHZdCw-hx2Wyd0jyBYoOzIsLBjrHVcjOEEIsrVuB8TtMcpvcnexulSNvEQEcMkI_W7AdUYNuGFZRmZrCKmsjiFJNj1iCot4iETRiTLkuQutEke3pXj3ba5z1srd2VDxqatUU6FgTeWgdvlrKhFRMAoiq95QshX-LSjNbmwJ8PWp68wWmTFgyE0coD3Dcg0uRVB8-rsDxOxVHTXIduV1kaWa8u_KSOaqPcjjCnn4tJtwhe7E7wmujBKn6vJo1zsREoR7jTt5Yzg5AtWBPJ-kca_LZ4YFR6PbJKDiEOUE0bP6vXqyjSoK9odZF668A04qzC2z5AFg9jrN4KLYP7Mj5GeodvOhW7XLGiLB1T2ucn99prw-Uz7hxRcLwIQAQbXRuIn9BXmHi7MMlzs9PZ4h5

#please-close !
The fix for this is now officially rolled out.  The userinfo endpoint for MSAs now returns correct data that matches AAD:
{
"sub": "AAAAAAAAAAAAAAAAAAAAAK0rc3MZ7ZKKpELMYTircG4",
"name": "Hirsch Singhal",
"given_name": "Hirsch",
"family_name": "Singhal",
"email"<EMAIL_ADDRESS>"picture": "https://graph.microsoft.com/v1.0/me/photo/$value"
}
@VQuery - please open a new issue and tag me in it, since it's not related to this one.


Product discounts

I am creating a cart and wanted to add discounts to the products, in the array of options, I put a discount option, but when I update this option, also updates the id, and I change the position of the product in the cart, so investigate your Code, and comment where this function to update the id, my question is how bad is it to do this?
public function updateFromArray(array $attributes)
{
$this->id       = array_get($attributes, 'id', $this->id);
$this->qty      = array_get($attributes, 'qty', $this->qty);
$this->name     = array_get($attributes, 'name', $this->name);
$this->price    = array_get($attributes, 'price', $this->price);
$this->priceTax = $this->price + $this->tax;
$this->options  = new CartItemOptions(array_get($attributes, 'options', $this->options));
    //$this->rowId = $this->generateRowId($this->id, $this->options->all());
}

The code works perfectly !! But is it bad to do so?

This has been asked before actually, so for more details look at previous issues.
But the rowid is regenerated because it is based on all properties (name, price, options, etc) of the item.
Because when I add one red shirt and one yellow shirt to the cart. And next I change the yellow shirt to red, I expect 1 row in the cart with two red shirts and not 2 rows each with the exact same red shirt. So I wouldn't recommend that change.


DevOps Mutualization SIG Meeting - June 16, 2022

Date
June 16 2022 - 12pm ET / 5pm BST
Untracked attendees

Name
Firm
Comment

Meeting notices

FINOS Project leads are responsible for observing the FINOS guidelines for running project meetings. Project maintainers can find additional resources in the FINOS Maintainers Cheatsheet.

All participants in FINOS project meetings are subject to the LF Antitrust Policy, the FINOS Community Code of Conduct and all other FINOS policies.

FINOS meetings involve participation by industry competitors, and it is the intention of FINOS and the Linux Foundation to conduct all of its activities in accordance with applicable antitrust and competition laws. It is therefore extremely important that attendees adhere to meeting agendas, and be aware of, and not participate in, any activities that are prohibited under applicable US state, federal or foreign antitrust and competition laws. Please contact<EMAIL_ADDRESS>with any questions.

FINOS project meetings may be recorded for use solely by the FINOS team for administration purposes. In very limited instances, and with explicit approval, recordings may be made more widely available.

Agenda

[ ] Convene, roll call, welcome new people
[ ] Display FINOS Antitrust Policy summary slide
[ ] Approve previous meeting minutes
[ ] Introduce SIG project board
[ ] Discuss SIG branding and direction - https://github.com/finos/devops-mutualization/issues/54
[ ] Discuss DevOps mutualization ideas for Q3 2022

[ ]  #49
[ ]  #50
[ ]  #51

[ ] AOB, Q&A & Adjourn (5mins)

WebEx info
Webex:
https://finos.webex.com/finos/j.php?MTID=md62056638ba05fbea86b9582df94f807
Dial-in

US +1-415-655-0003 US Toll
UK +44-20319-88141 UK Toll
Access code:<PHONE_NUMBER>

Github Repo: https://github.com/finos/devops-mutualization/
Mailing List: Email<EMAIL_ADDRESS>to subscribe to our mailing list

hello!

Hello 👋🏻

Present!