zookeeper  Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect

2018-07-27 17:19:31 WARN  Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2018-07-27 17:19:31 WARN  handleUpdate failed. caused:Instance must be connected., jobSize:1
com.navercorp.pinpoint.collector.cluster.zookeeper.exception.ConnectionException: Instance must be connected.
at com.navercorp.pinpoint.collector.cluster.zookeeper.DefaultZookeeperClient.checkState(DefaultZookeeperClient.java:273)
at com.navercorp.pinpoint.collector.cluster.zookeeper.DefaultZookeeperClient.exists(DefaultZookeeperClient.java:255)
at com.navercorp.pinpoint.collector.cluster.zookeeper.ZookeeperJobWorker.handleUpdate(ZookeeperJobWorker.java:341)
at com.navercorp.pinpoint.collector.cluster.zookeeper.ZookeeperJobWorker.handle(ZookeeperJobWorker.java:323)
at com.navercorp.pinpoint.collector.cluster.zookeeper.ZookeeperJobWorker.run(ZookeeperJobWorker.java:228)
at java.lang.Thread.run(Thread.java:745)
2018-07-27 17:19:32 WARN  handleUpdate failed. caused:Instance must be connected., jobSize:1
com.navercorp.pinpoint.collector.cluster.zookeeper.exception.ConnectionException: Instance must be connected.
at com.navercorp.pinpoint.collector.cluster.zookeeper.DefaultZookeeperClient.checkState(DefaultZookeeperClient.java:273)
at com.navercorp.pinpoint.collector.cluster.zookeeper.DefaultZookeeperClient.exists(DefaultZookeeperClient.java:255)
at com.navercorp.pinpoint.collector.cluster.zookeeper.ZookeeperJobWorker.handleUpdate(ZookeeperJobWorker.java:341)
at com.navercorp.pinpoint.collector.cluster.zookeeper.ZookeeperJobWorker.handle(ZookeeperJobWorker.java:323)
at com.navercorp.pinpoint.collector.cluster.zookeeper.ZookeeperJobWorker.run(ZookeeperJobWorker.java:228)
at java.lang.Thread.run(Thread.java:745)
2018-07-27 17:19:32 WARN  Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
pinpoint version：1.7.3
zookeeper  version: 3.4.6

Hello, @god8816
Do check your configuration on your zookeeper cluster's address

Hello,@RoySRose
my  zookeeper  single node , not cluster's。

@god8816
I hope you write a little more detailed explanation.
If disconnected from the zookeeper server, this is a normal log.
Therefore, it is impossible to judge whether there is a problem.
Please explain in detail what kind of operation after connection brokened.
It is even better if you give them a situation that can be reproduced.

@koo-taejin  zookeeper
[admin@vm-xaj-vmc-pinpoint-collector-d01 classes]$ telnet zookeeper-test.idc.yst.com.cn 2181
Trying <IP_ADDRESS>...
Connected to zookeeper-test.idc.yst.com.cn.
Escape character is '^]'
it can connected from the zookeeper server

@koo-taejin  why pinpoint can not Connected zookeeper?

@god8816
Please check your zookeepr related settings.
If there are no problems with the settings.
It is recommended to test the connection using the Zookeeper library.

same problem here.
But my pinpoint was working before today. Now It's broken, I check the log. It's the same issue. connects ZK refused.

zookeeper Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
"server null" indicates that you have forgotten to pass the server parameters while running your zookeeper CLI.
Your command should be like this:
./zkCli.sh -server hostname:port
If you don't specify port it will take the default 2181.


start-stop-daemon - ubuntu only?

This is the first release I've seen I think that delegates process monitoring to start-stop-daemon. Isn't it an ubuntu/not-centos only system? Are there some other pros why this is the suggestion vs the other methods used on other releases (run the process and create pid, which monit watches)?

It's unfortunately not available on CentOS, but could conceivably be packaged by a release (we haven't tested doing that, though, so it might not work).
We suggested using start-stop-daemon because it manages a lot of things for release authors that are easy to get wrong, such as locking. We also spiked on creating a custom tool (Sipid) that does this work for you and can be packaged on any non-Windows stemcell.
We should definitely think through the right recommendation for non-Ubuntu stemcells. Thanks for bringing this up.
/cc @xoebus

Hmm. Most of our releases are only expected to run on Ubuntu. Since this is opt-in for releases and not a mandated BOSH change I think recommending start-stop-daemon is still the right thing to do for releases which only need to work on Ubuntu.
Is there an equivalent for CentOS that could be used in releases which need to support both?

I was initially surprised to see the delegation from one process monitoring system - monit - to another. Is this a new idea for this release or are there other bosh releases where this has been used already?
E.g. At a glance I don't see it in concourse/concourse

It has been experimented with in some smaller releases to good effect.
We're looking for ways to get rid of the shame that is pid_utils.sh.
On Fri, Mar 31, 2017 at 2:23 AM, Dr Nic Williams<EMAIL_ADDRESS>wrote:

I was initially surprised to see the delegation from one process
monitoring system - monit - to another. Is this a new idea for this release
or are there other bosh releases where this has been used already?
E.g. At a glance I don't see it in concourse/concourse
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/cloudfoundry/exemplar-release/issues/3#issuecomment-290663791,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AACXntn3lgrUlJF0_yWbvv-XAydipmRQks5rrMYngaJpZM4MhWqe
.

There is a start-stop-daemon fork written in C which we include in src. I would also like to see this included in the Centos stemcells.
Does someone know the reason whyrunit - the supervisor used for bosh-agent & monit - is not used for release jobs? I mean, it's as close to supervision perfection as it comes, and the components that it ships with are something that we've been struggling to plug with cheap knock-offs for years. Monit is stuck at 5.2.5 because AGPL, a new supervisor has been discussed for years but fails to materialise, while we end up with threads like these when runit is able to make all this redundant. Why?

@cppforlife RE: the runit and supervisor decision.

I mean, it's as close to supervision perfection as they come

👍 for runit.


Cannot find entity, even though it is provided in the entities array

Describe the bug
While running schema:create or schema:update I get an error that it cannot find the Client entity and it should be provided in the entities array. The only problem with this is that I actually have provided it in the entities array.
I get this error on a class Client that extends Oidc. Client has a OneToMany relation with ClientCredentials and ClientCredentials (a normal entity) has a ManyToOne relations to Client. This is the relation on which the error occurs.
Stack trace
MetadataError: Entity 'Client' was not discovered, please make sure to provide it in 'entities' array when initializing the ORM (used in ClientCredentials.client)
    at Function.fromUnknownEntity (/path/to/project/node_modules/@mikro-orm/cli/node_modules/@mikro-orm/core/errors.js:121:16)
    at /path/to/project/node_modules/@mikro-orm/cli/node_modules/@mikro-orm/core/metadata/MetadataValidator.js:53:46
    at Array.forEach (<anonymous>)
    at /path/to/project/node_modules/@mikro-orm/cli/node_modules/@mikro-orm/core/metadata/MetadataValidator.js:51:67
    at Array.forEach (<anonymous>)
    at MetadataValidator.validateDiscovered (/path/to/project/node_modules/@mikro-orm/cli/node_modules/@mikro-orm/core/metadata/MetadataValidator.js:51:20)
    at MetadataDiscovery.findEntities (/path/to/project/node_modules/@mikro-orm/cli/node_modules/@mikro-orm/core/metadata/MetadataDiscovery.js:75:24)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at MetadataDiscovery.discover (/path/to/project/node_modules/@mikro-orm/cli/node_modules/@mikro-orm/core/metadata/MetadataDiscovery.js:34:9)
    at Function.init (/path/to/project/node_modules/@mikro-orm/cli/node_modules/@mikro-orm/core/MikroORM.js:40:24)
    at Function.handleSchemaCommand (/path/to/project/node_modules/@mikro-orm/cli/commands/SchemaCommandFactory.js:65:21) {
  entity: undefined
}

To Reproduce
Oidc class:
@Entity({
  tableName: 'oidc',
  discriminatorColumn: 'kind',
  discriminatorValue: 'Oidc',
})
export class Oidc {
  @PrimaryKey()
  id: string;

  @Property({ columnType: 'varchar' })
  kind: Kind;

  @Property({ type: 'json' })
  payload: AdapterPayload;

  @Property()
  expiresAt: Date;
}

Client class:
@Entity({
  discriminatorValue: Kind.Client,
})
export class Client extends Oidc {
  @OneToMany({
    entity: () => ClientCredentials,
    mappedBy: 'client',
    orphanRemoval: true,
  })
  clientCredentials = new Collection<ClientCredentials>(this);
}

ClientCredentials class:
@Entity({ tableName: 'client_credentials' })
export class ClientCredentials {
  @ManyToOne({ primary: true, hidden: true })
  client!: IdentifiedReference<Client>;

  @ManyToOne({ primary: true, hidden: true })
  organization!: IdentifiedReference<Organization>;

  [PrimaryKeyType]: [number, number];

  @Property({ nullable: true })
  deletedAt?: Date;
}

Expected behavior
Should create/update the schema.
Versions

Dependency
Version

node
14.4.0

typescript
3.9.0

mikro-orm
4.1.1

mysql
4.1.1

mikro-orm/nest
4.2.0

Please create full reproduction that I can try, there are lots of details missing here (e.g. the actual configuration).

We created a repository with one very simple step to reproduce the issue, please see this repository.
As outlined in the readme over there, run yarn && yarn mikro-orm schema:update -d
Thank you for looking into this!

Folder based discovery do not support multiple entities in single file, that's the problem here most probably.

I changed the configuration to this:
import { Module } from '@nestjs/common';
import { AppController } from './app.controller';
import { AppService } from './app.service';
import { MikroOrmModule } from '@mikro-orm/nestjs';
import { Client, Oidc } from './entities/Oidc.entity';
import { ClientCredentials } from './entities/ClientCredentials.entity';

@Module({
  imports: [
    MikroOrmModule.forRoot({
      entities: [Oidc, Client, ClientCredentials],
      dbName: 'my-db-name.sqlite3',
      type: 'sqlite',
    }),
  ],
  controllers: [AppController],
  providers: [AppService],
})
export class AppModule {}

However, I am still getting the same error. What I am trying right now is not folder based discovery anymore, right?
So then I moved every entity to its own file, after that it worked fine. Should be mark this as a bug and have a fix for it? At least the error message should be clearer and explain better what the problem is.

The error message is fine, it says the entity was not discovered, and that is exactly what happened (and there is no way to know why). It's quite weird that it is the same with the explicit approach (entities: [Oidc, Client, ClientCredentials]), maybe that is (another) bit different issue, will look into that too (is the logger really telling you those 3 entities are processed? folder discover has different logs that this).
I'd like to support this (having multiple entities in single file), the issue is with base entities that currently do not have any class decorator - and having such is required to be able to allow having multiple entities. Requiring it is a breaking change, will take a look at what's possible, I hope we could still preserve current behaviour, while allowing to import multiple entities (not base entities tho) from single file. We can add a special decorator for abstract entities too, keeping it optional for know, if one want's to have base entity in a file where multiple entities are, then using it will be mandatory.
The main problem is, that currently with folder based discovery, one file always results in (at most) one discovered entity. And it will be always the one that corresponds to the file name, where it is defined. In your example repo, you have Oidc.entity.ts, so only Oidc entity is discovered from that file, rest will be always ignored.

Looking into that issue with explicit approach, and indeed that is a user error - you have 2 configs in your app, one for CLI and one for nest, and you changed only the nest config while testing via CLI - so the error was the same, because nothing changed in the config.
https://github.com/RiskChallenger/nest-mikro-orm-sti/blob/main/src/app.module.ts
https://github.com/RiskChallenger/nest-mikro-orm-sti/blob/main/mikro-orm.config.ts
You should prefer to define the CLI config and import that in the AppModule so you do not fall into this trap :]
Btw I managed to get around that limitations of multiple entities in single file, so in v4.2 this will work fine without changes.

Thanks for fixing


chore(all): Move downcasting to a stable implementation.

Removes the need for raw::TraitObject and Any::get_type_id

can't merge

@seanmonstar fixed


feat(release): android

Production APKs getting built: https://github.com/daniellacosse/outline-client/runs/5042734396?check_suite_focus=true#step:8:18
Getting published to outline-releases: https://github.com/daniellacosse/outline-releases/tree/master/client - ignore the old aab/apk

Were you able to install and use the APK? Was the app functional? Was your server list preserved?

Were you able to install and use the APK? Was the app functional? Was your server list preserved?

I haven't had a chance yet, I had a couple 1:1s after the summit. I might need to screw with my device to root it, since I couldn't get approval to run arbitrary stuff there

I don't think the code is currently working. Please make sure the issues are fixed.

Yeah, sorry, juggling a couple things - set to draft for now.


Adding issue and pull request templates to comply with twitter OSS policies

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Hello, authors of twitter-text! 👋
Can someone please review this pull request? Refer: OSS-3068 :)


Add ability to save resources

Summary
This PR adds the ability to save attachments (and more generally, resources) to disk.

Adds the Resource.save() method
Adds tests for Resource.save() and Resource.read()

Checklist

[x] I have run nox and all the pipelines have passed.
[x] I have made unittests according to the code I have added/modified/deleted.

Related issues
None

Honestly I think I'm just gonna have to leave the tests up to somebody else to take care of cos I can't figure this out. I've emulated what you've done, and I'm getting all sorts of errors and have absolutely no idea what I'm doing wrong.

@parafoxia mind pushing what you have (can be with broken tests too, or just skip them entirely).

@FasterSpeeding, just bumping this as it's ready for review.

Test cases for the specialisations?

Test cases for the specialisations?

Working on those atm :)

@FasterSpeeding tests done. I also found some bugs in the code, so some parts are update too :)


Running Harambot on Unraid

Hello, this is a nice bot, and I have enjoyed using it in the past. Harambot was running last year when I configured it on Unraid.  I was able to add a container and add the environment variables manually. I see there are some new variables for HARAMBOT_KEY, and it wanted a port.  I was able to get the guild configured but now when I run the commands I see the following in the logs of the container:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/discord/app_commands/commands.py", line 857, in _do_call
    return await self._callback(self.binding, interaction, **params)  # type: ignore
  File "/usr/local/lib/python3.10/site-packages/harambot/cogs/yahoo.py", line 44, in wrapper
    guild = Guild.get_or_none(
  File "/usr/local/lib/python3.10/site-packages/peewee.py", line 6838, in get_or_none
    return cls.get(*query, **filters)
  File "/usr/local/lib/python3.10/site-packages/peewee.py", line 6833, in get
    return sq.get()
  File "/usr/local/lib/python3.10/site-packages/peewee.py", line 7285, in get
    return clone.execute(database)[0]
  File "/usr/local/lib/python3.10/site-packages/peewee.py", line 4580, in __getitem__
    self.fill_cache(item if item > 0 else 0)
  File "/usr/local/lib/python3.10/site-packages/peewee.py", line 4628, in fill_cache
    iterator.next()
  File "/usr/local/lib/python3.10/site-packages/peewee.py", line 4683, in next
    self.cursor_wrapper.iterate()
  File "/usr/local/lib/python3.10/site-packages/peewee.py", line 4603, in iterate
    result = self.process_row(row)
  File "/usr/local/lib/python3.10/site-packages/peewee.py", line 7914, in process_row
    data = super(ModelObjectCursorWrapper, self).process_row(row)
  File "/usr/local/lib/python3.10/site-packages/peewee.py", line 7880, in process_row
    result[attr] = converters[i](row[i])
  File "/usr/local/lib/python3.10/site-packages/harambot/database/fields.py", line 16, in python_value
    return f.decrypt(value.encode()).decode()
  File "/usr/local/lib/python3.10/site-packages/cryptography/fernet.py", line 84, in decrypt
    timestamp, data = Fernet._get_unverified_token_data(token)
  File "/usr/local/lib/python3.10/site-packages/cryptography/fernet.py", line 118, in _get_unverified_token_data
    raise InvalidToken
cryptography.fernet.InvalidToken

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/discord/app_commands/tree.py", line 1310, in _call
    await command._invoke_with_namespace(interaction, namespace)
  File "/usr/local/lib/python3.10/site-packages/discord/app_commands/commands.py", line 883, in _invoke_with_namespace
    return await self._do_call(interaction, transformed_values)
  File "/usr/local/lib/python3.10/site-packages/discord/app_commands/commands.py", line 876, in _do_call
    raise CommandInvokeError(self, e) from e
discord.app_commands.errors.CommandInvokeError: Command 'standings' raised an exception: InvalidToken:

@cazar86  what are you using for your bot's database and how did you generate the value for HARAMBOT_KEY? We've introduced encryption for the access_token and refresh_token columns in v0.4.0, HARAMBOT_KEY is what stores the key the bot will use to encrypt and decrypt these values. From the stack trace it looks like it might be an issue with your key. You can use the following command in bash to generate a key that fits the requirements:
head -c 32 /dev/urandom | base64 | tr -d '=' | tr '+/' '-_'

I am using linuxserver/mariadb container. Ver 15.1 Distrib 10.11.8-MariaDB. I tried to use the key provided from your command and still no luck, it doesn't create the tables in my harambot database, it is empty. The first lines I see in the log when launching the harambot container:
[2024-09-06 03:02:08] [WARNING ] discord.ext.commands.bot: Privileged message content intent is missing, commands may not work as expected.
[2024-09-06 03:02:08] [INFO    ] discord.client: logging in using static token
[2024-09-06 03:02:10] [INFO    ] discord.gateway: Shard ID None has connected to Gateway (Session ID: a76e61264f976799f45505402001b7ba).
[2024-09-06 03:02:12] [ERROR   ] discord.client: Ignoring exception in on_ready
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/peewee.py", line 3322, in execute_sql
    cursor.execute(sql, params or ())
  File "/usr/local/lib/python3.10/site-packages/MySQLdb/cursors.py", line 179, in execute
    res = self._query(mogrified_query)
  File "/usr/local/lib/python3.10/site-packages/MySQLdb/cursors.py", line 330, in _query
    db.query(q)
  File "/usr/local/lib/python3.10/site-packages/MySQLdb/connections.py", line 261, in query
    _mysql.connection.query(self, query)
MySQLdb.OperationalError: (4161, "Unknown data type: 'NOT'")

My container variables:
  -e HOST_OS="Unraid"
  -e HOST_HOSTNAME="Unraid_Hostname"
  -e HOST_CONTAINERNAME="Harambot"
  -e 'DISCORD_TOKEN'='Discord Token'
  -e 'YAHOO_KEY'='Yahoo Key'
  -e 'YAHOO_SECRET'='Yahoo Secret'
  -e 'DATABASE_URL'='mysql://user:password@IP:Port/harambot'
  -e 'LOGLEVEL'='DEBUG'
  -e 'HARAMBOT_KEY'='Key provided from command'
  -e 'PORT'='10000'
  -l net.unraid.docker.managed=dockerman
  -l net.unraid.docker.icon='https://raw.githubusercontent.com/DMcP89/harambot/main/assests/harambot-1.jpg' 'dmcp89/harambot:latest'

Can you try generating the key using this python script?
from cryptography.fernet import Fernet fernet_key = Fernet.generate_key() print(fernet_key.decode())


I can't see "retention" in the "Settings page of the new SharePoint admin center"

I was reading this page:
https://docs.microsoft.com/en-us/onedrive/set-retention
Which tells me to go to "Settings page of the new SharePoint admin center".
Once there, I cannot see the "retention" option.  I also tried the "classic settings page" but it's no help.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: c1a01697-2af8-ea42-05bb-7c36afe38870
Version Independent ID: e7f310af-5642-cee5-2048-98b054e54abc
Content: Set the OneDrive retention for deleted users - OneDrive
Content Source: OneDrive/set-retention.md
Service: one-drive
GitHub Login: @kaarins
Microsoft Alias: kaarins

Thank you for submitting feedback.
I think the best way forward is if you open a service ticket in your tenant so this can get resolved ASAP. Based on the outcome let me know if it is something that can be called out in the docs.
Please follow this link to contact support for business products:https://docs.microsoft.com/office365/admin/contact-support-for-business-products
Please keep us posted here on the resolution so we can feed whatever information you discover into the content.


Add append and prepend methods

Add append and prepend LINQ methods from .NET 4.7.1 / .NET Core 1.0 / .NET Standard 1.6.

https://docs.microsoft.com/dotnet/api/system.linq.enumerable.append
https://docs.microsoft.com/dotnet/api/system.linq.enumerable.prepend

Related: #9

Turns out concat was missing all this time too. Added all three.


Change os_release for Tiny

Value should be:
NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Cloud Foundry Tiny"
VERSION_ID="18.04"
HOME_URL="https://github.com/cloudfoundry/stacks"
SUPPORT_URL="https://github.com/cloudfoundry/stacks/blob/master/README.md"
BUG_REPORT_URL="https://github.com/cloudfoundry/stacks/issues/new"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

Note that we should pull non-customized fields from the upstream Bionic image (e.g., NAME, VERSION, ID, ID_LIKE, VERSION_ID, VERSION_CODENAME, UBUNTU_CODENAME)


[WIP] Support multiple underlay networkings on startup/installation

What type of this PR
Feature

See #903 .


use curl instead of ping to check network, permit https

the ping method fails if the webserver is on a non-standard port, and the current method doesn't support https.

Hi @StephanX! Sorry for the long delay on this.
We'd love to take your contribution.  First, we have a contributors agreement that you'll have to sign before we can accept your PR.  You can read about contributions and sign the agreement from this link: http://docs.projectcalico.org/en/latest/contribute.html
Let me know if you have any questions.

Closing as abandoned.


Error while installing

LOGS:
2022-03-21T18:19:15.144Z npm install --no-audit --no-update-notifier --no-fund --save --save-prefix=~ --production --engine-strict<EMAIL_ADDRESS>2022-03-21T18:19:18.414Z [err] npm
2022-03-21T18:19:18.415Z [err]  ERR!
2022-03-21T18:19:18.415Z [err]  code EBADENGINE
2022-03-21T18:19:18.420Z [err] npm
2022-03-21T18:19:18.420Z [err] ERR!
2022-03-21T18:19:18.421Z [err] engine Unsupported engine
2022-03-21T18:19:18.421Z [err] npm
2022-03-21T18:19:18.421Z [err] ERR! engine
2022-03-21T18:19:18.421Z [err]  Not compatible with your version of node/npm<EMAIL_ADDRESS>2022-03-21T18:19:18.421Z [err] npm ERR!
2022-03-21T18:19:18.421Z [err] notsup Not compatible with your version of node/npm<EMAIL_ADDRESS>2022-03-21T18:19:18.421Z [err] npm
2022-03-21T18:19:18.421Z [err]  ERR! notsup Required: {"node":"^0.12.7"}
2022-03-21T18:19:18.421Z [err] npm ERR! notsup Actual:   {"npm":"8.1.3","node":"v16.14.0"}
2022-03-21T18:19:18.436Z [err]
2022-03-21T18:19:18.436Z [err] npm ERR! A complete log of this run can be found in:
2022-03-21T18:19:18.436Z [err] npm ERR!     /root/.npm/_logs/2022-03-21T18_19_18_422Z-debug.log
2022-03-21T18:19:18.448Z rc=1

Hi @prbtt ,
Based on the logs you provided, it seems 'node-red-contrib-calendar-trigger' is not compatible with your version of node-red.
When installing a new package using npm, npm will itself ensure coherence of all enabled nodes....
Since the node-red-contrib-calendar-trigger has not been updated during these last 6 years, I guess it can sound logical...

Hi @prbtt ,
Based on the logs you provided, it seems 'node-red-contrib-calendar-trigger' is not compatible with your version of node-red. When installing a new package using npm, npm will itself ensure coherence of all enabled nodes....
Since the node-red-contrib-calendar-trigger has not been updated during these last 6 years, I guess it can sound logical...

Are there any workarounds? I'm using nodered on Home Assistant

I'm not familiar with node-red running on Home Assistant but I would say to uninstall 'node-red-contrib-calendar-trigger'.
I don't know whether you had it by yourself for a reason (there are probably other nodes to manage such triggers)


Bug: signal - _output_len() causes integer overflow for a large array

In signal/_upfirdn_apply.pyx, _output_len() gives a wrong answer for a large
array (line 67).
Multiplication exceeds the np.intp_t limit, and causes integer overflow.  For example,
_output_len(25228933, 1195968, 1196411, 1195968)
returns 560 etc instead of the correct 1196432 etc.  (Well, in this example, the first
arg 25228933 may seem too large, but it actually happened to me because of the
resulting infinite loop in signaltool.py.  You can see my point anyway)
Reproducing code example:

Sample code to reproduce the problem
_output_len(25228933, 1195968, 1196411, 1195968)

Error message:

infinite loop in signaltool.py - resample_poly()

Scipy/Numpy/Python version information:

1.7.0 1.21.1 sys.version_info(major=3, minor=8, micro=3, releaselevel='final', serial=0)

Hi thanks for posting @ohyeahq. Can you post a minimal example with a call to a public function?
It's correct that in some cases the types will cause overflow. We need to find a balance here and if the types should be updated, we can but it has to be motivated to not penalize normal execution.

Thanks for your reply.  Here's an example:
import numpy, scipy.signal
scipy.signal.resample_poly(numpy.ones(1195968), 1196411, 1195968)
which goes into an infinite loop.
I understand the balance is important, but I consider it a critical bug for the following reasons:
(1) The above example is not unusual (slight down-sampling of 27.1-s audio data).  It worked fine last year, and all suddenly started freezing this time.  It took me HOURS to figure out what is going on and to locate the bug.
(2) It causes an infinite loop in signaltool.py, without any error message or warning.
(3) If I understand correctly, np.intp_t is for indexing, so multiplication without casting to a bigger int type is always error-prone.  At least, some sort of limit checking (before Line 67 of _upfirdn_apply.pyx) is mandatory.
Hope this helps to clarify the point.  Thanks again.

Actually I am not able to reproduce.
import sys, scipy, numpy; print(scipy.__version__, numpy.__version__, sys.version_info)
# 1.7.0 1.21.1 sys.version_info(major=3, minor=8, micro=10, releaselevel='final', serial=0)
import numpy, scipy.signal
scipy.signal.resample_poly(numpy.ones(1195968), 1196411, 1195968)
# array([1.00067514, 1.00078161, 1.0005494 , ..., 1.00048649, 1.00088813,
#        1.00041915])

On which platform are you on? I am on macOS myself.

Actually I am not able to reproduce.
import sys, scipy, numpy; print(scipy.__version__, numpy.__version__, sys.version_info)
# 1.7.0 1.21.1 sys.version_info(major=3, minor=8, micro=10, releaselevel='final', serial=0)
import numpy, scipy.signal
scipy.signal.resample_poly(numpy.ones(1195968), 1196411, 1195968)
# array([1.00067514, 1.00078161, 1.0005494 , ..., 1.00048649, 1.00088813,
#        1.00041915])

On which platform are you on? I am on macOS myself.

Thanks for your time and effort.  That's strange.  Is it possible that your np.intp_t is different from mine?  How can I check the limit value?
Mine is Windows10 64bit Home.  Standard python was installed from the official site (python-3.8.3-amd64.exe).  numpy & scipy was installed via pip.

I can reproduce on Win10 1709, 64bit 1.7.0 1.20.1 sys.version_info(major=3, minor=9, micro=2, releaselevel='final', serial=0)

Thanks @ilayn. Then can you have a look? I don't have a windows machine so I will not be able to look around myself. I could see that the type was not change since a while though. So might come from someplace else than _output_len .

An additional comment: I said "It worked fine last year" but it does not mean I was using the latest version at that time.  The OK version could be older than the one last year.
So I did a small research: I downgraded scipy to see when this bug came in.
scipy 1.4.1 was fine; 1.5.0rc1 was the one, which changed _output_len() in scipy/signal/_upfirdn_apply.pyx.
I also verified an overflow in np.intp_t.  Strange -- but it is actually int32 on my win10x64 platform:
[https://github.com/scipy/scipy/blob/master/scipy/signal/_upfirdn_apply.pyx (Line 61):
_output_len(np.intp_t len_h, np.intp_t in_len, np.intp_t up, np.intp_t down) should return (((in_len - 1) * up + len_h) - 1) // down + 1]
import numpy, scipy.signal
scipy.signal._upfirdn._output_len(2147483647, 1, 1, 1) # gives<PHONE_NUMBER>
scipy.signal._upfirdn._output_len(2147483648, 1, 1, 1) # gives wrong -2147483648
numpy.iinfo(numpy.int32) # is<PHONE_NUMBER>
Hope it helps.

Happened to me as well, Windows 10, Scipy v1.5.0. Reverting to 1.4.1 solved it for me.


Added abbreviated HEX value for color.

For consistency, used white as an example for all 4 examples.

[x] I have read freeCodeCamp's contribution guidelines.
[x] My pull request has a descriptive title (not a vague title like Update index.md)
[x] My pull request targets the master branch of freeCodeCamp.
[x] None of my changes are plagiarized from another source without proper attribution.
[x] My article does not contain shortened URLs or affiliate links.

If your pull request closes a GitHub issue, replace the XXXXX below with the issue number.
Closes #XXXXX

Resolved merge conflict.  Looks good.


Include JSON as a possibility to be guessed for .txt files

Is your feature request related to a problem or area of OpenRefine? Please describe.
OpenRefine does not guess JSON format when opening a .txt file on Windows 10
Describe the solution you'd like
OpenRefine should guess JSON as the bestFormat in the rankedFormats, when a JSON file (or the beginning of the file chars are JSON content, maybe 50 chars?) even when the file extension is not .json but some other extension or no extension in the filename.
Describe alternatives you've considered
Opening the file in Notepad and then looking at the content of the file myself to make a judgement call of what format it might contain and then clicking the appropriate importer selection on OpenRefine's UI.
Test Dataset
Test file used was taken from this existing Issue https://github.com/OpenRefine/OpenRefine/issues/2265#issuecomment-569347365
Additional Context
Breakpoint on L970 in ImportingUtilities.java seems like a good start for debugging from what I saw and where it gave text/line-based as the best format in the ranked list.

Change this line https://github.com/OpenRefine/OpenRefine/blob/4b146acc6ef8a4f41168824db0967e24d7efb45c/main/webapp/modules/core/MOD-INF/controller.js#L238
to "text" instead of "text/line-based". The line-based guesser only handles CSV, TSV, & Fixed.

@tfmorris tried it, didn't make a difference in the ranking, it still automatically choose line-based in the selection after I choose to load that test file.

@tfmorris maybe it should be IM.registerExtension(".txt", "text/json"); since sometimes (oftentimes?) JSON is saved as a regular .txt file on Windows systems?  Probably need to test to see if that causes more problems for regular text detection.

maybe it should be IM.registerExtension(".txt", "text/json");

The vast majority of .txt files are not JSON, so that doesn't seem like a good idea. There's one more line that needs adjusting, but I think @chetan-v is looking into it for #2821.

what to do

What to do?

@chetan-v That doesn't look like the test file that is mentioned in the issue. It should be called "ODC_Parkanlagen.json.txt" if you are using the correct file. Please try that.
When the problem is fixed, OpenRefine should guess "JSON files" instead of "Line-based text files" for that test file.

I send th PR #2831

I am sorry I messed up everything

I am sorry I messed up everything

Don’t worry. Learning is a painful process. You can learn a lot by looking at old PR done by our core developers, Antonin & Tom.
Also, on the gGroups OpenRefine Dev, you can ask questions. It help a lot to start in a good direction.
I am myself a new contributor since a few month. Also, reading the documentation (Wiki, Dev sections) helps a lot. Good luck in this endeavour.
Regards,
Antoine

Hello, @tfmorris @thadguidry
 IM.registerExtension(".json.txt", "text/json");
I add this line.
I used this line in my local repo it will work fine.
For this "ODC_Parkanlagen.json.txt" file and also I test .txt file all worked finely

@chetan-v Thanks but that will not work with a file simply having a .txt extension or no extension.  OpenRefine needs to always inspect .txt files and no extension files in order to properly guess the content in them.  I have updated the issue comment to make this more clear of the problem and expectation.

@chetan-v Thanks but that will not work with a file simply having a .txt extension or no extension. OpenRefine needs to always inspect .txt files and no extension files in order to properly guess the content in them. I have updated the issue comment to make this more clear of the problem and expectation. As well as updated the Test file.

No it works actually I added this line not deleted anything
IM.registerExtension(".json.txt", "text/json"); and IM.registerExtension(".txt", "text");
We need both
Then it work perfectly

@chetan-v Thank you for continuing to work on this. As @thadguidry mentioned, we can't depend on the .json.txt fllename form. It needs to work with just `.txt'.
I suggest you investigate why the format is coming up as Line-based text, even though the .txt extension registration has been fixed to specify just text. There must be some other factor which is triggering that. If you can debug exactly what is causing that, it should lead you to discover what else needs to be changed.

@chetan-v Thank you for continuing to work on this. As @thadguidry mentioned, we can't depend on the .json.txt fllename form. It needs to work with just `.txt'.
I suggest you investigate why the format is coming up as Line-based text, even though the .txt extension registration has been fixed to specify just text. There must be some other factor which is triggering that. If you can debug exactly what is causing that, it should lead you to discover what else needs to be changed.

I understand the problem
text formate guesser method not work properly may be some problem
I find it out very soon


Not intuitive error message when value for property Format is not recognized

When a schema has an unexpected value for "format", it throw an unintuative error message. Perhaps a "unrecognized value of Int32 for format" would be better
TO REPRODUCE:
Schema:
{
  "id": "greg1",
  "$schema": "http://json-schema.org/draft-07/schema",
  "title": "test3 api schema",

  "type": "object",

  "properties": {
    "emplID": {
      "type": "integer",
      "format": "Int32",
      "description": "Employee's ID"
    }
  },
  "required": [ "emplID" ]
}

ERROR GENERATED :
NullReferenceException : Object reference not set to an instance of an object.
stack trace:
at Manatee.Json.Schema.FormatKeyword.Validate(SchemaValidationContext context)
at System.Linq.Enumerable.SelectIPartitionIterator2.ToList()    at System.Linq.Enumerable.ToList[TSource](IEnumerable1 source)
at Manatee.Json.Schema.JsonSchema.Validate(SchemaValidationContext context)
at Manatee.Json.Schema.PropertiesKeyword.Validate(SchemaValidationContext context)
at System.Linq.Enumerable.SelectIPartitionIterator2.ToList()    at System.Linq.Enumerable.ToList[TSource](IEnumerable1 source)
at Manatee.Json.Schema.JsonSchema.Validate(SchemaValidationContext context)
at Manatee.Json.Schema.JsonSchema.Validate(JsonValue json)
at Lcbo.Bimber.API.Utils.SchemaValidator2.Validate(String documentToValidate) in C:\Users\xtgab\Documents_projects\lcbo.bimber.api\Lcbo.Bimber.Api\Utils\SchemaValidator2.cs:line 76

This is a deserialization issue, not a validation issue, so validation error messages are moot here.
How are you deserializing the schema?  The should be a check to validate the schema before deserializing it that doesn't seem to be happening.

Well, the json deserializes OK. it is a valid JSON. The issue is that validation does not recognize the value of the format field.
In fact, it seems to have problems with other format values. I ran into the same issue with:
"format": "date", in something like this :
"Status": {
      "type": "object",
      "properties": {
        "Original_Hire_Date": {
          "type": "string",
          "format": "date",
        },
        "Start_Date": {
          "type": "string",
          "format": "date",
        }
      },
      "required": [
        "Original_Hire_Date"
      ]
    },

Here is my code to deserialize the schema and do the validation:
                var serializer = new JsonSerializer();
                var json = JsonValue.Parse(schemaRaw);
                var mySchema = serializer.Deserialize<Manatee.Json.Schema.JsonSchema>(json);
                var val = JsonValue.Parse(documentToValidate);
                SchemaValidationResults results = mySchema.Validate(val);

Where are you getting schemaRaw?
The thing here is that the schema isn't being validated.  If you're loading from a file, I suggest that you use JsonSchemaRegistry.Get() and pass in the file name as a URI.  This way the schema is validated.  You'll find that Int32 and date are not supported string format values (although date-time is supported).  If it's hard-coded, I suggest that you build the schema in code rather than as a string.
You can add support for new string formats by using one of the StringFormat constructors. These will also register the new format for deserialization.
All that said, I did find a bug in that the system doesn't throw an exception when deserializing an unrecognized string format.  I'll fix that.

@GregBalajewicz ping!

I'm putting an isKnownFormat boolean in the AdditionalInfo dictionary on the result object.  That should indicate to you whether the format was recognized.

https://www.nuget.org/packages/Manatee.Json/
Version 10.0.5 contains your fix.

Sorry @gregsdennis! I was not able to get to this.
I've consumed your update and it works as expected. However, personally, I think the failure should occur when the schema is read in, and an unrecognized format is encountered. Right now, the schema is parsed, but failure occurs at validation time - format not recognized. IMO, the failure should occur during schema load in time, but this is not a big deal.
And to answer your question, my schema is coming from an external file. I cannot build it in code

JSON Schema specifies

If the type of the instance to validate is not in this set, validation for this format attribute and instance SHOULD succeed.

These means that I shouldn't throw exceptions when I don't recognize a format.  Technically, I need to revisit what I did for this change.  Unrecognized formats should succeed by default, and this change makes them fail by default.
I'll add in an option to indicate whether they should succeed or fail, though.

Additionally, the spec also says

Implementations MAY support the "format" keyword as a validation assertion. Should they choose to do so:

they SHOULD implement validation for attributes defined below;
they SHOULD offer an option to disable validation for this keyword.

@GregBalajewicz with v10.1.0 I've added an option that will trigger an exception when deserializing a schema with an unknown format.


participations not added to student sometimes

when making a new a participation, the participation is sometimes not added to the participations of the student. On refresh, the participation is added to the student

When I do a get request to a student which has just added to 2 projects, I get the following answer:


Extended to make the saveAsHadoop file an async operation to leverage th...

...e FutureAction
@markhamstra @vnivargi create a wrapper for saveAsHadoopFile so that we can pass in the post complete function to suit our purpose of making the save action cancellable, sparky functional test is good with this locally

This doesn't look to me to be the correct approach.  Since all the saveAsHaddopFile functions eventually end up in saveAsHadoopDataset, the essence of the problem is to create a saveAsHadoopDatasetAsync that evaluates to a Future[Unit] instead of Unit.  saveAsHadoopDataset is composed of two parts: 1) runJob of writeToFile; and then 2) writer.commitJob().  Instead of all of this extending FutureAction in SimpleFutureWithPostCompleteAction (which, btw, is going to have a very difficult time making it into the Apache distribution with all of this code duplication), it looks to me like you just need a FutureAction that is a for comprehension over two Futures:
for {
  writeResult <- sparkContext.submitJob( ...writeToFile...)
  commitResult <- writer.commitJob()
} yield commitResult

The only real complication that I am seeing is the need to yield a FutureAction instead of a Future.  To address that, I think that all you need to do is make the writeToFile part a Future[JobWaiter[_]] and the commitJob part a SimpleFutureAction[Unit] constructed from that JobWaiter and the identity function.  That way a cancel() of the result of the for comprehension ends up calling cancel() on the JobWaiter from writeToFile.  In other words, the writeToFile part needs to expose the JobWaiter as soon as it is created (and typically before the awaited job actually completes) and the writer.commitJob part then produces a FutureAction whose cancellation cancels the exposed JobWaiter and whose successful completion is the committing of the awaited job results.

Yeah, think you are right, the extension to SimpleFutureAction does seems not really needed given that we already have the waiter after submitting job to scheduler, we can re-use the SimpleFutureAction to simplify things. Let me close this and test it out.

Re-open with the updates per @markhamstra review comments

replaced with this https://github.com/clearstorydata/spark/pull/43


Contribution to AI-mergency Control Room

@twgoetz @alexlang74 @szerfass @jstuetzle @krook
I hope this message finds you in great health. I am part of SJSU graduate students group who are interested in contributing to this incredible idea. Hence, I am hereby writing this message to ask for your ideas if there is anything you recommend for contribution in this project.
Looking forward to hearing from you soon.
Regards,

@FanusArefaine welcome! I hope you are well as well. I defer to the other 4 folks to reply. Great to hear about the interest.

@krook I Thank you very much for your instant reply. I look forward to hear from your folks soon as the project should be finalized within the coming five weeks.
Regards,

@twgoetz @alexlang74 @szerfass @jstuetzle Thoughts on the best way for @FanusArefaine and others to contribute?


Add SetRelease and SetEnvironment

resolve #22

Coverage decreased (-1.4%) to 62.731% when pulling 0d9c5675e110df76755e787f1c1260bfc6ba4194 on issue/22 into 3723d2d7cc850b6b5c738de1df8463211fb400c2 on master.

Coverage decreased (-1.4%) to 62.731% when pulling 0d9c5675e110df76755e787f1c1260bfc6ba4194 on issue/22 into 3723d2d7cc850b6b5c738de1df8463211fb400c2 on master.

Note: https://github.com/evalphobia/logrus_sentry/releases/tag/v0.2.6


Add @lit/localize to the monorepo

The last 3 commits are new. Everything else are original commits from the original repo.

@aomarks I just created 3 PRs about this new package:
https://github.com/Polymer/lit-html/pull/1409
https://github.com/Polymer/lit-html/pull/1410
https://github.com/Polymer/lit-html/pull/1411


Added Java time module to object mapper

Signed-off-by: Asif Sohail Mohammed<EMAIL_ADDRESS>Description
Fix: Found an issue with peer forwarder server serialization and deserialization of EventMetaData with Instant type timeReceived.
Issues Resolved
Check List

[ ] New functionality includes testing.
[ ] New functionality has been documented.

[ ] New functionality has javadoc added

[ ] Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

I don't think it would impact anything unless an object is an Instant, I might be wrong here.
Both DataPrepperAppConfiguration and  MetricsConfig uses the same ObjectMapper with YAMLFactory. I could create a new one in PeerForwarderAppConfig so that it won't be coupled to them.
I'll test the changes and push a new revision.

I don't think it would impact anything unless an object is an Instant, I might be wrong here.
Both DataPrepperAppConfiguration and MetricsConfig uses the same ObjectMapper with YAMLFactory. I could create a new one in PeerForwarderAppConfig so that it won't be coupled to them.
I'll test the changes and push a new revision.

I'd expect that the metrics include date values. Though, I don't know if they are Java 8 dates or the old java.util.Date values.

Codecov Report

Merging #1779 (2f29334) into main (7e2331f) will decrease coverage by 0.04%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##               main    #1779      +/-   ##
============================================
- Coverage     93.93%   93.88%   -0.05%
- Complexity     1449     1450       +1
============================================
  Files           188      188
  Lines          4252     4254       +2
  Branches        343      343
============================================
  Hits           3994     3994
- Misses          177      179       +2
  Partials         81       81

Impacted Files
Coverage Δ

...aprepper/peerforwarder/PeerForwarderAppConfig.java
93.75% <100.00%> (+0.89%)
:arrow_up:

...rwarder/discovery/AwsCloudMapPeerListProvider.java
92.59% <0.00%> (-2.47%)
:arrow_down:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


TEP-0075: Object variable replacement on Pipeline/PipelineRun level

Changes
Replace the following references with actual value

the reference to the whole object param defined in PipelineSpec
the reference to the individual keys of an object param

Submitter Checklist
As the author of this PR, please check off the items in this checklist:

[ ] Docs included if any changes are user facing
[ ] Tests included if any functionality added or changed
[ ] Follows the commit message standard
[ ] Meets the Tekton contributor standards (including
functionality, content, code)
[ ] Release notes block below has been filled in
(if there are no user facing changes, use release note "NONE")

Release Notes

/test pull-tekton-pipeline-alpha-integration-tests

/assign @ywluogg
/assign @dibyom

/assign @ywluogg

/retest

/test pull-tekton-pipeline-go-coverage

/retest

link back to #4723

/lgtm

/test tekton-pipeline-unit-tests

/retest

/retest pull-pipeline-kind-k8s-v1-21-e2e

/test pull-pipeline-kind-k8s-v1-21-e2e

/retest

/retest

/test pull-tekton-pipeline-unit-tests

/test pull-tekton-pipeline-build-tests
/test pull-tekton-pipeline-unit-tests

/test pull-tekton-pipeline-unit-tests

/retest


Add magnitude encoding

Per request from a user, add magnitude encoding support

Added
https://github.com/mit-han-lab/torchquantum/commit/f2033885d320954309635a83a5ff906b23afd7a2


support Direction: RTL

Version(if relevant): 5.1.2
Environment(if relevant): All
Code(if relevant):
html { direction: rtl; }

Expected:
Everything to work about the same
Actual:
There is a side-scroller that in the end shows the drop-down always open.
You can test it with your examples (happens in all - angular, vue, react) just inspect and change the html direction to rtl

I'm not sure what is correct when direction is rtl, but I'm willing to take PR about this.


AUTHORS: add Petr as contributor

Add Petr to the list of contributors.

@arkamar feel free to accept or reject this patch adding you as contributor in AUTHORS. If you want me to add an email, let me know (and which one) and I will amend it.
Thx!


javascript metric in many cases empty

Hi,
I have noticed that the javascript custom metric get an improvement in https://github.com/HTTPArchive/custom-metrics/pull/42 , which now also collect statistics over the server generated HTML.
That's really cool since that was something we have missed beforehand.
However, it seems that the JavaScript metric is, for example, on the url https://www.decathlon.de/ just empty.
SELECT
  JSON_EXTRACT_SCALAR(payload, '$._javascript') AS javascript,
FROM
  `httparchive.pages.2022_06_01_*`
where url like 'https://www.decathlon.de%'
limit 1

If have quickly validated that finding, and it looks like that this not unique to that url.
SELECT
  count(*) as records,
  count(JSON_EXTRACT_SCALAR(payload, '$._javascript')) AS javascript,
FROM
  `httparchive.pages.2022_06_01_*`

Shows that the metric is on ~2M cases null .
Also, the _wpt_bodies metric reports some errors in that cases.
{
  "log": [
    {
      "type": "error",
      "context": "anchors",
      "exception": {},
      "message": "Cannot read properties of null (reading 'length')"
    },
    {
      "type": "error",
      "context": "title",
      "exception": {},
      "message": "Cannot read properties of null (reading 'length')"
    },
    {
      "type": "error",
      "context": "meta_description",
      "exception": {},
      "message": "Cannot read properties of null (reading 'length')"
    },
    {
      "type": "error",
      "context": "hreflangs",
      "exception": {},
      "message": "Cannot read properties of null (reading 'length')"
    },
    {
      "type": "error",
      "context": "headings",
      "exception": {},
      "message": "Cannot read properties of null (reading 'length')"
    },
    {
      "type": "error",
      "context": "structured_data",
      "exception": {},
      "message": "Cannot read properties of null (reading 'length')"
    },
    {
      "type": "error",
      "context": "raw_html",
      "exception": {},
      "message": "Cannot read properties of null (reading 'length')"
    },
    {
      "type": "error",
      "context": "canonicals",
      "exception": {},
      "message": "Cannot read properties of null (reading '0')"
    },
    {
      "type": "error",
      "context": "robots",
      "exception": {},
      "message": "Cannot read properties of null (reading '0')"
    },
    {
      "type": "error",
      "context": "visible_words",
      "exception": {},
      "message": "Cannot read properties of null (reading 'length')"
    },
    {
      "type": "error",
      "context": "general",
      "message": "Failed to create the custom metrics object: Cannot read properties of null (reading 'length')",
      "exception": {}
    }
  ]
}

It looks like that the underlying issue is that the magic $WPT_REQUESTS is maybe null?
We see a similar issue on several other domains where we are
working with. All of them are quite large E-Commerce companies in Germany / EU. Not sure but that can of cause bias statistics which you may collect over that metrics.
It would be cool if someone can take a look into this or maybe give me a pointer how I can help to find the underlying issue,
Thanks in advance

There were some fixes in September to add bodies from netlog requests (service worker and a few others) which also added requests to the list of requests that come from netlog instead of dev tools and they may not have all of the same fields.
It looks like the JS metric probably doesn't react well if fields are missing.
i.e.
  let scripts = $WPT_BODIES
    .filter((request) => request.type == "Script")
    .map((file) => {
      return { url: file.url, body: file.response_body };
    });

What will that do if 'type' isn't available on the request?  At a minimum we should probably skip any records that don't have the fields we're interested in but for bonus points, there could be a pre-pass that populates 'type' from the sec-fetch-dest request header if it is present.

CC: @kevinfarrugia in case he has any insights on this custom metric

The issue is related to the custom metric, most likely the change in #41.
The root cause of the issue is because some scripts do not have a body and the custom metric is throwing an unhandled exception on:
  let sourcemapURLs = scripts
    .map((n) => {
      if (n) {
        let url = n.body?.match(sourcemapRegex)?.[1];

This happens in cases when the response does not include a body, for example a redirect (and possibly a 404). Take a look at Request 85 https://fast.smarketer.de/api/js/fast.js on https://decathlon.de as an example.

Nice that was fast! Thanks 👍
If I understand the fix correctly it will Not fix the wpt-body metric. Does the metric have a similar issue?

@fbuecklers can you help me understand the issue with _wpt_bodies or where you are seeing the error logs please (link to WPT if possible)? From what I see, it shouldn't be related to the JavaScript custom metric though.

Hi @kevinfarrugia
Actually, I have found that in the database as well:
SELECT
  JSON_EXTRACT_SCALAR(payload, '$._wpt_bodies') AS wptBodies,
FROM
  `httparchive.pages.2022_09_01_*`
where url like 'https://www.decathlon.de%'
limit 1

Will actually return this:
{
  "log": [
    {
      "type": "error",
      "context": "anchors",
      "exception": {},
      "message": "Cannot read properties of null (reading 'length')"
    },
    {
      "type": "error",
      "context": "title",
      "exception": {},
      "message": "Cannot read properties of null (reading 'length')"
    },
    {
      "type": "error",
      "context": "meta_description",
      "exception": {},
      "message": "Cannot read properties of null (reading 'length')"
    },
    {
      "type": "error",
      "context": "hreflangs",
      "exception": {},
      "message": "Cannot read properties of null (reading 'length')"
    },
    {
      "type": "error",
      "context": "headings",
      "exception": {},
      "message": "Cannot read properties of null (reading 'length')"
    },
    {
      "type": "error",
      "context": "structured_data",
      "exception": {},
      "message": "Cannot read properties of null (reading 'length')"
    },
    {
      "type": "error",
      "context": "raw_html",
      "exception": {},
      "message": "Cannot read properties of null (reading 'length')"
    },
    {
      "type": "error",
      "context": "canonicals",
      "exception": {},
      "message": "Cannot read properties of null (reading '0')"
    },
    {
      "type": "error",
      "context": "robots",
      "exception": {},
      "message": "Cannot read properties of null (reading '0')"
    },
    {
      "type": "error",
      "context": "visible_words",
      "exception": {},
      "message": "Cannot read properties of null (reading 'length')"
    },
    {
      "type": "error",
      "context": "general",
      "message": "Failed to create the custom metrics object: Cannot read properties of null (reading 'length')",
      "exception": {}
    }
  ]
}

Also some other metrics seems to be broken as well:
 "_robots_meta": {
    "error": "TypeError: Cannot read properties of null (reading 'filter')"
}

"_structured-data": "{\"structured_data\":{\"type\":\"error\",\"context\":\"structured_data\",\"exception\":{},\"message\":\"Cannot read properties of null (reading 'length')\"},\"log\":[{\"type\":\"error\",\"context\":\"structured_data\",\"exception\":{},\"message\":\"Cannot read properties of null (reading 'length')\"}]}",

"_valid-head": {
    "error": "TypeError: Cannot read properties of null (reading '0')"
},

Hm, have further looked into that, and it looks like that this was not the case in the httparchive.pages.2022_06_01_ table. So maybe it breaks recently though the changes which @pmeenan mentioned.

@kevinfarrugia To add some numbers here:
The following query gives the results regarding the different metric errors:
SELECT
  count(*) as records,
  countif(JSON_EXTRACT_SCALAR(payload, '$._javascript') is null) AS javascriptError,
  countif(JSON_EXTRACT_SCALAR(payload, '$._wpt_bodies') like '%Cannot read properties of null%') AS wptBodiesError,
  countif(JSON_EXTRACT_SCALAR(payload, '$._robots_meta.error') like '%TypeError: Cannot read properties%') AS robotsMetaError,
  countif(JSON_EXTRACT_SCALAR(payload, '$._structured-data') like '%Cannot read properties%') AS structedDataError,
  countif(JSON_EXTRACT_SCALAR(payload, '$._valid-head.error') like '%TypeError: Cannot read properties%') AS validHeadError,
FROM
  `httparchive.pages.2022_06_01_*`

Results in:
[{
  "records": "13371217",
  "javascriptError": "1910134",
  "wptBodiesError": "4078",
  "robotsMetaError": "188203",
  "structedDataError": "16",
  "validHeadError": "125579"
}]

With the except of the javascript metric, the numbers looks quite unobtrusive.
However, if I run the same query on the 2022_09_01 dataset I get the following result:
SELECT
  count(*) as records,
  countif(JSON_EXTRACT_SCALAR(payload, '$._javascript') is null) AS javascriptError,
  countif(JSON_EXTRACT_SCALAR(payload, '$._wpt_bodies') like '%Cannot read properties of null%') AS wptBodiesError,
  countif(JSON_EXTRACT_SCALAR(payload, '$._robots_meta.error') like '%TypeError: Cannot read properties%') AS robotsMetaError,
  countif(JSON_EXTRACT_SCALAR(payload, '$._structured-data') like '%Cannot read properties%') AS structedDataError,
  countif(JSON_EXTRACT_SCALAR(payload, '$._valid-head.error') like '%TypeError: Cannot read properties%') AS validHeadError,
FROM
  `httparchive.pages.2022_09_01_*`

[{
  "records": "25244732",
  "javascriptError": "7342080",
  "wptBodiesError": "4681181",
  "robotsMetaError": "4928074",
  "structedDataError": "4675306",
  "validHeadError": "5387378"
}]

So in around 1/5 of all tests the different metrics raise an exception.
Should I open a new issue for it?

Thanks @fbuecklers yes that looks serious and worth investigating. The dataset did grow by about 2x over this time period, but the error rate seems to have increased disproportionately.


Support swift 3

Will you support swift 3? It would be very helpful

I intend to, but I'm busy with other work at the moment, so I can't give a timetable. It should take little, however, to port the code to Swift 3. Pull requests are welcome.

I was not able to easily convert the source to swift 3. It is needed for my projects as well. It will be a little bit before I will have time to learn how to convert it.

I'll try to carve out some time to work on this. Can you give me a heads up on what the hurdles are going to be?

26 issues after an auto convert. You have to convert each target schema as well to be updated. I can do that for tvOS after you get it going.
https://github.com/spunkedy/SwiftDDP
It's checked in not working to my master. I have specific versions tagged in my cartfile for the dependencies. So if you want a "psuedo" starting point grab my cartfile.
The logger dep is in a feature branch quite active. I think it should be soon.
It appears to be already pushed 4 hours ago.
https://github.com/DaveWoodCom/XCGLogger/commit/636e042ce15a241ddeffc62f80839fadcc1bfa90

Could we have a timeline as to when this will be done? This library is quite important for me and I am having no luck fixing it myself.

I have fixed the library so that it works fine with my app and have been using it for a few weeks. If any one wants access to it please send me a message.

@JoshuaUnrau could you send a pull request?

@Sef1995 @JoshuaUnrau Indeed, if you send a pull request, I'll merge it into a Swift 3 branch.

I can test it if you do.

On Oct 27, 2016, at 9:05 PM, Peter Siegesmund<EMAIL_ADDRESS>wrote:
@Sef1995 @JoshuaUnrau Indeed, if you send a pull request, I'll merge it into a Swift 3 branch.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

Sorry I am relatively inexperienced with GitHub. Which branch should I
request to pull from? Anyway I attached my files to this email.
On Thu, Oct 27, 2016 at 7:05 PM, Peter Siegesmund<EMAIL_ADDRESS>wrote:

@Sef1995 https://github.com/Sef1995 @JoshuaUnrau
https://github.com/JoshuaUnrau Indeed, if you send a pull request, I'll
merge it into a Swift 3 branch.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/siegesmund/SwiftDDP/issues/74#issuecomment-256818695,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AJIvWm_JrAoiEJwCOpOV-MQiWAbD6fR6ks5q4ViGgaJpZM4J6S7K
.

Here are a couple resources:
https://help.github.com/articles/creating-a-pull-request/
https://www.digitalocean.com/community/tutorials/how-to-create-a-pull-request-on-github
I don't see the source attached because it is showing up on the comments section

I just submitted a pull request for the Swift 3 upgrade if anyone is interested
The pull request can be found here: #82

Reviewed and merged.


[KYUUBI#1253] Support use helm to deploy kyuubi server on k8s

Why are the changes needed?

We usally use helm to manager application yaml on k8s, we can use helm to delopy kyuubi server too.
Usages:
helm install kyuubi ${kyuubi_helm_url} -n kyuubi

helm uninstall kyuubi -n kyuubi

How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[X] Run test locally before make a pull request

thanks, merged to master


Import CMake target as an INTERFACE instead of a SHARED library

Under Windows, importing TBB as a CMake target with
target_link_libraries(executable tbb)
does not work (See also #8 ).
The problem, if I understand it correctly, is that an imported shared target can refer to only one file, which poses a problem with IMPORTED_LOCATION when there are several dll files, as is the case for the debug version or when one has several components.
With this PR, tbb is imported as an INTERFACE library instead, with all dependencies set correctly.
This PR solves #8.

You are correct about imported libraries only allowing a single target. Unfortunately interface libraries don't allow the library to be exported either.
The correct solution is to import each library individually and link them to a single interface like library target.  This is what it did at one point.
I will take a look at this in more detail.

Yes, you are correct, I saw that the tests didn't pass just before your comment.
I was a bit curious why this doesn't work so I added a few commands in FindTBB.cmake to export the target; the tests now pass but I am not sure how much sense this makes :-/.

Ok, now it appears to be working :)

I will review your changes in more detail later tonight before merging the change. It looks like you are on the right path.


protocol: add file edit notification protocol for KBFS

Here's how I envision we use this protocol:

We add a new notification channels: kbfsrequests.
KBFS subscribes to kbfsrequests.
The GUI subscribes to the kbfs channel, as usual, when it starts up.
After the kbfs subscription, (and until it sees its first complete list of notifications, periodically), the GUI sends requests via kbfsrequests for each favorite it cares about:

FSEditListRequest(FavoriteName, requestID)
FSSyncListRequest(FavoriteName, requestID)

When KBFS receives one of these requests, it responds on the kbfs notification channel with:

FSEditListRequest: a list of up to the last N file edit notifications per writer for the TLF.
FSSyncListRequest: a complete list of outstanding sync operations for the TLF.

Each of the above requests causes KBFS to subscribe for updates from the mdserver for that TLF.
For every TLF to which KBFS is currently subscribed, KBFS sends out the following notifications on the kbfs notification channel:

Notifications for every file edit operation, as they happen, per-TLF.  Each notification message can actually contain a list of individual edit notifications, since each MD revision can include edits to several files (e.g., conflict resolution).
Notifications when an async write operation is queued up.
Notifications when an async write operation finishes (or makes progress).

It will be up to the GUI to improve performance by keeping a cached copy of the complete list of changes, and modify it correctly as new notifications roll in.  It would perform badly to request the complete list for every render.

Issue: KBFS-1300

some q's:

will this perform after a fresh install? For example, if I install keybase and then click my widget, will it have activity? it seems either of these would be ok:

it's just fast :-)
records are incomplete for a short period as it's building a history/cache
As opposed to:
the widget takes a long time to load the first time I click it
the widget never gets anything from before my install

Nojima mentioned this but I wonder too about the per-TLF subscription...if a user has 500 TLF's they've worked in are they making 500 calls to this?

Should bookkeeping for this be done in desktop or core?

Thanks for thinking about this @malgorithms.
Question 1 and question 2 are very related.  I feel pretty strongly that the responses at least should be per-TLF, because it will allow the possibility for the GUI to show an incomplete history before the data for all TLFs is available.  Each TLF represents (at least one) round-trip to the server (parallelized when possible of course), and if we wait for them all to come back, the GUI will lag significantly while it blocks on the slowest response.
I prefer a per-TLF request as well (rather than something that says "give me everything"), because it allows for more possibilities (say a GUI-less application that cares only about specific TLFs, or completely separate GUIs for private vs public TLFs, etc).  That said, if for some reason it's a burden for the GUI to send out that many requests, we can talk about having one big request, but it's hard to imagine a world where the local RPCs will be a bottleneck.
As far as I know, a fresh install vs. a restart are identical cases.  We don't have any disk caches.  Even when we do, I assume you'd prefer that we check with the server first to get the latest updates for each TLF?  We'd at least have to do this for new favorites not yet seen by this device.
As for keeping the full history for the widget up-to-date (I assume that's what you mean by "bookkeeping"), I think the GUI is the best place -- it will be processing the followup notifications anyway in order to show toast notifications, and fetching the full history over RPC seems needlessly wasteful to me.  But there might be other considerations I'm not aware of.  If the GUI can't handle this for some reason, and would rather load it by RPC for every request, then we might as well get it all the way from KBFS, which will have up-to-date per-TLF histories already.

One reason for bookkeeping in core is if we want a CLI interface that does roughly the same thing as the widget.  But if that interface is very different (i.e., requests full dumps every time), then that reasoning falls apart.
In rekey, we decided for better or worse to have core do most of the bookkeeping for the reasons I mentioned just above.

Also, just to double check: is there still a longer term plan to move to a chat-based data structure, and if so, how does that change the design above?

Sure.  I was guessing the CLI interface would be more like a tail situation where it prints the initial history and then updates.  But if you plan on some fancy curses thing, then I could see that argument.  Except if the history is big, sending it to the GUI on every window refresh seems like overkill, but I don't really know the tradeoffs there.

Also, just to double check: is there still a longer term plan to move to a chat-based data structure, and if so, how does that change the design above?

Yes, still the plan.  I tried to design this so it's agnostic to the underlying impl as much as possible.  Even with the chat-based way, I think there will still be a per-TLF RTT, so I believe this interface still holds.

Almost certainly no curses, thought that would be cool!
OK, cool, thanks!

Is it ok to have a hangout to discuss this stuff further?  @malgorithms has a bunch of UX/performance questions.

Re: CLI implementation -- thinking about it more, I guess there could be a CLI command that just prints out the history on demand each time, rather than following it live.  That's probably what you're thinking of.  So yeah, in that case it could make sense to do bookkeeping in core, but I think we'd still need to play around with what happens when the GUI is fetching it on every refresh.

Is it ok to have a hangout to discuss this stuff further?

Yeah whenever.  After lunch PT would work for me.

yeah, in a couple hours would be better than now, just because I'm trying to get some search code out the door.
I have very few opinions about the architecture, I only have 2 general considerations:

it gets correct answers fast
we don't have to reinvent the wheel elsewhere (like for the CLI)

a little more specifically, we need to make sure:

clicking the widget loads instantly. we can't have 100 round trips or slow calls to calculate it. It seems something is building these clusters in the background
we have very minimal time to useful results after logging into the app / starting up / installing... like if I click the thing 10 seconds after I've installed, it's ok if it says "getting recent activity" or something...but that should go away really fast.

When I was saying incomplete results are ok, if we go that route I'd have a big preference for something that fills in backwards in time, starting from now, so recent stuff shows up right away, and the 10th or 20th cluster of the widget could take longer to calculate. This seems reasonable based on the fact we have to look further into history to find the older rows, but it doesn't seem to make match up with working linearly across all TLF's. It would be a weird experience to see a file I edited 8 weeks ago at the top of the list in some random TLF that got calculated first.  Giving people answers they perceive as wrong or buggy would be dangerous. (TLDR: Clusters missing from the bottom of the list, or files missing from the bottom of a cluster are ok; clusters missing from the top or files missing from the top of a cluster feel buggy and should be rare.)
Max makes a good point about CLI access. Similarly, we've talked in the past about apps wanting to know programmatically ( via KBFS or RPC or something) a list of recent changes inside KBFS. I imagine that wouldn't cluster in the same way, but there may be similar considerations. A program might also want to know:

the last files edited in a TLF (who edited them and when)
the last files edited across all TLF's.

clicking the widget loads instantly. we can't have 100 round trips or slow calls to calculate it. It seems something is building these clusters in the background

I was imagining the GUI would start preparing this info on startup, and have it available in advance of someone clicking the widget.

When I was saying incomplete results are ok, if we go that route I'd have a big preference for something that fills in backwards in time, starting from now,

Seems like even more of an argument for having per-TLF fetching.  Maybe we'd also want the list favorites RPC to the API server to supply the last modification time for each favorite from its SQL table ,then the GUI can wait order requests (perhaps in batches) and wait for responses in the right order.

the last files edited in a TLF (who edited them and when)

I have a PR out for KBFS that provides this in a virtual file.  (It's slow though until we have the chat-backed version, so I wouldn't advertise it yet.)

So this is my understanding of the flow

Gui starts up
Gui registers for kbfs notifications (already does this)
Gui hits API server to get list of all tlfs sorted by recent file activity
Gui iterates of this list of tlfs and makes calls to FSSyncListRequest and FSEditListRequest. It now has a snapshot of all the data it needs for the widget
Gui receives updates on the kbfs notifications channel with FSSyncListRequest/FSEditListRequest like subsets which it uses to update its internal model
When new tlfs are added (through a gregor notification?) we ask for new FSSyncListRequest/FSEditListRequest so we're synced up again

This allows us to build the clusters and do whatever filtering we want to build the list and allows us to understand the syncing state so we can display the 'you're uploading' at the bottom of the widget.
As a side effect the GUI will be subscribed to every tlf always
Accurate? @strib  @maxtaco

@chrisnojima: yes, looks right.  I would refine step 4 a bit and say you probably don't want to iterate the TLFs one-by-one, but in small groups to overlap the RTTs as much as possible.  But you can certainly try the easy way first and see whether it's good enough.

You'll definitely get a gregor on a new TLF, that's what marco is working on!

@strib how can i group them, the api takes 1 parameter??

@strib how can i group them, the api takes 1 parameter??

Fire off 5 in parallel and wait for the 5 to come back.  If that's hard for the app, we could change the protocol to take a list and kbfs could do the waiting -- I just don't want to encourage someone putting all the TLFs into one request.

Oh yeah that's simple on our side. Will bad things happen if we don't
throttle
On Wednesday, August 10, 2016, Jeremy Stribling<EMAIL_ADDRESS>wrote:

@strib https://github.com/strib how can i group them, the api takes 1
parameter??
Fire off 5 in parallel and wait for the 5 to come back. If that's hard for
the app, we could change the protocol to take a list and kbfs could do the
waiting -- I just don't want to encourage someone putting all the TLFs into
one request.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/keybase/client/pull/3741#issuecomment-239028173, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAfhU6kQYiNjdVHSZj_gGRTgBzs3xV_dks5qelT8gaJpZM4JcOES
.

Oh yeah that's simple on our side. Will bad things happen if we don't throttle.

By not throttling, you mean asking for the entire list in parallel at once?  That will result in a lot of simultaneous load both in kbfs and potentially to the servers as well, and it could slow down the ones you care about (i.e., the most recently changed ones).  So it seems better to focus first on the ones you know you want to show.

this looks fine to me (after you rebase). One question: what is the requestID, just some number i make up?

this looks fine to me (after you rebase). One question: what is the requestID, just some number i make up?

Yes, it just lets you associate the request and the response, since they are done through disconnected notification mechanisms.  It's opaque to KBFS< we'll just pass it along.

I just realized we also need to notify you when a file is renamed, so I just added a notification type for that (the old name will be in the params map).  I'll merge when CI is green.

@strib looks like this PR made an avdl change (notify_fs_request.avdl) without recompiling the protocol and checking in the generated Go/JS.  Please fix?

Oops!  The protocol was recompiled but I indeed forgot to add the new files.  #3872 fixes it.  Thanks @cjb!


compat: refactor avro compatibility code

Keeping in line with the same logic present in the avro java library
that handles type promotions, this aims to be a faithful adaptation of
that code in python.
See https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/SchemaCompatibility.java

@gabriel-tincu there are many interesting test cases in https://github.com/apache/avro/blob/8c15c1241fb52f3ce6eb480bbe16ff70d3bc426a/lang/java/avro/src/test/java/org/apache/avro/TestSchemaCompatibility.java#L42 Any plans porting them?

@gabriel-tincu there are many interesting test cases in https://github.com/apache/avro/blob/8c15c1241fb52f3ce6eb480bbe16ff70d3bc426a/lang/java/avro/src/test/java/org/apache/avro/TestSchemaCompatibility.java#L42 Any plans porting them?

Actually i do plan on porting as many of the tests as possible, given the minor api differences


Allow http versions >= 0.12.0

packge:flutter hardcodes 0.12.0: https://github.com/flutter/flutter/blob/master/packages/flutter/pubspec.yaml#L39

How urgent is this, @zanderso ?

I don't have anything to do with this package.

It's blocking third_party package rolls into Fuchsia, which blocks adding new third_party packages. It'd be great to get this resolved in the next couple days.

This repository contains generated apis which we publish on pub as well as some utility scripts. The actual constraint must be changed in the https://github.com/dart-lang/discoveryapis_generator repository.


ENH: Custom referencing

Describe the new feature or enhancement
In some cases, especially intracranial EEG, re-referencing of channels should be local (e.g. within the same shaft/grid/strip) rather than global (i.e., ref all channels to one or an average of channels). This thus requires some flexible re-ref schemes, but the existing functions set_eeg_reference and set_bipolar_reference are not super convenient for such purposes.
On the basis of the proposed function, multiple re-ref schemes can be added (for example see https://doi.org/10.1016/j.neuroimage.2018.08.020)
Describe your proposed implementation
A new preprocessing function mne.set_custom_reference(inst, ref_dict, copy=True, on_bad='warn', verbose=None)
Where ref_dict contains a dictionary with keys being the source channel names and values being ref channel names. For example: ref_dict = {'A1': 'A1', 'A2': 'A1', 'A3': 'A1', 'A4': 'A1'} performs monopolar electrode shaft reference (ESR); ref_dict = {'A1': 'A2', 'A2': ['A1', 'A3'], 'A3': ['A2', 'A4'], 'A4': 'A3'} performs Laplacian reference.
I'm happy to open a pull request draft if someone can confirm the value of the proposed function.
Describe possible alternatives
Alternatively we can expand mne.set_eeg_reference instead of writing a new function.
Additional context
No response

This has been asked before:

https://github.com/mne-tools/mne-python/issues/10163
https://github.com/mne-tools/mne-python/issues/8481

If you're willing to make a PR we'd be happy to have support for electrode-shaft referencing.
Can you have a look at those other issues (particularly the ideas about the API) and then tell us what approach best supports your use case as well as the other use cases mentioned there (if they're different)?  Your dict-based approach looks very flexible --- we could maybe even allow mne.set_eeg_reference(ref_channels=dict(...)) --- but I want to be sure we're considering all use cases before we commit to an API.

This has been asked before:

Add laplace reference method to mne/io/reference.py #10163
Re-reference methods about sEEG data #8481

If you're willing to make a PR we'd be happy to have support for electrode-shaft referencing.
Can you have a look at those other issues (particularly the ideas about the API) and then tell us what approach best supports your use case as well as the other use cases mentioned there (if they're different)? Your dict-based approach looks very flexible --- we could maybe even allow mne.set_eeg_reference(ref_channels=dict(...)) --- but I want to be sure we're considering all use cases before we commit to an API.

Thanks for mentioning the 2 relevant issues. My reflection is in iEEG there is no one-size-fits-all scheme. The optimal reference method varies depending on the research question (ERP? oscillations? epileptic activity? decoding?). I also echo with @larsoner in this comment that we cannot add support specifically for every scheme, especially since MNE doesn't support electrode grouping at the moment.
Thus in my opinion we can offer a flexible function that allows for local ref and people can start from there based on their needs, thus the dict approach. What do you think? We can probably add tutorials about how to implement different reference schemes with the API using ECoG or sEEG datasets.
Regarding whether to create a new API or just expand set_eeg_reference(), it has been suggested that we can add a new picks argument to set_eeg_reference(). This would be a nice addition but probably requires several iterations if people want to reref several shafts at the same time. I would personally prefer having a new API.

My colleague @AlexLepauvre is also interested in this topic. Alex, from your perspective (especially having seen different iEEG data from multiple centers) what might be most useful to the community?

I'm a bit in favor of expanding the existing API instead of creating a new one. mne.set_eeg_reference(ref_channels=dict(...)) seems like an elegant solution with ref_channels : str | list[str] | dict[str, str | list[str]].

thinking further about this, I do really like the ref_channels=dict(...) approach: it's flexible and seems (to me at least) easy to understand, and probably not too hard to implement.
However, if we implement this, we probably can't prevent users from applying it to regular scalp EEG data.  So one thing we should also think about though is our mechanisms for keeping track of whether EEG reference has been applied; how will we do that for this referencing approach? what will we do during apply_inverse when it checks for (and doesn't find) the desired situation of "EEG average reference applied"?  (we do have mechanisms for overriding that particular safeguard --- and maybe that's already enough --- but there may be some documentation / user education to be done around the possible downsides of this kind of referencing)

For the dict stuff, I think it should be okay for inverse modeling so long as we set CUSTOM_REF_ON so that modeling can't be done for now. Later when we want to support it we can keep track of the channels used as references properly in info and account for it properly in the inverse. But that can and should be follow-up PRs.
For the relocalization, I don't think we should mess with info['chs']['loc'] at all. I think it can be useful maybe for visualization to think about moving where the activation gets plotted or so. So this would be an option for project_sensors_onto_brain and should probably be a separate PR from the rereferencing-dict one as well.

To sum the discussion above, it seems that most are in favor of mne.set_eeg_reference(ref_channels=dict(...)) with certain things to keep in mind:

Make sure to have inst.info['custom_ref_applied'] = FIFF.FIFFV_MNE_CUSTOM_REF_ON
Safeguard the inclusion of bad channels, either by throwing warnings or preventing the channel from being used

If this sounds good I'm going to create a PR draft.

To sum the discussion above, it seems that most are in favor of mne.set_eeg_reference(ref_channels=dict(...))

yes I think that's right

If this sounds good I'm going to create a PR draft.

go for it!

Great :)
@qian-chu, I will give it a go next week and we can see where we go from my initial commit.


Update to app registration instructions

Docs Build status updates of commit 0d92558:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/exchange-web-services/how-to-authenticate-an-ews-application-by-using-oauth.md
:bulb:Suggestion
View
Details

docs/exchange-web-services/how-to-authenticate-an-ews-application-by-using-oauth.md

Line 29, Column 1: [Suggestion-preserve-view-not-set] Did you mean to link to a specific version? If so, add &preserve-view=true to the URL: /dotnet/api/microsoft.identity.client?view=azure-dotnet

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Docs Build status updates of commit 4851d94:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/exchange-web-services/how-to-authenticate-an-ews-application-by-using-oauth.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel


text widget changes (related to #80)

this is a cleaner PR than #80
The biggest change from that PR other than cleaning up based on comments was that the application.scss already had a facility for data-driven status changes and I got rid of my custom style concept in favour of using the already built-in capability.
So now in the job you can pass:
...
widgetStatus: "status-warning" // or "status-danger"

and the widget will flash red or yellow.
@Zorbash - I notice this application.scss defines warning as red and danger as yellow. I feel like that's not intentional and they should be reversed. Do you want a PR for that?

Coverage remained the same at 87.162% when pulling 057a30d0cd0b0e1a181b5fd3f507295fa2c256a0 on mackenza:text-widget-pr into ad560015d71a904d57ced912e54b333f1e743e95 on kittoframework:master.

if we are good with the status thing, it should probably be added to all/most of the core widgets as it's common functionality defined cross-widget in application.scss

Ebert has finished reviewing this Pull Request and has found:

84 fixed issues! 🎉

You can see more details about this review at https://ebertapp.io/github/kittoframework/kitto/pulls/82.

I don't think ebert is behaving correctly... it always finds 84 issues and then times out after 15 mins.

Having danger as yellow and warning as red is intentional and is actually originating from dashing (see: https://github.com/Shopify/dashing/blob/v1.3.50/templates/project/assets/stylesheets/application.scss#L15).

Having danger as yellow and warning as red is intentional and is actually originating from dashing (see: https://github.com/Shopify/dashing/blob/v1.3.50/templates/project/assets/stylesheets/application.scss#L15)

I see that... doesn't make it right, though ;)
It is pretty standard convention that warning is yellow and error/danger/bad is red. I propose it get changed.

Coverage remained the same at 87.162% when pulling 2b2cd223ee1ab42db171b9c148aa900ba2dc967c on mackenza:text-widget-pr into ad560015d71a904d57ced912e54b333f1e743e95 on kittoframework:master.

how does one update wiki pages? It's not part of the PR is it?

@mackenza I think there's no way to submit a PR for the wiki repo on GitHub. I'll update them appropriately myself after it get's merged.

Concering the colours for warning/danger i concur with you, but discuss any changes in a separate PR.

Coverage remained the same at 87.162% when pulling b36eaac4949666f75c5f7814200414b842210e73 on mackenza:text-widget-pr into ad560015d71a904d57ced912e54b333f1e743e95 on kittoframework:master.

that's what I get for coding while my ass froze watching my son play hockey... it actually wasn't even calling the function properly let alone generating the proper output. ;)
fixed.

Squashed and cherry-picked as 9bb44ba6661a8126a71fd5cb4e8fff8156623b85
Thanks a lot @mackenza :+1:

Ebert has finished reviewing this Pull Request and has found:

2 possible new issues (including those that may have been commented here).

But beware that this branch is 2 commits behind the kittoframework:master branch, and a review of an up to date branch would produce more accurate results.
You can see more details about this review at https://ebertapp.io/github/kittoframework/kitto/pulls/82.


Using existing connection hangs on query

I have defined a typegoose model in another package and am trying to use it as follows.
(some pieces omitted for brevity)
Snippet in TypeScript:
import { Position } from '@studio/core';
import mongoose = require('mongoose');

async function subscribe(connection: mongoose.Connection) {
  const model = new Position({}).getModelForClass(Position, {
    existingConnection: connection,
  });

  try {
    console.log('fetching...');
    const data = await model.find({}).exec();
    console.log(data);
  } catch (e) {
    console.error(e);
  }
}

mongoose
  .connect(MONGO_URL)
  .catch(reason => {
    console.error(reason);
    process.exit(1);
  })
  .then(() => {
    subscribe(mongoose.connection);
  });

the promise will hang never complete. Is there something wrong here or a bug with using an existing connection?
If i instead just use the typegoose model to get the schema and create another model, it works, but thats completely unnecessary
  const schema = new Position({}).getModelForClass(Position, {
    existingConnection: connection,
  }).schema;
  const model = mongoose.model('Positions', schema, 'positions');

@ssilve1989 what was your solution?

@darbio please know that his repository is discontinued, if you have issues try upgrading to<EMAIL_ADDRESS>


fix: the mac build

Macs are (mostly) case-insensitive. A change to LLVM resulted in
a file called CXXABI.h being visible in the system search path
ahead of the actual cxxabi.h. This caused certain Abseil libraries
to fail to build as they depend on the content of that header.
This PR pushes Abseil to head (why not?), LLVM to head (as they
fixed this problem upstream), and rules_cc to head (because the
LLVM fix requires a more recent version).
Addresses #5133

This is the fix to the non-Abseil issue https://github.com/abseil/abseil-cpp/issues/1067 (mentioning in case someone gets there via search)


Have a button where the player is 100% of the viewport

For purposes of running in a window.. I think would be nice if there was a button that instead of full screened just expanded the player to the full height and width of the window or say under a tiling window manager of some sort.

Have you tried the Kiosk option? When I turn that on in a regular window the weather displays fill up the available space in the window, letterboxing to keep the aspect ratio. I can freely resize the window and the display continues to fill the available area.

So what you are saying is to fully resolve the issue I should read the readme and plan to check said readme before making feature requests? LOL Sorry for the redundant noise ;)

It's not all noise. Someone else who goes searching for the same window, viewport or 100% idea can now discover this solution and find that it's called kiosk.


Move PublicNetworkAccess from the SiteConfig to the Site object

Description
This PR fixes a bug that causes the provisioning of the Azure Function App that is used to forward the logs from a Storage Account to New Relic to fail.
Change details
Moved the PublicNetworkAccess property from SiteConfig.PublicNetworkAccess to Site.PublicNetworkAccess.
Error details
To reproduce the error, follow the instructions for Deploying the New Relic Blob Storage ARM template.
The ARM template fails to deploy the Function App with the error:
There was a conflict. SiteConfig.PublicNetworkAccess cannot be set by itself. Please use the Site.PublicNetworkAccess property.
(Target: [azure_resource_identifier])

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


"before:content-['']" doesn't work in class property uno/windi preset

I'm trying to write ul>li menu with dots.
If I used before:content-[''] like in code below, then nothing happens, there is no class in browser.
<li
    v-for="(item, i) in navList"
    :key="`menu-item-${i}`"
    class="text-base text-white flex items-center before:(block rounded-full w-1 h-1 mx-7 bg-white/15 content-[''])"
>
    <NuxtLink :to="item.link">{{ item.title }}</NuxtLink>

But if I'm trying to @apply class, all works fine

Why?
Deps:

Nuxt 3: rc.12
@unocss/nuxt: 0.46.1

The default splitter will split on ' so content-[''] usually is not available. You may use content-none instead.

can no work?

try content-[quoted::].

 can no work?

try content-[quoted::]

 can no work?

try content-[quoted::], you can get more infos in #1741.

Yes , It work, Thank You!


[SPARK-7397][Streaming] Add missing inputInfo report code

I think this issue is introduced in #5732

Merged build triggered.

Merged build started.

Test build #31966 has started for   PR 5937 at commit 4206ee0.

Test build #31966 has finished for   PR 5937 at commit 4206ee0.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31966/
Test FAILed.

Jenkins, retest this please.

Merged build triggered.

Merged build started.

Test build #31974 has started for   PR 5937 at commit 4206ee0.

Test build #31974 has finished for   PR 5937 at commit 4206ee0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31974/
Test PASSed.

I am choosing #5950 in favor of this one as that one has a test to prevent this from happening in the future.Thanks though for catching this! Mind closing this for now.

Hi @tdas, please take a look at this PR, I think this is a problem introduced by #5732 , and should be fixed. Thanks a lot :)

OK, cool.


Idea: add function for QC only

Could you consider the adding disable option at once?
--disable_trim_poly_g --disable_adapter_trimming --disable_quality_filtering –disable_length_filtering
No need trim/filtering function for QC.

If QC only, you can just do not set the output files.


HDDS-9362. Intermittent failure in acceptance test due to DISK_OUT_OF_SPACE

What changes were proposed in this pull request?
Acceptance test started failing recently due to DISK_OUT_OF_SPACE error from datanodes.  It turns out 5GB is still left, so container (1GB) could have been created, but datanode wanted to preserve free space.  This change reduces the minimum free remaining space which is required when creating containers to 100MB from default 5GB.
Also, remove Ozone binary tarball (~350MB) after extraction to really free up some space.
https://issues.apache.org/jira/browse/HDDS-9362
How was this patch tested?
CI:
https://github.com/adoroszlai/hadoop-ozone/actions/runs/6344084423

@swamirishi please help to review it?

Thanks @hemantk-12, @swamirishi for the review.


chore(deps): upgrade configuration

Upgrades project dependencies. See details in workflow run.

Automatically created by projen via the "upgrade-configuration-k8s-25-main" workflow

⚪ Backport skipped
The pull request was not backported as there were no branches to backport to. If this is a mistake, please apply the desired version labels or run the backport tool manually.
Manual backport
To create the backport manually run:
backport --pr 3758

Questions ?
Please refer to the Backport tool documentation


Add Storage.create() method which takes a restartable input stream

Filing feature request that was raised in:
https://github.com/googleapis/google-cloud-java/issues/2620
So it doesn't get lost.

Will it be enough to implement upload based on WriterChannel?

Hi @frankyn,
Is this functionality really needed?
Justification for that request (https://github.com/googleapis/google-cloud-java/issues/2620) sounds like:  so that it is easier for users to create blobs without loading the entire content into memory.
But Storage.create(IntputStream) doesn't load the entire content into memory. I successfully uploaded 1.5Gb on the JVM with the heap of 50M (-Xmx50m).  Upload is implemented in:
com.google.api.client.http.AbstractInputStreamContent {
    public void writeTo(OutputStream out) throws IOException {
        IOUtils.copy(this.getInputStream(), out, this.closeInputStream);
        out.flush();
    }
}

IOUtils.copy() uses buffer of 4096 bytes to read from and write to the given streams, so no problem with memory.
As you probably remember, #179 will introduce a number of upload() methods which read the content portion by portion and applies resumable approach.
Even if upload with a restartable input stream is still needed for some reason, it could be implemented without introducing a new class representing a restartable input stream (as it implemented in https://github.com/googleapis/google-cloud-java/pull/2682), no need to expose internal details to public. Upload details can be reflected in the method name:
  public Blob createBlobUsingRestorableInputStream(blobInfo, inputStream, options) {
     RestorableInputStream  restorable = new RestorableInputStream (inputStream);
   ...
  }

I personally do not see any benefits from adding such method. Do you?