Stuck in 'connect' state with Cluster and Node 10.x

I'm seeing ioredis 3.2.2 intermittently getting indefinitely stuck in the 'connect' state when run with Redis.Cluster under all versions of Node 10 up to the latest (10.3.0), and not under previous major versions (e.g. 9.11.1). I'm working on a minimal repro to illustrate this, but right now it looks as though the Node 10 stream is not emitting a 'close' event to trigger eventHandler.closeHandler(_this)  under Node 10.
I'm not yet sure why this happens, but I thought I'd give a heads-up as I look at it in case anyone else is seeing the same.

It's slightly harder to get it to stall out with a single connection (dialing down nRedis to 1), but after a few runs of that on Node 10.3.0 and IORedis debug enabled, I get:
Thu, 31 May 2018 20:46:52 GMT ioredis:cluster status: [empty] -> connecting
Thu, 31 May 2018 20:46:52 GMT ioredis:redis status[redis1:6379]: [empty] -> wait
Thu, 31 May 2018 20:46:52 GMT ioredis:redis status[redis2:6379]: [empty] -> wait
Thu, 31 May 2018 20:46:52 GMT ioredis:redis status[redis3:6379]: [empty] -> wait
Thu, 31 May 2018 20:46:52 GMT ioredis:redis status[redis4:6379]: [empty] -> wait
Thu, 31 May 2018 20:46:52 GMT ioredis:redis status[redis5:6379]: [empty] -> wait
Thu, 31 May 2018 20:46:52 GMT ioredis:redis status[redis6:6379]: [empty] -> wait
Thu, 31 May 2018 20:46:52 GMT ioredis:cluster getting slot cache from redis5:6379
Thu, 31 May 2018 20:46:52 GMT ioredis:redis status[redis5:6379]: wait -> connecting
Thu, 31 May 2018 20:46:52 GMT ioredis:redis queue command[0] -> cluster(slots)
Thu, 31 May 2018 20:46:52 GMT ioredis:redis status[redis2:6379]: wait -> connecting
Start burning CPU
Stop burning CPU, burned 3693ms
Thu, 31 May 2018 20:46:56 GMT ioredis:cluster getting slot cache from redis2:6379
Thu, 31 May 2018 20:46:56 GMT ioredis:redis queue command[0] -> cluster(slots)
Thu, 31 May 2018 20:46:56 GMT ioredis:redis status[xx.xxx.x.xx:6379]: connecting -> connect
Thu, 31 May 2018 20:46:56 GMT ioredis:redis write command[0] -> info()
Thu, 31 May 2018 20:46:56 GMT ioredis:redis status[xx.xxx.x.xx:6379]: connecting -> connect
Thu, 31 May 2018 20:46:56 GMT ioredis:redis queue command[0] -> info()
Thu, 31 May 2018 20:46:56 GMT ioredis:redis status[xx.xxx.x.xx:6379]: connect -> ready
Thu, 31 May 2018 20:46:56 GMT ioredis:connection send 1 commands in offline queue
Thu, 31 May 2018 20:46:56 GMT ioredis:redis write command[0] -> cluster(slots)
Thu, 31 May 2018 20:46:56 GMT ioredis:redis status[redis1:6379]: wait -> close
Thu, 31 May 2018 20:46:56 GMT ioredis:connection skip reconnecting since the connection is manually closed.
Thu, 31 May 2018 20:46:56 GMT ioredis:redis status[redis1:6379]: close -> end
Thu, 31 May 2018 20:46:56 GMT ioredis:redis status[redis3:6379]: wait -> close
Thu, 31 May 2018 20:46:56 GMT ioredis:connection skip reconnecting since the connection is manually closed.
Thu, 31 May 2018 20:46:56 GMT ioredis:redis status[redis3:6379]: close -> end
Thu, 31 May 2018 20:46:56 GMT ioredis:redis status[redis4:6379]: wait -> close
Thu, 31 May 2018 20:46:56 GMT ioredis:connection skip reconnecting since the connection is manually closed.
Thu, 31 May 2018 20:46:56 GMT ioredis:redis status[redis4:6379]: close -> end
Thu, 31 May 2018 20:46:56 GMT ioredis:redis status[redis6:6379]: wait -> close
Thu, 31 May 2018 20:46:56 GMT ioredis:connection skip reconnecting since the connection is manually closed.
Thu, 31 May 2018 20:46:56 GMT ioredis:redis status[redis6:6379]: close -> end
Thu, 31 May 2018 20:46:56 GMT ioredis:redis status[xx.xxx.x.xx:6379]: [empty] -> wait
Thu, 31 May 2018 20:46:56 GMT ioredis:redis status[xx.xxx.x.xx:6379]: [empty] -> wait
Thu, 31 May 2018 20:46:56 GMT ioredis:redis status[xx.xxx.x.xx:6379]: [empty] -> wait
Thu, 31 May 2018 20:46:56 GMT ioredis:redis status[xx.xxx.x.xx:6379]: [empty] -> wait
Thu, 31 May 2018 20:46:56 GMT ioredis:redis status[xx.xxx.x.xx:6379]: [empty] -> wait
Thu, 31 May 2018 20:46:56 GMT ioredis:redis status[xx.xxx.x.xx:6379]: [empty] -> wait
Thu, 31 May 2018 20:46:56 GMT ioredis:cluster status: connecting -> connect
Thu, 31 May 2018 20:46:56 GMT ioredis:redis queue command[0] -> cluster(info)
Thu, 31 May 2018 20:46:56 GMT ioredis:redis status[xx.xxx.x.xx:6379]: ready -> close
Thu, 31 May 2018 20:46:56 GMT ioredis:connection skip reconnecting since the connection is manually closed.
Thu, 31 May 2018 20:46:56 GMT ioredis:redis status[xx.xxx.x.xx:6379]: close -> end
Thu, 31 May 2018 20:46:56 GMT ioredis:redis status[xx.xxx.x.xx:6379]: wait -> connecting
Thu, 31 May 2018 20:46:56 GMT ioredis:redis status[xx.xxx.x.xx:6379]: connecting -> connect
Thu, 31 May 2018 20:46:56 GMT ioredis:redis write command[0] -> info()
Thu, 31 May 2018 20:46:56 GMT ioredis:redis status[xx.xxx.x.xx:6379]: connect -> ready
Set redis key on cluster 0
### STUCK: CLUSTER (0) STALLED OUT ###

On a healthy run (this is with Node 10.3.0 as well, but just on a run where we don't see the intermittent failure), we see the cluster get into the ready state instead of stalling out in 'connect':
Thu, 31 May 2018 20:56:09 GMT ioredis:cluster status: [empty] -> connecting
Thu, 31 May 2018 20:56:09 GMT ioredis:redis status[redis1:6379]: [empty] -> wait
Thu, 31 May 2018 20:56:09 GMT ioredis:redis status[redis2:6379]: [empty] -> wait
Thu, 31 May 2018 20:56:09 GMT ioredis:redis status[redis3:6379]: [empty] -> wait
Thu, 31 May 2018 20:56:09 GMT ioredis:redis status[redis4:6379]: [empty] -> wait
Thu, 31 May 2018 20:56:09 GMT ioredis:redis status[redis5:6379]: [empty] -> wait
Thu, 31 May 2018 20:56:09 GMT ioredis:redis status[redis6:6379]: [empty] -> wait
Thu, 31 May 2018 20:56:09 GMT ioredis:cluster getting slot cache from redis6:6379
Thu, 31 May 2018 20:56:09 GMT ioredis:redis status[redis6:6379]: wait -> connecting
Thu, 31 May 2018 20:56:09 GMT ioredis:redis queue command[0] -> cluster(slots)
Thu, 31 May 2018 20:56:09 GMT ioredis:redis status[redis2:6379]: wait -> connecting
Start burning CPU
Stop burning CPU, burned 3359ms
Thu, 31 May 2018 20:56:12 GMT ioredis:cluster getting slot cache from redis1:6379
Thu, 31 May 2018 20:56:12 GMT ioredis:redis status[redis1:6379]: wait -> connecting
Thu, 31 May 2018 20:56:12 GMT ioredis:redis queue command[0] -> cluster(slots)
Thu, 31 May 2018 20:56:12 GMT ioredis:redis status[xx.xxx.x.xx:6379]: connecting -> connect
Thu, 31 May 2018 20:56:12 GMT ioredis:redis queue command[0] -> info()
Thu, 31 May 2018 20:56:12 GMT ioredis:redis status[xx.xxx.x.xx:6379]: connecting -> connect
Thu, 31 May 2018 20:56:12 GMT ioredis:redis write command[0] -> info()
Thu, 31 May 2018 20:56:12 GMT ioredis:redis status[xx.xxx.x.xx:6379]: connecting -> connect
Thu, 31 May 2018 20:56:12 GMT ioredis:redis write command[0] -> info()
Thu, 31 May 2018 20:56:12 GMT ioredis:redis status[xx.xxx.x.xx:6379]: connect -> ready
Thu, 31 May 2018 20:56:12 GMT ioredis:redis status[xx.xxx.x.xx:6379]: connect -> ready
Thu, 31 May 2018 20:56:12 GMT ioredis:connection send 1 commands in offline queue
Thu, 31 May 2018 20:56:12 GMT ioredis:redis write command[0] -> cluster(slots)
Thu, 31 May 2018 20:56:12 GMT ioredis:redis status[redis3:6379]: wait -> close
Thu, 31 May 2018 20:56:12 GMT ioredis:connection skip reconnecting since the connection is manually closed.
Thu, 31 May 2018 20:56:12 GMT ioredis:redis status[redis3:6379]: close -> end
Thu, 31 May 2018 20:56:12 GMT ioredis:redis status[redis4:6379]: wait -> close
Thu, 31 May 2018 20:56:12 GMT ioredis:connection skip reconnecting since the connection is manually closed.
Thu, 31 May 2018 20:56:12 GMT ioredis:redis status[redis4:6379]: close -> end
Thu, 31 May 2018 20:56:12 GMT ioredis:redis status[redis5:6379]: wait -> close
Thu, 31 May 2018 20:56:12 GMT ioredis:connection skip reconnecting since the connection is manually closed.
Thu, 31 May 2018 20:56:12 GMT ioredis:redis status[redis5:6379]: close -> end
Thu, 31 May 2018 20:56:12 GMT ioredis:redis status[xx.xxx.x.xx:6379]: [empty] -> wait
Thu, 31 May 2018 20:56:12 GMT ioredis:redis status[xx.xxx.x.xx:6379]: [empty] -> wait
Thu, 31 May 2018 20:56:12 GMT ioredis:redis status[xx.xxx.x.xx:6379]: [empty] -> wait
Thu, 31 May 2018 20:56:12 GMT ioredis:redis status[xx.xxx.x.xx:6379]: [empty] -> wait
Thu, 31 May 2018 20:56:12 GMT ioredis:redis status[xx.xxx.x.xx:6379]: [empty] -> wait
Thu, 31 May 2018 20:56:12 GMT ioredis:redis status[xx.xxx.x.xx:6379]: [empty] -> wait
Thu, 31 May 2018 20:56:12 GMT ioredis:cluster status: connecting -> connect
Thu, 31 May 2018 20:56:12 GMT ioredis:redis status[xx.xxx.x.xx:6379]: ready -> close
Thu, 31 May 2018 20:56:12 GMT ioredis:connection skip reconnecting since the connection is manually closed.
Thu, 31 May 2018 20:56:12 GMT ioredis:redis status[xx.xxx.x.xx:6379]: close -> end
Thu, 31 May 2018 20:56:12 GMT ioredis:redis status[xx.xxx.x.xx:6379]: ready -> close
Thu, 31 May 2018 20:56:12 GMT ioredis:connection skip reconnecting since the connection is manually closed.
Thu, 31 May 2018 20:56:12 GMT ioredis:redis status[xx.xxx.x.xx:6379]: close -> end
Thu, 31 May 2018 20:56:12 GMT ioredis:redis status[xx.xxx.x.xx:6379]: wait -> connecting
Thu, 31 May 2018 20:56:12 GMT ioredis:redis status[xx.xxx.x.xx:6379]: connecting -> connect
Thu, 31 May 2018 20:56:12 GMT ioredis:redis write command[0] -> info()
Thu, 31 May 2018 20:56:12 GMT ioredis:redis status[xx.xxx.x.xx:6379]: connect -> ready
Thu, 31 May 2018 20:56:12 GMT ioredis:cluster getting slot cache from xx.xxx.x.xx:6379
Thu, 31 May 2018 20:56:12 GMT ioredis:redis write command[0] -> cluster(slots)
Thu, 31 May 2018 20:56:12 GMT ioredis:delayqueue send 1 commands in failover queue
Thu, 31 May 2018 20:56:12 GMT ioredis:redis status[xx.xxx.x.xx:6379]: wait -> connecting
Thu, 31 May 2018 20:56:12 GMT ioredis:redis queue command[0] -> cluster(info)
Thu, 31 May 2018 20:56:12 GMT ioredis:redis status[xx.xxx.x.xx:6379]: connecting -> connect
Thu, 31 May 2018 20:56:12 GMT ioredis:redis write command[0] -> info()
Thu, 31 May 2018 20:56:12 GMT ioredis:redis status[xx.xxx.x.xx:6379]: connect -> ready
Thu, 31 May 2018 20:56:12 GMT ioredis:connection send 1 commands in offline queue
Thu, 31 May 2018 20:56:12 GMT ioredis:redis write command[0] -> cluster(info)
Thu, 31 May 2018 20:56:12 GMT ioredis:cluster status: connect -> ready
Set redis key on cluster 0
Thu, 31 May 2018 20:56:13 GMT ioredis:redis write command[0] -> set(test.set.key,1527800173790)
Set redis key on cluster 0
Thu, 31 May 2018 20:56:14 GMT ioredis:redis write command[0] -> set(test.set.key,1527800174791)
Set redis key on cluster 0
Thu, 31 May 2018 20:56:15 GMT ioredis:redis write command[0] -> set(test.set.key,1527800175792)
<<< SUCCESS, WE DID NOT STALL OUT >>>

This is the most minimal simulation I have been able to generate for what we're seeing for the good and bad cases in our actual application in staging on Node 10 and Cluster 3.2.2. This may boil down to a Node bug or some kind of improper usage on our end, but it seems like understanding what's causing the cluster to get stuck in 'connect' as the cluster(info) command goes through and the redis goes to 'ready' is the beginning, and that's where I'm having some trouble.

Unsurprisingly (this was the most suspicious-looking change in the (changelog)[https://github.com/nodejs/node/blob/master/doc/changelogs/CHANGELOG_V10.md] for this), this traces back to https://github.com/nodejs/node/commit/9b7a6914a7. If you build Node at the previous revision (https://github.com/nodejs/node/commit/74553465e6), this test does not repro, at 9b7a6914a7f0bd754e78b42b48c75851cfd6b3c4 it does.
It looks like there is some hubbub around 'close' events in the subsequent Node 10 changelogs, but concerning http, e.g. https://github.com/nodejs/node/pull/20941.

So you're suspecting that this is really a node issue and not an ioredis issue?  Would probably to be good to log an issue there as well.  Try to get some of those folks to re-review that commit.  This issue is going to prevent folks using ioredis to upgrade to Node 10.

Yes, I'm suspecting that Node isn't emitting 'close' under certain circumstances, and that that is a bug with that commit, but it'll probably take me a little while to boil it down to something that takes ioredis out of it.
Currently we can work around this for the purposes of this test by handling 'end' rather than close
diff --git a/lib/redis.js b/lib/redis.js
index 8870bec..39b9b53 100644
--- a/lib/redis.js
+++ b/lib/redis.js
@@ -279,7 +279,7 @@ Redis.prototype.connect = function (callback) {
   stream.once(CONNECT_EVENT, eventHandler.connectHandler(_this));
   stream.once('error', eventHandler.errorHandler(_this));

 stream.once('close', eventHandler.closeHandler(_this));

 stream.once('end', eventHandler.closeHandler(_this));
 stream.on('data', eventHandler.dataHandler(_this));

... but the 'close' handling seems more correct to me, and it looks as though that event should always be emitted.

Ah, ok, if the socket is paused there may not be a 'close':
https://github.com/nodejs/node/pull/19241#issuecomment-371639579

... but at the point when we get the EOF (https://github.com/nodejs/node/commit/9b7a6914a7#diff-e6ef024c3775d787c38487a6309e491dL645), the connection isPaused() returns false, so it's not that.
The issue is that we get into onReadableStreamEnd with the socket's this.writable set to true, so the socket's maybeDestroy doesn't call socket.destroy and the socket never emits closed.
It's a little hard to tell if that's intentional or not, from the PR and checkin, but it's looking more like 'not'.

There is more on what exactly it looks like is happening with the socket at https://github.com/nodejs/node/pull/19241#issuecomment-396133420 -- the long and short of it appears to be that if we hit the connect timeout and call .end before the socket is connected, the socket gets in a state that causes it to never call .destroy and thus never emit close. It might be reasonable to work around this issue in the connection timeout logic or by handling end events for the time being, as we work this out with an issue against Node.js (if they agree that the behavior is incorrect).

This will be fixed when this ships on Node 10: https://github.com/nodejs/node/pull/21290. I'll leave this open until then in case anyone else sees it.

Great investigation work!

@ccs018 Thanks! The fix is out with Node 10.5.0, so I'll close this.


Duplicate Application Command

The two commands marked as global: true commands will be loaded twice. Likely one being loaded as a guild specific command. While the other being loaded as a global command. Difference being; global commands can be accessed in any guild (server) or via direct messages to the bot, while guild commands are only accessed within the specific guild they're registered in. Majority of the bots commands are guild dependant and are registered as guild commands, excluding commands/ping.js and /commands/stats.js.
Seen below is the duplication of the command.

The RegisterSlashCommands() function is called from the events/ready.js file that is run when the bot is ready.
  // Calls register for guild and global commands
  RegisterSlashCommands() {
    RegisterGlobalCommands(this);
    this.guilds.cache.forEach((guild) => RegisterGuildCommands(this, guild.id));
  }

So RegisterGlobalCommands() should only be called once.
While RegisterGuildCommands() is called once per guild.
inside the RegisterGuildCommands() we find this section of code,
  for (const file of commandFiles) {
    const command = require(`../commands/${file}`);
    if (!commands.global) commands.push(command); // don't include global commands
  }

As seen above, global commands should be excluded from the array of commands to register to a guild. Yet, somewhere it is still being registered.

Fixed in ^v9.7.1


Write shortest possible code: How to join lists (python)

Please write shortest code example for this question:
How to join lists in python
How to do it:

Go to python codes
Create new file (named in underscore case, should contain key words from title) with md extension (markdown file).
Propose new file with following content (please use all three blocks if possible - title, code itself and explanations list):

# How to join lists

```python
code part1 part2 part3 ...
```

- part1 - explain code part 1
- part2 - explain code part 2
- ...

More advanced template for examples and linked solutions. More docs here.

@nonunicorn  Please assign this to me


x/tools/cmd/goimports: does not work if there are spaces in the path

What version of Go are you using (go version)?

$ go version
go version go1.14.6 darwin/amd64

Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/xor/Library/Caches/go-build"
GOENV="/Users/xor/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/xor/go"
GOPRIVATE=""
GOPROXY="direct"
GOROOT="/Users/xor/sdk/go1.14.6"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/Users/xor/sdk/go1.14.6/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/xor/Documents/Source Code/golden/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/yh/x8t18v653t752p5nk400qt580000gn/T/go-build156301565=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?
I running command in /Users/xor/Documents/Source Code/golden:
test -z $(go list -f {{.Dir}} ./... | while read d; do goimports -e -l -local github.com/xorcare/golden -w "$d/*.go"; done)

goimports version: golang.org/x/tools v0.0.0-20200729173947-1c30660f9f89
What did you expect to see?
I want to see formatted imports...
What did you see instead?
test -z $(go list -f {{.Dir}} ./... | while read d; do goimports -e -l -local github.com/xorcare/golden -w "$d/*.go"; done)
stat /Users/xor/Documents/Source Code/golden/*.go: no such file or directory
stat /Users/xor/Documents/Source Code/golden/internal/integration/*.go: no such file or directory
stat /Users/xor/Documents/Source Code/golden/v2/*.go: no such file or directory
stat /Users/xor/Documents/Source Code/golden/v2/abstract/*.go: no such file or directory
stat /Users/xor/Documents/Source Code/golden/v2/goldenjson/*.go: no such file or directory
stat /Users/xor/Documents/Source Code/golden/v2/internal/cast/*.go: no such file or directory
stat /Users/xor/Documents/Source Code/golden/v2/internal/extension/*.go: no such file or directory
stat /Users/xor/Documents/Source Code/golden/v2/internal/goldenbuilder/*.go: no such file or directory
stat /Users/xor/Documents/Source Code/golden/v2/internal/testing/*.go: no such file or directory

You are literally passing "directory/*.go" to goimports, which doesn't work. I don't know how to do what you're trying to do offhand, but this is not a bug in goimports.

Oh, probably just change "$d/*.go" to "$d"/*.go.

Oh, probably just change "$d/*.go" to "$d"/*.go.

It worked, thanks! 😄


Added an option to only select required fields

added an option to only add selected fields to the csv files by specifying fields after the path arguments
python json_to_csv.py <node> <json_in_file_path> <csv_out_file_path> <field1> <field2>

resolves #35

Hi @varnitsingh,
Thanks for submitting this PR! This not only resolves #35 but is definitely going to be a useful feature. However, I think this needs a bit more discussion around the implementation...
Some initial thoughts/questions I have are...

The field that is being indicated by the user can be at any level in the json. How to handle this? Does the user get options to remove from everywhere or only at say 1st level?
Should the fields be excluded from the output csv or input json?
What happens to complex relationships? As in, if let's say parent2 is being requested in a case like parent1.parent2.parent3.child1 = value by the user then how to handle?
What happens to dictionary inside list?
The script in it's current from can handle non-homogeneous objects. Will implementation vary for these?

Let me know what you think.

Answering your questions

The acceptable fields are only at level one. The arguments specify which fields to keep in the final output csv file.
Fields not mentioned in the arguments are excluded from the final output csv file.
Right now only level one fields are being worked upon, I can write something to handle complex relationships. For parent2 argument could be parent1.parent2
4-5. Can you expand on that, I don't know which list or dictionary you are talking about. Also can you give a example of homogeneous object.
Also I could also add a delete feature. So with arguments starting with -field1 field2 these fields would be removed from the final output csv file.


Memory leak with several ANSI sequences

Environment
Windows build number: 10.0.19042.508
Windows Terminal version (if applicable): 1.5.3142.0

Any other software?
Node v15.2.0

Steps to reproduce

Save the following content in a file, say, name it with test.js

var readline = require('readline'),count = 0;
setInterval(() =>
{
	console.log(++count,new Date)
	readline.moveCursor(process.stdout,0,-1)
},10)

Open the cmd.exe in the terminal
Run the script saved above with path\to\node.exe path\to\test.js

Expected behavior

No significant memory increasing should be observed
Actual behavior

To compare with, another cmd.exe is also directly started by Win+R and run the same script.
Here is the initial memory usage

Also, reported by Process Explorer, the WS Private and Working Set are
WindowsTerminal.exe	36976 K	101816 K

Then, leave them run for hours, and here is the memory usage after the counter hit 400k+.

Next, Ctrl+C to end both node processes, and wait for 5min.

WindowsTerminal.exe	96444 K	157840 K

It seems that it memory leaked.
And by roughly counting during the progress, the memory usage grow about 0.1 MB / 10 sec.

Hmm, you're right. Interesting.

Good catch!


feat: ‖a * b * c‖ ≤ ‖a‖ * ‖b‖ * ‖c‖

and prime the existing lemmas for a norm on a multiplicative group to make space for the ring ones.
From GrowthInGroups

Looks sensible enough to me
maintainer merge

Thanks!
bors merge


made datadog on windows dependent on ci flag

Proposed changes:

made windows datadog profiling dependent on explicit flag. Installation of the docker container on windows takes 2-6 minutes and adds significantly to run time.

Status (please check what you already did):

[ ] added some tests for the functionality
[ ] updated the documentation
[ ] updated the changelog (please check changelog for instructions)
[ ] reformat files using black (please check Readme for instructions)

test

test


fix types resolution for expect-puppeteer

Summary

This pull request contains :

A refactoring of the expect-puppeteer types so that types resolution for the puppeteer matchers work in Typescript (issue 598).
Additional type assertions on the Error objects in the tests to eliminate any types and comply to @typescript-eslint/recommended.
A few upgrades of deprecated methods on the puppeteer API (see resetBrowser.test.ts).
An update on the main README to reflect the changes.

Test plan

No breaking changes or regressions should occur since the implementation was left untouched, only types declarations were changed.
Test suite passes 100%.

When testing the minimal repo with the following dependencies :

Typescript complains, types resolution does not work

Then changing dependencies to point to my fork, adding @types/jest and npm i :

Typescript is ok and types resolution works again 👍

Great, thanks for that!


use fastify as router instead native http

After some testing in the becnhmark repo look like using fatify as http server instead native http could improve perfromance
Experingment branch here: https://github.com/MionKit/Benchmarks/tree/experiment-use-fastify-instead-http-to-tets-if-metrics-improve
It was concluded that using fastify with a simple router i.e: fastify.get,  fastify.post was faster than using @mionkit/http, the problem comes when serving any router or multiple routes  together as mion/router does as it completely ignores method

build a fastify plugin to server http and ignore the method.

An experiment has been created in the benchmarks repo and look slike using fastify as a wrapper for the router has worst results than using native http

branch: https://github.com/MionKit/Benchmarks/tree/experiment-use-fastify-instead-http-to-tets-if-metrics-improve
fastify wrapper: https://github.com/MionKit/Benchmarks/blob/experiment-use-fastify-instead-http-to-tets-if-metrics-improve/apps/src/fastMion.ts

So we not going ahead with this idea!


ImportError: cannot import name 'XMLSchemaProxy' from 'elementpath'

In Python 3.7.8:
python xml_to_json.py ./foo.xml

returns ImportError: cannot import name 'XMLSchemaProxy' from 'elementpath'.

I got this error as well.
Try a workaround by installing elementpath=1.0.12. Seems like there are some breaking changes in elementpath between version 1 and 2.

Sounds like you have circular dependent imports. In other words, xml_to_json.py is imported from the ParqConverter class before the class the other classes that need to be defined are actually defined. Secondarily, xml_to_json tries to import ParqConverter, which is already initializing. Remove the dependency to xml_to_json from the module completely.


Suggested method to encrypt integration key for module?

Happen to have a suggestion on how to encrypt the integration key?

I would recommend storing your keys safely in something like Azure Key Vault


Merged in updated astropy-helpers

So astropy_helpers had gotten out of sync somehow. I'm pretty sure this was my own doing, though even in hindsight I'm not sure how to have recognized how it was happening. It is a big of a nuisance that we have an entirely separate git history located in a sub-package, because it means we need to remember to manually check the astropy_helpers git history before committing to master.
Bottom line: with these changes, all of the changes made by @duncandc seem to have paid off: from within my bare bones virtualenv, the code passes all tests and raises no errors! Woohoo!

Well, durn, Travis is more finicky than my virtualenv.


Add referential equality to pitfalls

Add a section about referential equality to pitfalls, with the simple
example of using indexOf on a draft array to match an element.
Fixes #730

Thanks!

Thanks!

:tada: This PR is included in version 8.0.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This PR is included in version 8.0.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Memory issue(?) with remote_write on ARMv7 with

(Somewhat of a follow-up on #2663)
What did you do?
I'm trying to send logs remotely using remote_write on ARMv7.
What did you expect to see?
I expected Prometheus to send a payload to the URL specified in remote_write.
What did you see instead? Under which circumstances?
All. It works fine if I remove the remote_write stanza.
Environment
A Raspberry Pi 3 Model B running ResinOS.
In case the memory allocation is a concern, here is a dump of that:
# free -m
             total       used       free     shared    buffers     cached
Mem:           972        541        431         49         60        264
-/+ buffers/cache:        215        756
Swap:            0          0          0

System information:

Linux 4.4.48 armv7l

Prometheus version:

prometheus, version 1.6.1 (branch: master, revision: 4666df502c0e239ed4aa1d80abbbfb54f61b23c3)
  build user:       root@ca1b452514b7
  build date:       20170419-13:53:08
  go version:       go1.8.1

Prometheus configuration file:

# cat /etc/prometheus.yml
# my global config
global:
  scrape_interval:     30s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 30s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      monitor: 'tmp-resin'

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first.rules"
  # - "second.rules"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'tmp'
    static_configs:
      - targets: ['localhost:8000']

remote_write:
    - url: https://requestb.in/abc123

Logs:

# prometheus -config.file=/etc/prometheus.yml
INFO[0000] Starting prometheus (version=1.6.1, branch=master, revision=4666df502c0e239ed4aa1d80abbbfb54f61b23c3)  source=main.go:88
INFO[0000] Build context (go=go1.8.1, user=root@ca1b452514b7, date=20170419-13:53:08)  source=main.go:89
INFO[0000] Loading configuration file /etc/prometheus.yml  source=main.go:251
INFO[0000] Loading series map and head chunks...         source=storage.go:421
INFO[0000] 0 series loaded.                              source=storage.go:432
INFO[0000] Starting target manager...                    source=targetmanager.go:61
INFO[0000] Listening on :9090                            source=web.go:259
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x4 pc=0x11580]

goroutine 182 [running]:
sync/atomic.addUint64(0x12448474, 0x1, 0x0, 0x1035a2c, 0x1244844c)
        /usr/local/go/src/sync/atomic/64bit_arm.go:31 +0x4c
github.com/prometheus/prometheus/storage/remote.(*ewmaRate).incr(0x12448474, 0x1, 0x0)
        /go/src/github.com/prometheus/prometheus/storage/remote/ewma.go:65 +0x30
github.com/prometheus/prometheus/storage/remote.(*shards).enqueue(0x123175c0, 0x1250f060, 0x0)
        /go/src/github.com/prometheus/prometheus/storage/remote/queue_manager.go:427 +0x38
github.com/prometheus/prometheus/storage/remote.(*QueueManager).Append(0x12448400, 0x1250ed20, 0x1265e960, 0x0)
        /go/src/github.com/prometheus/prometheus/storage/remote/queue_manager.go:239 +0x190
github.com/prometheus/prometheus/storage/remote.(*Writer).Append(0x1265e960, 0x1250ed20, 0x0, 0x0)
        /go/src/github.com/prometheus/prometheus/storage/remote/write.go:79 +0x8c
github.com/prometheus/prometheus/storage.Fanout.Append(0x124c6c80, 0x2, 0x2, 0x1250ed20, 0x123902f0, 0x125d9338)
        /go/src/github.com/prometheus/prometheus/storage/storage.go:60 +0x58
github.com/prometheus/prometheus/storage.(*Fanout).Append(0x124c71a0, 0x1250ed20, 0x14047d14, 0x123902d3)
        <autogenerated>:3 +0x64
github.com/prometheus/prometheus/retrieval.(*countingAppender).Append(0x126da0a0, 0x1250ed20, 0x14047cfc, 0x125d9378)
        /go/src/github.com/prometheus/prometheus/retrieval/target.go:261 +0x3c
github.com/prometheus/prometheus/retrieval.ruleLabelsAppender.Append(0x1e053b8, 0x126da0a0, 0x1254c140, 0x1250ed20, 0x126da0c0, 0x1265340)
        /go/src/github.com/prometheus/prometheus/retrieval/target.go:201 +0x138
github.com/prometheus/prometheus/retrieval.(*ruleLabelsAppender).Append(0x126da0c0, 0x1250ed20, 0x1e06ad8, 0x126da0c0)
        <autogenerated>:35 +0x64
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append(0x12328140, 0x124be700, 0xb, 0xc8, 0xe, 0x256ae0f6, 0x1eabaa0)
        /go/src/github.com/prometheus/prometheus/retrieval/scrape.go:519 +0x224
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0x12328140, 0xfc23ac00, 0x6, 0x540be400, 0x2, 0x0)
        /go/src/github.com/prometheus/prometheus/retrieval/scrape.go:429 +0x4e0
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
        /go/src/github.com/prometheus/prometheus/retrieval/scrape.go:258 +0x330

Just FYI, if I comment out the remote_write stanza, it loads and I get the following output:
# prometheus -config.file=/etc/prometheus.yml
INFO[0000] Starting prometheus (version=1.6.1, branch=master, revision=4666df502c0e239ed4aa1d80abbbfb54f61b23c3)  source=main.go:88
INFO[0000] Build context (go=go1.8.1, user=root@ca1b452514b7, date=20170419-13:53:08)  source=main.go:89
INFO[0000] Loading configuration file /etc/prometheus.yml  source=main.go:251
INFO[0001] Loading series map and head chunks...         source=storage.go:421
WARN[0001] Persistence layer appears dirty.              source=persistence.go:846
WARN[0001] Starting crash recovery. Prometheus is inoperational until complete.  source=crashrecovery.go:40
WARN[0001] To avoid crash recovery in the future, shut down Prometheus with SIGTERM or a HTTP POST to /-/quit.  source=crashrecovery.go:41
INFO[0001] Scanning files.                               source=crashrecovery.go:55
INFO[0001] File scan complete. 0 series found.           source=crashrecovery.go:83
INFO[0001] Checking for series without series file.      source=crashrecovery.go:85
INFO[0001] Check for series without series file complete.  source=crashrecovery.go:131
INFO[0001] Cleaning up archive indexes.                  source=crashrecovery.go:411
INFO[0001] Clean-up of archive indexes complete.         source=crashrecovery.go:504
INFO[0001] Rebuilding label indexes.                     source=crashrecovery.go:512
INFO[0001] Indexing metrics in memory.                   source=crashrecovery.go:513
INFO[0001] Indexing archived metrics.                    source=crashrecovery.go:521
INFO[0001] All requests for rebuilding the label indexes queued. (Actual processing may lag behind.)  source=crashrecovery.go:540
WARN[0001] Crash recovery complete.                      source=crashrecovery.go:153
INFO[0001] 0 series loaded.                              source=storage.go:432
INFO[0001] Starting target manager...                    source=targetmanager.go:61
INFO[0001] Listening on :9090                            source=web.go:259
[...]

No crash as far as I can tell.

It doesn't look like a nil pointer exception, but some other segmentation fault that happens in the arm64 implementation of the sync/atomic package.
I suspected an alignment issue like in https://github.com/elastic/beats/issues/3273, but the field we atomically increment is already the first in its struct, so it should be aligned properly: https://github.com/prometheus/prometheus/blob/master/storage/remote/ewma.go#L24
From https://golang.org/pkg/sync/atomic/:
"On both ARM and x86-32, it is the caller's responsibility to arrange for 64-bit alignment of 64-bit words accessed atomically. The first word in a global variable or in an allocated struct or slice can be relied upon to be 64-bit aligned."
So now I don't know if this is a Go bug or what... I don't have an ARM laying around.
I wonder if you could try compiling the following Go program for your rpi and see if it crashes?
https://play.golang.org/p/Wz42HYfwT2

@juliusv
It doesn't seem like it crashes from what I can tell:
root@3c7a719:~# go build test.go
root@3c7a719:~# ls -l
total 412
-rwxr-xr-x 1 root root 414536 May  1 18:33 test
-rw-r--r-- 1 root root    139 May  1 18:32 test.go
root@3c7a719:~# ./test
root@3c7a719:~# echo $?
0

@vpetersson Thanks! Strange. Did you also use Go 1.8.1 for this, which was the Go version from which your broken Prometheus was built?
Maybe I should get a Raspberry Pi :) @discordianfish Can you reproduce this on yours, perhaps?

@juliusv No, this was not 1.8.1. I simply installed the golang apt package available for Raspbian:
$ go version
go version go1.3.3 linux/arm

Ah. Since I'm suspecting a Go bug, it would be super helpful if someone could try it out with Go 1.8.1.

@juliusv Pulled 1.8.1 from upstream and ran it again. Same result:
root@3c7a719:~# /usr/local/go/bin/go build test.go
root@3c7a719:~# ./test
root@3c7a719:~# echo $?
0
root@3c7a719:~# /usr/local/go/bin/go version
go version go1.8.1 linux/arm

Thanks! Hm ok. Anyone want to donate an rpi to me? :) Or rather, looks like someone who has one has to dig deeper into this...

I can send you one. I've followed you on Twitter. Follow me back and DM me your address.

You can find the address in
https://github.com/prometheus/prometheus/blob/master/MAINTAINERS.md
On Wed, May 3, 2017 at 11:50 AM Viktor Petersson<EMAIL_ADDRESS>wrote:

I can send you one. I've followed you on Twitter. Follow me back and DM me
your address.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/prometheus/prometheus/issues/2666#issuecomment-298867442,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAANaJigiDQAqmxasv2wMM5-GuasWMk0ks5r2E3KgaJpZM4NL5_Q
.

@grobie It's kinda hard to FedEx a parcel to an email address. ;)

Wooops, sorry for the noise. The message before was collapsed in my client
;-)
On Wed, May 3, 2017 at 12:07 PM Viktor Petersson<EMAIL_ADDRESS>wrote:

@grobie https://github.com/grobie It's kinda hard to FedEx a parcel to
an email address. ;)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/prometheus/prometheus/issues/2666#issuecomment-298870773,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAANaERXEI46nGZTXVKlAb9UJ3gnjIcEks5r2FHIgaJpZM4NL5_Q
.

@vpetersson What a scheme to get followers! Hehe. But this is awesome, thanks! Done.

@juliusv Probably the most expensive scheme ever invented. :)

@juliusv it could be that the embedded ewmaRates are not aligned: https://github.com/prometheus/prometheus/blob/master/storage/remote/queue_manager.go#L188.
Go seems to offer very few guarantees re: alignment - https://golang.org/ref/spec#Size_and_alignment_guarantees
@SuperQ was running the remote-write code on a Pi at KubeCon, but that may have been a Pi3 (64bit) or with an earlier version that didn't include the dynamic sharding code.
I have a Pi3 here so can give this a quick go.

@tomwilkie Yeah I've had it running in the past too. Please note that this was running ResinOS (which is basically running Docker on the Pi). Not sure if that changes anything.

I was running on a pine64 .
I have a pi3 and pi1 I can test on.

Oh man, even the unit test reproduce the issue:
$ go test
--- FAIL: TestSampleDelivery (4.83s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x4 pc=0x126b4]

goroutine 28 [running]:
testing.tRunner.func1(0x1076a180)
	/usr/local/go/src/testing/testing.go:622 +0x254
panic(0x28ebd0, 0x425250)
	/usr/local/go/src/runtime/panic.go:489 +0x288
sync/atomic.addUint64(0x10794074, 0x1, 0x0, 0x243150, 0x1079404c)
	/usr/local/go/src/sync/atomic/64bit_arm.go:31 +0x4c
github.com/prometheus/prometheus/storage/remote.(*ewmaRate).incr(0x10794074, 0x1, 0x0)
	/home/pi/go/src/github.com/prometheus/prometheus/storage/remote/ewma.go:65 +0x30
github.com/prometheus/prometheus/storage/remote.(*shards).enqueue(0x107f4150, 0x136fd900, 0x0)
	/home/pi/go/src/github.com/prometheus/prometheus/storage/remote/queue_manager.go:427 +0x38
github.com/prometheus/prometheus/storage/remote.(*QueueManager).Append(0x10794000, 0x107f2720, 0x64, 0x2a05f200)
	/home/pi/go/src/github.com/prometheus/prometheus/storage/remote/queue_manager.go:239 +0x190
github.com/prometheus/prometheus/storage/remote.TestSampleDelivery(0x1076a180)
	/home/pi/go/src/github.com/prometheus/prometheus/storage/remote/queue_manager_test.go:106 +0x3e0
testing.tRunner(0x1076a180, 0x2dcac8)
	/usr/local/go/src/testing/testing.go:657 +0x88
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:697 +0x25c
exit status 2
FAIL	github.com/prometheus/prometheus/storage/remote	4.923s

Bring on the ARM CI...

Okay, its embedded struct alignment:
package main

import (
        "sync/atomic"
)

type a struct {
        counter int64
}

type b struct {
        b uint32
        a a
}

func main() {
        foo := &b{}
        atomic.AddInt64(&foo.a.counter, 1)
}

$ go run test.go
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x4 pc=0x11204]

goroutine 1 [running]:
sync/atomic.addUint64(0x1030a0a4, 0x1, 0x0, 0x103000f0, 0x103000f0)
	/usr/local/go/src/sync/atomic/64bit_arm.go:31 +0x4c
main.main()
	/home/pi/test.go:18 +0x50
exit status 2

And it seems like multiple embedded structs are tightly-packed:
package main

import (
        "sync/atomic"
)

type a struct {
        counter int64
        a uint32
}

type b struct {
        a1 a
        a2 a
        b uint32
}

func main() {
        foo := &b{}
        atomic.AddInt64(&foo.a1.counter, 1)
        atomic.AddInt64(&foo.a2.counter, 1)
}

$ go run test.go
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x4 pc=0x11204]

goroutine 1 [running]:
sync/atomic.addUint64(0x1036600c, 0x1, 0x0, 0x1, 0x0)
	/usr/local/go/src/sync/atomic/64bit_arm.go:31 +0x4c
main.main()
	/home/pi/test.go:21 +0x60
exit status 2

TBH I don't see a want of guaranteeing alignment that isn't super fragile!

Wat? Ah finally context :) I just now got the notification.. If you still have something to test, let me know. Otherwise there is also scaleway.

thanks @discordianfish; I've got a pi at home and tested the above code.  TLDR; atomicly-accessed int64s must be 64-bit aligned.  Can't see a way to guarantee this in go that isn't super fragile.
Some more reading:

https://github.com/golang/go/issues/599
https://github.com/golang/go/issues/18955
https://groups.google.com/forum/#!topic/golang-nuts/_nHK6P8_lhw
https://github.com/golang/go/issues/11891

@vpetersson mind testing out the 266-alignment-on-arm branch?  Its work for me.

@tomwilkie I guess you mean this branch in your fork: https://github.com/tomwilkie/prometheus/tree/2666-alignment-on-arm
Ok, so it's an alignment problem after all. Interesting that the statement about first words in structs being aligned from the sync/atomic docs is apparently not true.
We also use atomic accesses in other places, such as for counter increments in the Prometheus client library. So it's probably pure luck that those places aren't crashing on ARM right now. We did have the same problem with x86 32-bit earlier, but solved that by having the atomically accessed variable as the first member in the struct (which doesn't seem to help this time).

Ok, so it's an alignment problem after all. Interesting that the statement about first words in structs being aligned from the sync/atomic docs is apparently not true.

You need to be careful about how to parse the phrase, as what it means is "The first word ... in an allocated struct ... can be relied upon to be 64-bit aligned."  Embedded structs are not allocated structs.

@tomwilkie Man, that's a gotcha. Thanks.  Saves us all the extra padding fields in https://github.com/tomwilkie/prometheus/commit/f1806c3672f34f2064b219d8fdc2af35e1fafe86. AFAIU there is no guarantee about those padding fields creating alignment either?

Sure, happy with that.

I can confirm that the problem is still present in 1.6.2:
May 14 11:59:03 3c7a719 prometheus[255]: time="2017-05-14T11:59:03Z" level=info msg="Loading configuration file /etc/prometheus.yml" source="main.go:251"
May 14 11:59:03 3c7a719 prometheus[255]: time="2017-05-14T11:59:03Z" level=info msg="Loading series map and head chunks..." source="storage.go:421"
May 14 11:59:03 3c7a719 prometheus[255]: time="2017-05-14T11:59:03Z" level=warning msg="Persistence layer appears dirty." source="persistence.go:846"
May 14 11:59:03 3c7a719 prometheus[255]: time="2017-05-14T11:59:03Z" level=warning msg="Starting crash recovery. Prometheus is inoperational until complete." source="crashrecovery.go:40"
May 14 11:59:03 3c7a719 prometheus[255]: time="2017-05-14T11:59:03Z" level=warning msg="To avoid crash recovery in the future, shut down Prometheus with SIGTERM or a HTTP POST to /-/quit." source="crashrecovery.go:41"
May 14 11:59:03 3c7a719 prometheus[255]: time="2017-05-14T11:59:03Z" level=info msg="Scanning files." source="crashrecovery.go:55"
May 14 11:59:03 3c7a719 prometheus[255]: time="2017-05-14T11:59:03Z" level=info msg="File scan complete. 228 series found." source="crashrecovery.go:83"
May 14 11:59:03 3c7a719 prometheus[255]: time="2017-05-14T11:59:03Z" level=info msg="Checking for series without series file." source="crashrecovery.go:85"
May 14 11:59:03 3c7a719 prometheus[255]: time="2017-05-14T11:59:03Z" level=info msg="Check for series without series file complete." source="crashrecovery.go:131"
May 14 11:59:03 3c7a719 prometheus[255]: time="2017-05-14T11:59:03Z" level=info msg="Cleaning up archive indexes." source="crashrecovery.go:411"
May 14 11:59:03 3c7a719 prometheus[255]: time="2017-05-14T11:59:03Z" level=info msg="Clean-up of archive indexes complete." source="crashrecovery.go:504"
May 14 11:59:03 3c7a719 prometheus[255]: time="2017-05-14T11:59:03Z" level=info msg="Rebuilding label indexes." source="crashrecovery.go:512"
May 14 11:59:03 3c7a719 prometheus[255]: time="2017-05-14T11:59:03Z" level=info msg="Indexing metrics in memory." source="crashrecovery.go:513"
May 14 11:59:03 3c7a719 prometheus[255]: time="2017-05-14T11:59:03Z" level=info msg="Indexing archived metrics." source="crashrecovery.go:521"
May 14 11:59:03 3c7a719 prometheus[255]: time="2017-05-14T11:59:03Z" level=info msg="All requests for rebuilding the label indexes queued. (Actual processing may lag behind.)" source="crashrecovery.go:540"
May 14 11:59:03 3c7a719 prometheus[255]: time="2017-05-14T11:59:03Z" level=warning msg="Crash recovery complete." source="crashrecovery.go:153"
May 14 11:59:03 3c7a719 prometheus[255]: time="2017-05-14T11:59:03Z" level=info msg="579 series loaded." source="storage.go:432"
May 14 11:59:03 3c7a719 prometheus[255]: time="2017-05-14T11:59:03Z" level=info msg="Starting target manager..." source="targetmanager.go:61"
May 14 11:59:03 3c7a719 prometheus[255]: time="2017-05-14T11:59:03Z" level=info msg="Listening on :9090" source="web.go:259"
May 14 11:59:05 3c7a719 prometheus[255]: panic: runtime error: invalid memory address or nil pointer dereference
May 14 11:59:05 3c7a719 prometheus[255]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x4 pc=0x11580]
May 14 11:59:05 3c7a719 prometheus[255]: goroutine 209 [running]:
May 14 11:59:05 3c7a719 prometheus[255]: sync/atomic.addUint64(0x12230174, 0x1, 0x0, 0xf51a38, 0x1223014c)
May 14 11:59:05 3c7a719 prometheus[255]: /usr/local/go/src/sync/atomic/64bit_arm.go:31 +0x4c
May 14 11:59:05 3c7a719 prometheus[255]: github.com/prometheus/prometheus/storage/remote.(*ewmaRate).incr(0x12230174, 0x1, 0x0)
May 14 11:59:05 3c7a719 prometheus[255]: /go/src/github.com/prometheus/prometheus/storage/remote/ewma.go:65 +0x30
May 14 11:59:05 3c7a719 prometheus[255]: github.com/prometheus/prometheus/storage/remote.(*shards).enqueue(0x122a6690, 0x124d7020, 0x0)
May 14 11:59:05 3c7a719 prometheus[255]: /go/src/github.com/prometheus/prometheus/storage/remote/queue_manager.go:427 +0x38
May 14 11:59:05 3c7a719 prometheus[255]: github.com/prometheus/prometheus/storage/remote.(*QueueManager).Append(0x12230100, 0x124d6d60, 0x12330990, 0x0)
May 14 11:59:05 3c7a719 prometheus[255]: /go/src/github.com/prometheus/prometheus/storage/remote/queue_manager.go:239 +0x190
May 14 11:59:05 3c7a719 prometheus[255]: github.com/prometheus/prometheus/storage/remote.(*Writer).Append(0x12330990, 0x124d6d60, 0x0, 0x0)
May 14 11:59:05 3c7a719 prometheus[255]: /go/src/github.com/prometheus/prometheus/storage/remote/write.go:79 +0x8c
May 14 11:59:05 3c7a719 prometheus[255]: github.com/prometheus/prometheus/storage.Fanout.Append(0x1247a5a0, 0x2, 0x2, 0x124d6d60, 0x134abb00, 0x13568e28)
May 14 11:59:05 3c7a719 prometheus[255]: /go/src/github.com/prometheus/prometheus/storage/storage.go:60 +0x58
May 14 11:59:05 3c7a719 prometheus[255]: github.com/prometheus/prometheus/storage.(*Fanout).Append(0x1247a7a0, 0x124d6d60, 0x1350bd14, 0x13568e10)
May 14 11:59:05 3c7a719 prometheus[255]: <autogenerated>:3 +0x64
May 14 11:59:05 3c7a719 prometheus[255]: github.com/prometheus/prometheus/retrieval.(*countingAppender).Append(0x134ee5d0, 0x124d6d60, 0x1350bcfc, 0x13568e68)
May 14 11:59:05 3c7a719 prometheus[255]: /go/src/github.com/prometheus/prometheus/retrieval/target.go:261 +0x3c
May 14 11:59:05 3c7a719 prometheus[255]: github.com/prometheus/prometheus/retrieval.ruleLabelsAppender.Append(0x1cb54e0, 0x134ee5d0, 0x12488540, 0x124d6d60, 0x134ee5e0, 0x11761f0)
May 14 11:59:05 3c7a719 prometheus[255]: /go/src/github.com/prometheus/prometheus/retrieval/target.go:201 +0x138
May 14 11:59:05 3c7a719 prometheus[255]: github.com/prometheus/prometheus/retrieval.(*ruleLabelsAppender).Append(0x134ee5e0, 0x124d6d60, 0x1cb6b60, 0x134ee5e0)
May 14 11:59:05 3c7a719 prometheus[255]: <autogenerated>:35 +0x64
May 14 11:59:05 3c7a719 prometheus[255]: github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append(0x121f2900, 0x12267500, 0xb, 0xc8, 0xe, 0x33e0cd1, 0x1d5e298)
May 14 11:59:05 3c7a719 prometheus[255]: /go/src/github.com/prometheus/prometheus/retrieval/scrape.go:519 +0x224
May 14 11:59:05 3c7a719 prometheus[255]: github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0x121f2900, 0xfc23ac00, 0x6, 0x540be400, 0x2, 0x0)
May 14 11:59:05 3c7a719 prometheus[255]: /go/src/github.com/prometheus/prometheus/retrieval/scrape.go:429 +0x4e0
May 14 11:59:05 3c7a719 systemd[1]: prometheus.service: main process exited, code=exited, status=2/INVALIDARGUMENT
May 14 11:59:05 3c7a719 systemd[1]: Unit prometheus.service entered failed state.
May 14 11:59:05 3c7a719 prometheus[255]: created by github.com/prometheus/prometheus/retrieval.(*scrapePool).sync
May 14 11:59:05 3c7a719 prometheus[255]: /go/src/github.com/prometheus/prometheus/retrieval/scrape.go:258 +0x330
May 14 11:59:05 3c7a719 systemd[1]: scanbt-worker.service holdoff time over, scheduling restart.

@tomwilkie Could you check the latest release version (1.6.2) on your rpi 3 again? I wonder if there is a difference between your rpi and @vpetersson's, but at least they both seem to be an rpi 3. I doubt that the exact OS should make a difference, as it's an architectural memory access issue that should be independent of the OS?

I think the issue is that the fix[1] didn't make it onto the 1.6 branch[2].  Since its been merged already, should I make a cherry pick for 1.6.3?
[1] https://github.com/prometheus/prometheus/commit/3141a6b36be4aacd1f84dda8927cef9f3f2f1368
[2] https://github.com/prometheus/prometheus/commits/release-1.6

I think there were plans to release 1.7 soon, are we sure we'll have a 1.6.3 release? We need to get better on marking changes to be included in the next patch release in general.

This fix should be out now.

I can confirm that this is now working. Thanks, guys!


Load jsonlite and rlang from custom lib.loc so session watcher can start from renv project

Right now when running R from a renv project where the packages are not installed the session watcher does not work but throws:
VSCode R Session Watcher requires jsonlite, rlang. Please install manually in order to use VSCode-R.

I can of course install the packages in the project but don't want my projects to include unnecessary dependencies. Would it be possible to set a custom library location for requireNamespace() somehow?

This could be done in https://github.com/Ikuyadeu/vscode-R/blob/master/R/.Rprofile by sourcing project .Rprorfile created by renv after sourcing session watcher init.R.
Would you like to try editing your project .Rprofile by putting the following code before renv init:
if (!exists(".vsc")) {
  source(file.path(
    Sys.getenv(if (.Platform$OS.type == "windows") "USERPROFILE" else "HOME"),
    ".vscode-R", "init.R")
  )
}

Thanks! That would work, but now I'm sourcing init.R from ~/.Rprofile which seems better. Sourcing init.R from within the project has the same effect as including the packages in renv: I introduce an unneeded dependency on VSCode.
Is there maybe a way to get R to source ~/.Rprofile before project .Rprofile?

The "R: Create R terminal" is trying to mimic the R's native behavior that if a project .Rprofile is present, then it will not source ~/.Rprofile.
It works for you then I think we could provide a vscode setting so that user could customize the order of execution in https://github.com/Ikuyadeu/vscode-R/blob/master/R/.Rprofile:

optionally source user profile
source project profile
source session watcher script

Ahh right I set RENV_CONFIG_USER_PROFILE=TRUE in ~/.Renviron to ensure my user Rprofile is still sourced.
Problem with "R: Create R terminal" is that I'm using tmux on a server with SSH Remote and need to start R from within tmux.

Problem with "R: Create R terminal" is that I'm using tmux on a server with SSH Remote and need to start R from within tmux.

Yes, if you are using tmux to preserve your R session, then you need to customize your profile to source session watcher on your own, which actually gives you more freedom of control the order and behavior.

Exactly, thats what I'm trying to do :). Trying to do it without bogging down my research project with personal coding preferences that the others working in the project are not interested in.
Was thinking that the easiest way would be to somehow get the init.R script to use a custom library instead of renv, but the alternative would be to get ~/.Rprofile to be sourced through renv before project .Rprofile which would require changed to that package instead.

When using renv, after loading the project custom environment for the first time I have to install the languageserver and whatever dependency it needs. These won't be included in the renv.lock file because it scans from calls inside your code; so, it's a bit annoying, of course, but I don't count as "introducing unneeded dependency", since when you share the code and others perform an renv::restore(), these packages won't be installed.

Indeed thats what I'm doing right now. Works, but a bit frustrating when opening new projects etc.

Think I found a solution: Install languageserver etc in your main library, then set the env variable R_LIBS_USER in .bashrc or similar and run lsp in vanilla: "r.lsp.args": ["--vanilla"]. Seems to work!


refactor(release): use ncc to compile dependencies

Hi @eddiejaoude
Nice action! I saw it linked from the GitHub blog today.
I noticed you checked in node_modules into your repository. Just wanted to let you know that you can avoid doing that by using ncc to compile your source code and dependencies into a single Javascript file. This is common practice for all official GitHub actions. I do the same with all of my Typescript/Javascript actions.
You can compile and generate the dist/index.js file by running npm run build.
Don't feel any pressure to merge this if you want to continue checking in node_modules. Both methods work. 😄

I noticed you checked in node_modules into your repository
Don't feel any pressure to merge this if you want to continue checking in node_modules

Thank you, I didnt like how I had to check them in 😢 . This is great 👍, thank you 🤓


Move from github.com/AkihiroSuda to somewhere else

github.com/lima is already taken by another project, so we have to come up with another name, e.g., github.com/linuxmachines.
We can also consider renaming lima itself, as there has been also Lima GPU driver project (https://gitlab.freedesktop.org/lima). (But finding an unique/short/rememberable name is not easy 😅 )

I like the name lima, so would prefer not to rename the project itself. I think it will be clear from the context if somebody is referring to lima the GPU driver, or lima, the VM manager. 😄
I'm partial to using lima-vm as the org name, as it keeps the association with the lima project name, and makes it clear that is is not about GPUs... You would have a lima-vm/lima repo, and if you decide to move it, lima-vm/vde_vmnet as well, which looks very descriptive to me.
(I mistyped lima as lime so many times that I used  as the color for the lima tag in the rancher-desktop repo 😂 )

A lot of organisations revolving around a single project add some sort of extension for the org name (e.g. preactjs/preact, clap-rs/clap, darlinghq/darling) or spell out the domain name they have (e.g. warpdotdev/warp for https://warp.dev, browsh-org/browsh for https://browsh.org). So regardless of the name, whether you keep Lima or not, lima-sh/lima, limash/lima, lima-org/lima, lima-vm/lima, lima-dev/lima and the likes could all be interesting options (maybe with bonus point if there's an available domain name that could be useful later on?).
As for the name of the project itself I personally like the name Lima, and although there's the Lima GPU Driver, I feel like they don't really overlap and that people wouldn't really confuse them (it's not like they're both libs for the same language). I've tried to come up with short, pronounceable two-syllable alternative names that could potentially be considered, and tried to think about the command, org, domain that could go with that:

Name
Meaning
Command
Control Command
GitHub
Domain Name
Conflicts
Comments

Lima
Linux on macOS
lima
limactl
lima-vm/lima
lima-vm.com
Lima GPU Driver?
Peru's capital

Lama
Linux and macOS
lama
lamactl
lama-vm/lama
lama-vm.com

The animal could make for a great mascot

Mali
macOS and Linux
mali
malictl
mali-sh/mali
mali.sh
Mali GPUs?

Malin
macOS and Linux
ml/malin
mlctl/malinctl
malinsh/malin
malin.sh

French word for "Smart"

Dali
Darwin and Linux
dali
dalictl
dali-sh/dali
dali.sh
NVIDIA's GPU-accelerated lib for Deep Learning
Salvador Dali

Dalin
Darwin and Linux
dln/dalin
dlnctl/dalinctl
dalin-sh/dalin
dalin.sh
?

Darli
Darwin and Linux
drl/darli
drlctl/darlictl
darli-sh/darli
darli.sh
?

Lada
Linux and Darwin
lada
ladactl
lada-sh/lada
lada.sh

Russian car brand, cheap but very reliable

Lida
Linux on Darwin
lida
lidactl
lidadotsh/lida
lida.sh

Linda
Linux on Darwin
linda
lindactl
lindash/linda, lindadotsh/linda
linda.sh

Spanish word for "Pretty", "Beautiful"

This is the result of a quick brainstorm. Something to keep in mind is that there is the Darling project (darlinghq/darling) that allows you to run macOS apps in Linux.

Quick comment on "lama": (in English) that "is a title for a teacher of the Dharma in Tibetan Buddhism". The animal is spelled "llama", and for me that has always been associated with the O'Reilly Llama book (Learning Perl).
Since I prefer to keep the name "Lima" I don't want to press on this too much, but Lima is actually not restricted to running Linux on Mac; it supports running Linux on Linux as well. And while I don't think there is any point in running qemu on Windows (you already have WSL), I think there may be a role for using the lima guest and hostagents with WSL2.
So if there is a name change, it doesn't have to be restricted to some combination of Linux and macOS, but could use any virtualization related name. My vote is still for lima though (not sure if there is voting though 😄 ).

Yeah I knew it wasn't the right spelling but figured that if it was going to be the name, might be fun to play on the fact that they're homophonous 😄. My personal preference is also to keep Lima.
It may go a little be off-topic but just so I understand: is there a point/use-case for using Lima on Linux? Or is it just that it's possible but no one is ever going to use it that way? (kind of like Homebrew exists for Linux but practically no one really uses it)
The description of the repo is "Linux virtual machines, on macOS (aka Linux-on-Mac, macOS subsystem for Linux, containerd for Mac, unofficially)". It may be a little bit outdated already (you can go without containerd for instance). Does Lima now simply mean Linux Machines instead of having ties to macOS? (@AkihiroSuda wanted to call the org linuxmachines)
I'm just wondering because I was working on a logo proposal for the project and the actual meaning is important for that

is there a point/use-case for using Lima on Linux?

I think so (isolation), but it is not as strong as the macOS use case. But let's say you want to create a local kubernetes cluster with multiple worker nodes, then Lima would be very helpful, as you cannot directly run multiple kubelets on the same host.
As for linuxmachines, that is certainly correct right now, but I wonder if it will eventually be possible to run Windows VMs with Windows containers on qemu too. Maybe it won't, but that's why I prefer to think of lima as a VM manager, like an opinionated higher level abstraction than qemu itself.

I see yeah that makes a lot of sense, thank you!

is there a point/use-case for using Lima on Linux?

This is the exact use case I was searching for a solution when I found Lima. I would like a dependency free tool for launching test VMs (like Vagrant, but without the Ruby baggage). I know Vagrant is being rewritten in Go, but it could be awhile before 3.0 is ready and there is no official qemu/libvirt driver. If I could find Go replacements for Vagrant/Virtualbox and Kitchen/Inspec, I would be in heaven. Bonus points for working on both macOS and Linux. Lima seems like it's headed in the right direction.

As for linuxmachines, that is certainly correct right now, but I wonder if it will eventually be possible to run Windows VMs with Windows containers on qemu too. Maybe it won't, but that's why I prefer to think of lima as a VM manager, like an opinionated higher level abstraction than qemu itself.

I wasn't planning to support Windows guests, but I'm happy to accept PRs 😃

I wasn't planning to support Windows guests, but I'm happy to accept PRs

This is mostly personal curiosity to see if this will be possible at all. I suspect people actually interested in running Windows containers will typically be using Windows laptops/desktops for development too.
Running on a Linux or macOS host will also be impractical because you would need an extra Windows license for the guest VM (I think you need at least Windows Pro to run the Windows version of containerd for development purposes; the hypervisor support in Windows Home is restricted to WSL2).

I was about to create github.com/linuxmachines org but I noticed that https://github.com/linuxmachine (without s) is already taken, so  github.com/linuxmachines might be confusing... 😅

I was about to create github.com/linuxmachines org but I noticed that https://github.com/linuxmachine (without s) is already taken, so  github.com/linuxmachines might be confusing... 😅

github.com/linux-machines looks available, I think adding that dash makes it visually different enough (plus a quick look at the README can let people know they're at the wrong place?)

https://github.com/linuxmachine (without s) is already taken

It is a user that has been created more than 2½ years ago, and hasn't been touched since it was registered:
$ gh api /users/linuxmachine | jq -r .created_at,.updated_at
2018-12-01T20:17:44Z
2018-12-01T20:17:44Z

I suspect whoever created it has probably lost the credentials for it by now, so I think it makes little difference if this shows an empty user page, or a Github 404 page... 😄
So I wouldn't worry about it too much. On the other hand, lima-vm is still available and maybe less confusing. 😆


Make sure getNumber() works as expected with every ar-* (Arabic) locale that will be supported.

Is your feature request related to a problem? Please describe.
We have to make sure that getNumber() works as expected with every ar-* (Arabic) locale we are about to support.
Describe the solution you'd like
New get-number.ar-*.spec.js files must be created to test thoroughly that getNumber() works as expected with every ar-* locale.
Describe alternatives you've considered
N/A.
Additional context
The tests must include some manual test cases and, then, the needed automatic tests as stated here.

Hello @gkamtzir
I would like to try and help with this issue.
So I am kindly asking you to assign this issue to me.

@st3rgios Sure, go ahead!


Login Page CSS Not Loading - Incorrect Path

Is there an existing issue for this?

[X] I have searched the existing issues

What happened?
Hey @anuragverma108 . This issue reports that the CSS file for the login page (login.css) is not loading correctly.
The current path specified in login.html is:    href="login.css"
However, the correct path should be:    href="./assets/css/login.css"
This discrepancy is causing the CSS to not be applied to the login page.
In addremovebook.html also the css file is not connected properly.
please add the gssoc labels and I would like to work on it under GSSOC'24. Thank you!!
Record

[X] I agree to follow this project's Code of Conduct
[X] I'm a GSSOC'24 contributor
[X] I want to work on this issue

Assigned!
Make sure to attach before & after screenshots in your PR. Also, star the repo before working on it. While creating your PR, ensure you pull all existing changes beforehand to avoid conflicts.
Happy open-source contributing!

Please assign this work to me sir


virtustotal - Change file name to file path

Please check the following conditions for your PR.

[ ] apiDefinition.swagger.json is validated using paconn validate command.
[ ] apiProperties.json has a valid brand color. Invalid brand colors are #007ee5 and #ffffff.

@sriyen-msft can you please review this?


feat: Add ActiveMQ Artemis module

What does this PR do?

Adds an ActiveMQ Artemis container to the list of supported Testcontainers.
Why is it important?

Provides an ActiveMQ Artemis test container.
How to test this PR

Run Testcontainers.ActiveMq.Tests

Nowadays, we can find ActiveMQ Classic and ActiveMQ Artemis. Although, the module refers to activemq, in general, would be great to be specific about Artemis instead of ActiveMQ. In Testcontainers for Java, the plan is to have one module to host both of them.

@eddumelendez I can also include the classic container in this project, if that is okay? Or should they be hosted in different projects?

ActiveMQ Classic doesn't support credentials in env vars right now, the image have not been published yet, that's why the PR is still open in Testcontainers for Java.
My comment is related to the naming rather than adding support for ActiveMQ Classic.

@eddumelendez Is there anything left we should address before merging the PR?


Improve logging setup - Logback/SLF4J instead of log4j?

Currently with Terasology we use Logback/SLF4J for logging and control it and any "bridged" logging frameworks from a single logback config file. I noticed while reviewing #415 that we're still doing log4j things in DS. For consistency's sake (and the added power of the newer frameworks) we should probably look into switching over.
See Terasology's setup for further details and https://logback.qos.ch/
Extra consideration here: Does Android require any additional considerations?

Not sure what extras they provide, but there are 2 repos I found:
sl4j-android: https://github.com/nomis/slf4j-android
logback-android: https://github.com/tony19/logback-android
The first repo is just the SL4J abstraction: https://www.slf4j.org/android/
The second repo is sl4j + the modified logback library.
I'll try out the second one, make a PR and ask someone to test it on Android.


Edit Profile dialog doesn't close or show any successful message

How to reproduce:
Click Edit Profile on user profile page, edit it, then click Save.

The dialog has no response at all: It doesn't close or show any successful message.

Fixed in #4495 #4496


Hello, I'd like to ask how the reward function in the code is designed, referring to the thesis?

thank you very much!

Hi, thanks for your interests.
The reward design are mainly refered to this paper, and you can also find more details in my slides. Specific rewards are defined in task config files, and computed in this function for each environment.


Unauthorized ssh push upon repo rename

Description
Renaming a repository results in a Gitea: Unauthorized message upon trying to push to the old ssh-based url
Gitea Version
1.18.0+dev-432-g548387b2b

Can you reproduce the bug on the Gitea demo site?
Yes
Log Gist
No response
Screenshots
No response
Git Version
No response
Operating System
No response
How are you running Gitea?
I've used try.gitea.io to confirm
Database
No response

I guess it would take a symlink on the disk to support such ssh-based "redirect"


epoll_wait produces an EINVAL error since 4.1.30

Fixes regression with 6d8745179844caba0f3a0f3d7f3a1fbf392c925f
Closes https://github.com/netty/netty/issues/8348

@normanmaurer I verified using debugger it works as expected but had trouble adding a unit test, I tried a bunch of different ways eg.
        Future<Integer> pendingTasks = el.submit(new Callable<Integer>() {
            @Override
            public Integer call() throws Exception {
                return ((EpollEventLoop) el).pendingTasks();
            }
        });
        assertEquals(Integer.valueOf(1), pendingTasks.get(1000, TimeUnit.SECONDS)); // accepted scheduled task

but no nice.

@netty-bot test this please

@normanmaurer yep

@johnou why not just make EpollEventLoop (but still keep it package-private) not final and change handleLoopException to be not static and package-private (+ add a comment this is because of testing). This way you could extend the class and override and just assert that it is not called.
WDYT ?

@normanmaurer i'll give that a go after lunch :+1:

@normanmaurer I could only get it working by removing the final from EpollEventLoopGroup too..

@johnou it should not be needed to do this... what was the problem ?

@normanmaurer PTAL I swapped out the EpollEventLoopGroup with EpollEventLoop directly in the test to avoid removing final from EpollEventLoopGroup.

@johnou this looks great... can you adjust the commit message to match our template tho ?

@normanmaurer yep sure, done.

@johnou thanks a lot!

LGTM


PDF display uses unwanted magnification

in the menu Files upload a PDF with the sheet size of A4
click on the projector icon
now look at the projector output

result: the PDF is displayed with a magnification of 200% - making it therefore often unreadable

now look at the projector preview

result: there the PDF is correctly displayed

Here is an example PDF:
Flyer-1.pdf
You can see that the default magnification is too large and definitely 100%. Maybe the bug is that the magnification is always displayed as "100%":

(no matter how often you press the + or - buttons besides the "100%")

100% means "full width" of the projector.
Examples:

If you project a widescreen presentation pdf you get black border on top/bottom.
If you project a A4 pdf you see the upper half of the page. But you can scroll down with the projector controls.

So this behavior is not a bug for me.
A valid feature request could be to show a PDF in full height (with black borders left/right for A4 page). Or better: By default each pdf projection should show the whole pdf document (= full width or height). For A4 = full height. For a widescreen presentaiton = full width.
@donovaly Is that what you wanted to report?

100% means "full width" of the projector.

Ah, now I understand.

By default each pdf projection should show the whole pdf document (= full width or height). For A4 = full height. For a widescreen presentaiton = full width.
@donovaly Is that what you wanted to report?

Yes, this would be a solution.
Btw. with the OS developer version you sent me in July I cannot present any PDF file. With OS 2.0 it works.

In OS 2.1 dev the situation is better but still not solved: On a A4 PDF one can now click on the button "reset zoom" but this results in a display where the whole projector width is used so that the majority of the PDF is invisible. Instead the whole height should be used.

Summarize final suggestion to fix this issue:
By default each pdf projection should show the whole pdf document (= full width or height).

For DIN A4 page = full height (horizontal centered, with black borders left/right)
For a widescreen presentaiton = full width (vertical centered, with black borders top/bottom).

Duplicate, see #2097


Positional parameters bug with late-resolved arity variable

Observed behaviour
For positional parameters, defining arity with a variable, like this: @Parameters(arity = "${xxx:-1..*}") has a bug:
regardless of the value of the system property xxx, the positional parameter is never mandatory/required.
Expected behaviour
if system property xxx is not defined, the default arity of 1..* means that at least one value is required. But this is not enforced.
How to reproduce
@Test(expected = CommandLine.MissingParameterException.class)
public void testRequiredPositionalWithUndefinedVariable() {
    class Cmd {
        @Parameters(arity = "${xxx1:-1..*}") List<String> positional;
    }
    new CommandLine(new Cmd()).parseArgs(); // should fail with missing param but doesn't
}

Analysis:
ArgSpec.required is derived from arity.min > 0, without considering whether the arity Range object has unresolved variables.
Any variables in the ArgSpec's arity are not resolved until the ArgSpec is added to its CommandSpec.
At this point, the ArgSpec's arity is replaced by a new Range object with valid values, but the ArgSpec's required attribute is not updated.

I finally got around to fixing this. Thanks again for raising this!


fix: use thanos default port in service and containerPort

Before thanos was only listening on localhost. Relying on the default setting allows user to modify that via server side apply and add a proxy chain.
Fixes: https://issues.redhat.com/browse/COO-14

/lgtm any reason on why it was only listening on localhost before? I assume it was proxied but it'd be good to clarify this on the PR message.

No reason, just a mistake :)

hmm it does seem that tests still are targeting the 9090 port E0208 11:47:09.149727   35900 portforward.go:409] an error occurred forwarding 9090 -> 9090: error forwarding port 9090 to pod 4d6e4b0f907b07e5f24151733d584f37bb83a6b9bc2483e568a8eaae49b9eb99, uid : failed to execute portforward in network namespace" so this would probably have to be changed as well.

Ok at least my local test failure reproduces here too.
E0208 11:47:09.149727   35900 portforward.go:409] an error occurred forwarding 9090 -> 9090: error forwarding port 9090 to pod 4d6e4b0f907b07e5f24151733d584f37bb83a6b9bc2483e568a8eaae49b9eb99, uid : failed to execute portforward in network namespace "/var/run/netns/cni-d3ec7578-2ca0-8f70-59dd-2a9a054707e9": failed to connect to localhost:9090 inside namespace "4d6e4b0f907b07e5f24151733d584f37bb83a6b9bc2483e568a8eaae49b9eb99", IPv4: dial tcp4 <IP_ADDRESS>:9090: connect: connection refused IPv6 dial tcp6 [::1]:9090: connect: connection refused
panic: lost connection to pod

Not sure what's happening here yet.

hmm it does seem that tests still are targeting the 9090 port E0208 11:47:09.149727 35900 portforward.go:409] an error occurred forwarding 9090 -> 9090: error forwarding port 9090 to pod 4d6e4b0f907b07e5f24151733d584f37bb83a6b9bc2483e568a8eaae49b9eb99, uid : failed to execute portforward in network namespace" so this would probably have to be changed as well.

I don't think so. This error happens in TestMonitoringStackController/Verify_multi-namespace_support and the port forward is to prometheus. Here we change the thanos service port.

Or am I just not smart enough to read the output correctly :thinking:

Or am I just not smart enough to read the output correctly 🤔

Sure looks that way. Testing locally now since here it was failing in a prometheus test.

Ok this is the error I'm seeing locally too:
=== RUN   TestMonitoringStackController/Verify_multi-namespace_support
E0208 14:22:19.405068   28162 portforward.go:409] an error occurred forwarding 9090 -> 9090: error forwarding port 9090 to pod b29bce2e1202569aeef6862fdfe16660317d86f50ea82774ee9b1177a236ec14, uid : failed to execute portforward in network namespace "/var/run/netns/cni-a601776d-003d-7bbe-fde4-a5b9daa71b03": failed to connect to localhost:9090 inside namespace "b29bce2e1202569aeef6862fdfe16660317d86f50ea82774ee9b1177a236ec14", IPv4: dial tcp4 <IP_ADDRESS>:9090: connect: connection refused IPv6 dial tcp6 [::1]:9090: connect: connection refused
panic: lost connection to pod

Port 9090 is correct in this context and this code didn't change.

hmm it does seem that tests still are targeting the 9090 port E0208 11:47:09.149727 35900 portforward.go:409] an error occurred forwarding 9090 -> 9090: error forwarding port 9090 to pod 4d6e4b0f907b07e5f24151733d584f37bb83a6b9bc2483e568a8eaae49b9eb99, uid : failed to execute portforward in network namespace" so this would probably have to be changed as well.

I don't think so. This error happens in TestMonitoringStackController/Verify_multi-namespace_support and the port forward is to prometheus. Here we change the thanos service port.

Yep, I was a bit too quick into this as the port was just matching the one you changed over there, but after checking the tests code this should be unrelated.

@sthaha mind taking a look at this? thanks!

I have disabled the test for evicting alertmanager pods under the PDB for now. I'll keep investigating this, but I also wonder whether this is a test we need to run. I'd argue we test k8s api here and that is out of scope for this operator.

/retest

/retest

/retest

/retest

/retest

/retest

/lgtm


Nullable fields

You say that all fields are nullable, but now I got error
sql: Scan error on column index 4: unsupported Scan, storing driver.Value type \u003cnil\u003e into type *string

Same here, getting an error for nil-able string columns.
sql: Scan error on column index 6: unsupported Scan, storing driver.Value type <nil> into type *string

I get the same error when having a string field but the database has a null value. IMHO the value should be loaded as empty string in Kallax instead of throwing an error.

Right now, db NULLs are translated to nil if go type is nullable (aka pointers). So *string should be set to NULL if set on the database, but trying to scan NULL into string (no pointer) should throw an error.
@qJkee Could you provide the code triggering this behaviour? This is, your model definition and how are you retrieving the data.

type Account struct { 	kallax.Model  table:"account"	ID            int64pk:"autoincr" 	Name          string 	IconURL       string 	RefuseComment string }
For example, if RefuseCimment is null, it causes error

Try changing RefuseComment string to RefuseComment *string.

So, is it possible to set default values instead of returning an error? As I knew, that was that before

You might want to give custom constructors a try. I'm not sure if they will work on all queries, tho.

looks like this is the only solution, coz if I use pointers, I will  get a segfault when accessing field, which is a null pointer

Well, a database NULL is translated to a nil pointer, yes. This is the expected behaviour. Using NULL as an alias for some default value is a terrible practice. Maybe in the future, when schema generation gets improved, we can add some support for default values via struct tags.
I'm closing this. Feel free to reopen if some other unexpected behaviour appears.


[#373] add fsd extension to supported scl types

@Sander3003 I just assigned you assuming you know best who is able to have a look at it, feel free to reassign it if needed

Fixes #373

Looks good to me, from a technical point of view, @trusz and @clepski are already offline for this year. They can check next year.


unexpected kernel restart with an omitted semicolon

System Information
Platform: OSX 10.12.16
conda 4.5.2 env info (partial list):
clangdev                  5.0.0                   cling_1  [cling]  QuantStack
cling                     0.5                           7  [cling]  QuantStack
cling-patches             1                             0    QuantStack
cppzmq                    4.2.3                   cling_0  [cling]  QuantStack
jsonschema                2.6.0                    py36_1    conda-forge
jupyter_client            5.2.3                    py36_0    conda-forge
jupyter_core              4.4.0                      py_0    conda-forge
nlohmann_json             3.1.2                         0    QuantStack
notebook                  5.4.1                    py36_0    conda-forge
xeus                      0.12.0                  cling_0  [cling]  QuantStack
xeus-cling                0.4.3                         0    QuantStack
xtl                       0.4.8                         0    QuantStack
xz                        5.2.3                         0    conda-forge
zeromq                    4.2.3                   cling_3  [cling]  QuantStack
zlib                      1.2.11                        0    QuantStack

Issue
I was uncertain what to expect from the following part of the readme (in a screenshot)

Omitting the ; in the last statement of a cell results in an output being printed

With the following cell in a notebook:
#include <iostream>
#include <nlohmann/json.hpp>
nlohmann::json j;
j["foo"] = "bar";
j

I received the message "The kernel appears to have died". Is the object not passed to std::cout?
Changing the last line to this works:
std::cout << j << std::endl;

Minor additional question on the cell parsing logic
Please let me know if I should move this to a separate issue or if it's a duplicate. With the following cell:
#include <iostream>
std::cout << "jupyter" << std::endl;
//extra comment at end

I see an additional output due to the trailing comment:
        jupyter
Out[1]: @0x112c6e010

Is that due to the comment? I don't see the additional output otherwise.
Thanks for developing this great notebook kernel! Looking forward to using it!

@mmaz thanks for posting this.

The crash for printing JSON objects seems to come from a bug in upstream cling when calling cling::printValue which is what happens under the hood when displaying objects for which we have not defined a custom renderer.

However, we have a workaround for this, which is to define the rich renderer for JSON
You can do something like
#include "nlohmann/json.hpp"

#include "nlohmann/json.hpp"

namespace nlohmann
{
    json mime_bundle_repr(const json&j)
    {
        json repr;
        repr["application/json"] = j;
        repr["text/plain"] = "A JSON object" //  j.dump(4) is probably too long
        return repr;
    }
}

This will give you a plain text representation in the case of the classic notebook and a nicer interactive tree UI in JupyterLab.

If you can think of a decently sized plain text representation, that would be better than j.dump(4), let me know.
Regarding the trailing comment issue, it is a known issue, which is due to the fact the way we decide on whether the last output should be displayed is very naive... (Essentially removing trailing white spaces and checking for a semicolon).

We are thinking of including this sort of rich mime bundle by default.

@SylvainCorlay Thanks for the the example! Having an explorable tree in JupyterLab is great; way more useful than I was initially expecting.
Thanks also for the explanation regarding the trailing comment, makes sense. Is the current approach regex-based, or is it walking an AST for the cell? It sounds like the former from your explanation, but I wasn't sure after skimming the discussion in #67

We are not at all using the AST for the cell.
I am not sure if the AST would have that information.
The very naive trimming of the last block is done here:
https://github.com/QuantStack/xeus-cling/blob/master/src/xinterpreter.cpp#L181
Although the output variable is a cling Value, which holds both the AST context and the underlying value
https://github.com/root-project/cling/blob/master/include/cling/Interpreter/Value.h#L40
If you know how to extract that information from the cling value, a PR would be very welcome!

Labelling this as an upstream-cling bug because the standard PrintValue mechanism should not choke on nlohmann_json and display the address of the object instead.


Feature request: add ability to set module angles without robot moving

My team uses yagsl, and would like the ability to orient the drive angles without moving, to help precise alignment using April tags.
The SwerveDrive class has a 'lockPose()method that sets the module desired states, and usesforce=truein calls tomodule.setDesiredState()`   But from the outside of SwerveDrive, I don't see a great way to directly set the orientation without a speed above 1%, due to anti-jitter logic.
One fix would be to add an optional 'force' boolean to setModuleStates, and pass that along to setRawModuleStates
  public void setModuleStates(SwerveModuleState[] desiredStates, boolean isOpenLoop) {
    setRawModuleStates(
        kinematics.toSwerveModuleStates(kinematics.toChassisSpeeds(desiredStates)), isOpenLoop, false);
  }
  public void setModuleStates(SwerveModuleState[] desiredStates, boolean isOpenLoop, boolean force) {
    setRawModuleStates(
        kinematics.toSwerveModuleStates(kinematics.toChassisSpeeds(desiredStates)), isOpenLoop, force);
  }

If desired, I could make a PR for this.    My ability to test on a robot is limited, because the team needs the robot for actual driving.

Thanks for the response.   I will use that in code for sure.
It does not seem like the most obvious approach, which is why I still think there is value in an expanded API.   Maybe next year?


Support HTML Like Label Parsing to AST

I have a requirement for more advanced label formatting in a node, e.g.
new Node('id', {
    label: '<<B>BOLD text</B>\n\nsome nice non bold text>'
});

Currently when I try this, the label seems to disappear entirely...
I'm assuming this feature is what is missing to support this - is there any plan to address this issue soon?


Fix ValidPalindrome

Seems your request only changes the parameter name, and the new name is no better than previous one.

@soapyigu  the name change is needed because isAlphanumeric is used in the main method instead of isValid.


better syntax for specifying render method

allow the render method to be specified like this:
  class Foo < React::Component::Base
    render do
      div.foo do
        "hello".span
        "there".span
      end
    end
  end

as an alternative to
  class Foo < React::Component::Base
    def render
      div.foo do
        "hello".span
        "there".span
      end
    end
  end

Why?

render methods are very often longer than the 10 lines recommend by style guides.  Specifying the render method as a block avoids linter (i.e. rubocop) errors.

we can now also specify the container as the first parameter, shortening things up:

class Foo < React::Component::Base
  render(:div, class: :foo) do
    "hello".span
    "there".span
  end
end

an alternative would be to assume a :div unless something else was specified.
If nil was specified then the container would be defined by the block:
class Foo < React::Component::Base
  render do
    "hello".span
    "there".span
  end
end

class FooBar < React::Component::Base
  render(MyComponent, some_param: 1) do
    ...
  end
end

class Fob < React::Component::Base
  render(nil) do
     if do_it_this_way
      div { "i'm a div" }
     else
       "i'm a span".span
     end
  end
end

I don't really see a need for this if the primary reason is because rubocop or similar complains that it's over 10 lines.  That's like saying html, haml, etc. can't be longer than 10 lines, which is kind of silly

I'm not sure I like this syntax. To me it makes the render "magic" and not just another method.
If the problem is rubocop I guess we could add the necessary comments to it to not bother you with it inside the generator

I too dislike adapting code to work around a tool that is intended to help.
Otherwise it should be renamed SkyNET! 🤖
As for the div wrapper I'd like to just point out that Inesita does that adding a div if it sees more than one root. But I'd avoid the inline options (i.e. render(:div, class: :foo)).

@elia - well I am using rubocop as an automation for the ruby style guide. d
Lets put it this way several years ago, if @zetachang had said.... hmmm.... how will I specify what to render.... hmmm... I could just add another hook, called render... I think we would all be happy with that.  (well maybe not) I don't know.
Its all components have to render some element, so why not let that element be specified optionally as a parameter to a hook, instead of this kind of redundant
def render
  div do
    ....
  end
end

its just two extra lines you have to have with every component, and more over most of the time its going to be a div so just default that as well.
The inestia solution is not really helpful it the our case, as we would typically need to pass in params like the class or styles

@loicboutet - I'm not seeing why a render call back hook is any more magic than the before_mount hook


Respect chosen subtitle format when embedding

Please follow the guide below

You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
Put an x into all the boxes [ ] relevant to your issue (like that [x])
Use Preview tab to see how your issue will actually look like

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2017.07.23. If it's not read this FAQ entry and update. Issues with outdated version will be rejected.

[x] I've verified and I assure that I'm running youtube-dl 2017.07.23

Before submitting an issue make sure you have:

[x] At least skimmed through README and most notably FAQ and BUGS sections
[x] Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

[ ] Bug report (encountered problems with youtube-dl)
[ ] Site support request (request for adding support for a new site)
[x] Feature request (request for a new functionality)
[ ] Question
[ ] Other

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:
Add -v flag to your command line you run youtube-dl with, copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):
[debug] System config: []
[debug] User config: ['-f', 'bestvideo[ext=mp4][height<=?1080]+bestaudio[ext=m4a]/best[ext=mp4]/best', '--external-downloader', 'aria2c', '--external-downloader-args', '-x 5', '--hls-use-mpegts', '--write-sub', '--sub-format', 'srt/best', '--convert-subs', 'srt', '--embed-subs']
[debug] Custom config: []
[debug] Command-line args: ['-v', '--geo-bypass-country', 'US', 'http://www.nick.com/the-thundermans/videos/parks-t-rex-s4-ep409-full-episode/']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.07.23
[debug] Python version 3.6.1 - Linux-4.4.0-43-Microsoft-x86_64-with-Ubuntu-17.04-zesty
[debug] exe versions: ffmpeg 3.2.4-1build2, ffprobe 3.2.4-1build2, rtmpdump 2.4
[debug] Proxy map: {}
[debug] Using fake IP <IP_ADDRESS> (US) as X-Forwarded-For.
[nick.com] parks-t-rex-s4-ep409-full-episode: Downloading webpage
[nick.com] 2eb32294-c0ff-11e6-bfd4-0026b9414f30: Downloading info
[nick.com] 46fb9330-9c04-4596-a495-f36728bf22c3: Extracting information
[nick.com] 46fb9330-9c04-4596-a495-f36728bf22c3: Downloading video urls
[nick.com] 46fb9330-9c04-4596-a495-f36728bf22c3: Downloading m3u8 information
[nick.com] 4b62f08d-0175-49a3-a1de-5e339a6f282b: Extracting information
[nick.com] 4b62f08d-0175-49a3-a1de-5e339a6f282b: Downloading video urls
[nick.com] 4b62f08d-0175-49a3-a1de-5e339a6f282b: Downloading m3u8 information
[nick.com] cdbe9383-d8c9-4173-a993-6b8b17d42367: Extracting information
[nick.com] cdbe9383-d8c9-4173-a993-6b8b17d42367: Downloading video urls
[nick.com] cdbe9383-d8c9-4173-a993-6b8b17d42367: Downloading m3u8 information
[nick.com] c168ca93-7010-4a6a-be34-8cdd2c36fe00: Extracting information
[nick.com] c168ca93-7010-4a6a-be34-8cdd2c36fe00: Downloading video urls
[nick.com] c168ca93-7010-4a6a-be34-8cdd2c36fe00: Downloading m3u8 information
[download] Downloading playlist: The Thundermans: "Parks & T-Rex"
[nick.com] playlist The Thundermans: "Parks & T-Rex": Collected 4 video ids (downloading 4 of them)
[download] Downloading video 1 of 4
[info] Writing video subtitles to: The Thundermans - 'Parks & T-Rex' S1-46fb9330-9c04-4596-a495-f36728bf22c3.en.vtt
[debug] Invoking downloader on 'https://cp479184-vh.akamaihd.net/i/mtvnorigin/gsp.alias/mediabus/kids.com/2017/01/25/01/46/24/3fad5cc19537470f94b57fe46bc5655c/1548644/Zf7MRb8DN7_HDTHN409A1_1548644_7110228_,384x216_278,512x288_498,640x360_1028,768x432_1528,960x540_2128,1280x720_3128,1920x1080_5128,.mp4.csmil/index_6_av.m3u8?null=0&id=AgBUAnSWaFZvF8eZdFmXGOFpmw%2fWL0IbFd2WhRi2fP4S6bYcJbEd2Ucm3tXjZ5HHePJIwiOOmMGWvA%3d%3d&hdntl=exp=1500900167~acl=%2fi%2fmtvnorigin%2fgsp.alias%2fmediabus%2fkids.com%2f2017%2f01%2f25%2f01%2f46%2f24%2f3fad5cc19537470f94b57fe46bc5655c%2f1548644%2fZf7MRb8DN7_HDTHN409A1_1548644_7110228_,384x216_278,512x288_498,640x360_1028,768x432_1528,960x540_2128,1280x720_3128,1920x1080_5128,.mp4.csmil%2f*~data=hdntl~hmac=5686d73d4e32ee00346fa31275acef1c2d5052f692762c5bd0eedcc94eaafe2d'
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 47
[download] Destination: The Thundermans - 'Parks & T-Rex' S1-46fb9330-9c04-4596-a495-f36728bf22c3.mp4
[download] 100% of 163.26MiB in 01:16
[debug] ffmpeg command line: ffprobe -show_streams 'file:The Thundermans - '"'"'Parks & T-Rex'"'"' S1-46fb9330-9c04-4596-a495-f36728bf22c3.mp4'
[ffmpeg] Fixing malformed AAC bitstream in "The Thundermans - 'Parks & T-Rex' S1-46fb9330-9c04-4596-a495-f36728bf22c3.mp4"
[debug] ffmpeg command line: ffmpeg -y -i 'file:The Thundermans - '"'"'Parks & T-Rex'"'"' S1-46fb9330-9c04-4596-a495-f36728bf22c3.mp4' -c copy -f mp4 -bsf:a aac_adtstoasc 'file:The Thundermans - '"'"'Parks & T-Rex'"'"' S1-46fb9330-9c04-4596-a495-f36728bf22c3.temp.mp4'
[ffmpeg] Converting subtitles
[debug] ffmpeg command line: ffmpeg -y -i 'file:The Thundermans - '"'"'Parks & T-Rex'"'"' S1-46fb9330-9c04-4596-a495-f36728bf22c3.en.vtt' -f srt 'file:The Thundermans - '"'"'Parks & T-Rex'"'"' S1-46fb9330-9c04-4596-a495-f36728bf22c3.en.srt'
Deleting original file The Thundermans - 'Parks & T-Rex' S1-46fb9330-9c04-4596-a495-f36728bf22c3.en.vtt (pass -k to keep)
[ffmpeg] Embedding subtitles in 'The Thundermans - 'Parks & T-Rex' S1-46fb9330-9c04-4596-a495-f36728bf22c3.mp4'
[debug] ffmpeg command line: ffmpeg -y -i 'file:The Thundermans - '"'"'Parks & T-Rex'"'"' S1-46fb9330-9c04-4596-a495-f36728bf22c3.mp4' -i 'file:The Thundermans - '"'"'Parks & T-Rex'"'"' S1-46fb9330-9c04-4596-a495-f36728bf22c3.en.srt' -map 0 -c copy -map -0:s -c:s mov_text -map 1:0 -metadata:s:s:0 language=eng 'file:The Thundermans - '"'"'Parks & T-Rex'"'"' S1-46fb9330-9c04-4596-a495-f36728bf22c3.temp.mp4'
Deleting original file The Thundermans - 'Parks & T-Rex' S1-46fb9330-9c04-4596-a495-f36728bf22c3.en.srt (pass -k to keep)

Same happens with the other 3 parts.
The output is similar for the other parts as well.

Description of your issue, suggested solution and other information
If I use --embed-subs --convert-subs srt and the audio format is compatible with the video format or both are in a single file, I get MP4 files with TX3G subtitles, rather than MKV with SRT subtitles. --sub-format and --convert-subs should take priority and should be also considered for container format compatibility.

Sorry, I just noticed this is a duplicate of #6996.


Add kusk lint command

As a user I just want to run a command to check if my x-kusk policies are configured correctly, but I don't want to apply it to the cluster just yet.
Can we have a command kusk lint that uses the validation that we already run with kusk deploy?

perhaps kusk validate would be better - linting is something we could support separately!?

Validate will lint as well. But let's focus first on validation