Including open source libraries to support YAML cmdlets

Need guidance on creating YAML cmdlets by including this library.
https://github.com/aaubry/YamlDotNet

@dfinke : Try looking at the Ansible repos from https://github.com/trondhindenes
to see how he handled it as Ansible is YAML

We discussed having a DSC resource to handle some common text files. YAML would fit there too.
https://github.com/PowerShell/DscResources/issues/155

@ArieHein Could you say more about your comment? I've wrapped the YAMLDotNet library in PowerShell so I can do  ConvertFrom-Yaml. Works fine. I was thinking it'd be good to add this to PowerShell directly so these cmdlets are available out of the box.
I was thinking of pulling YAMLDotNet into the PowerShell code, and create a cmdlet for it. I wanted guidance on adding other oss into the core product. If v6.0 ships with Windows, are there legal restrictions? Is there a mechanism to add these types of cmdlets so it ships with the OSS version and not the Windows release? etc

The starting point was adding a DSC resource to handle various text formats. We didnt mention YAML as an option as implementing it would take longer time. Naturally it would be based on the appropriate .net classes and types (were not inventing the wheel).
Trons' repo was something I found along the way when I was learning Ansible and he has created DSC resources to handle YAML, so I figured you might want to take a look if anything looks reasonable to you. Perhaps the core code is already inside PowerShell without bringing additional libraries.

I agree with @dfinke that it would be good to get some guidance on whether it is possible to be able to add this directly in the PowerShell 6 release as this would also help where there are other possible options that could also be included from other OSS projects too

This would be quite valuable.  Yaml is a common format in the wider Microsoft ecosystem, and beyond.  Some common use cases:

Introduce more opportunity for tooling and automation for systems using yaml for configuration
Provide an friendlier (IMHO - at the very least, "another") way to serialize and deserialize data for our own PowerShell tools

Cheers!

This raise the question "What should be available in the core distro and what should be downloadable from the gallery". Historically (before Windows PowerShell v5), it was hard to relay on any modules that are not part of the distro. With https://www.powershellgallery.com/ it became much easier to install and depend on 3rd party modules.

@vors - personally it makes sense for as much to be pushed to gallery as possible and "decoupled" from the core of PowerShell where I've rationalised why in #2136 & #1979
But saying that I feel that any ConvertTo/ConvertFrom cmdlets like the proposed YAML ones should really be bundled into Microsoft.PowerShell.Utility.
But then again that's just my thoughts :wink:

I'd be for decoupling.  Presumably, if it goes into PowerShell itself, it's locked to that and higher versions of PowerShell (and difficult to update outside of that cadence, as you mentioned @kilasuit).
On top of that, presumably an external module could target down-level PowerShell versions.  IMHO this alone, assuming you target PS3 or PS4, would justify a separate module - many of us on the operations side don't have a choice - not everyone runs a farm of cattle unfortunately : )
Cheers!

This is where module metadata in the PowerShell Gallery plays a very important role. Even modules released to the Gallery can be downloaded without having PackageManagement installed using something like this
$file = 'https://www.powershellgallery.com/api/v2/package/ISESteroids/<IP_ADDRESS>'
$Request = [System.Net.WebRequest]::Create($file)
$Request.Timeout = '1000000000'
$URL = $Request.GetResponse()
$Filename = $URL.ResponseUri.OriginalString.Split('/')[-1]
$WC = New-Object -TypeName System.Net.WebClient
$WC.DownloadFile($file,('C:\Files\{0}' -f $Filename))
$WC.Dispose()
$URL.Close()

So things like PowerShellVersion & PowerShellHostName etc should be filled in where possible - else why bother having the metadata available to us in the first place?

Based on the discussion, our resolution is: we will keep YAML parsing capabilities outside of the scope for PowerShell core (inbox modules) in the observable future.
We are encouraging you to use PowerShell gallery as a delivery vehicle for the module.

Just another data point.
AWS CloudFormation Introduces YAML Template Support
https://aws.amazon.com/about-aws/whats-new/2016/09/aws-cloudformation-introduces-yaml-template-support-and-cross-stack-references/

I understand that every feature needs a business case. Currently, YAML is supported by pretty much every language out there, except for PowerShell. When I tell my coworkers that they should be looking into PowerShell on Linux or Mac OS, and then I say they're going to have to add another external library to handle YAML, it's going to seriously hurt PowerShell's position relative to other languages.

If this language is so important and needful (is it really true?), I see three direction to add support it:

Module as external package
New System.Yaml namespace in CoreCLR
Native Yaml cmdlets and native [yaml] (easy after 2.)

@bgshacklett hmm what language supports YAML as a core type ? even python requires 'another external library'. So I cant see how this will 'hurt' PowerShell position.
@iSazonov I dare say there wont be a System.Yaml for the same reason there's no System.JSON (not counting Silverlight) though you have some abilities via Serialization but this is left for the 3rd party to
create their own implementations and not be bound. MS are good at making frameworks with enough hooks for the programmers to build upon and extend and thus create a partner ecosystem.
As @RamblingCookieMonster  said, decoupling is a good thing as you're not bounding the language to a specific implementation.


Change sleep action icon to syndie pyjama

Change sleep and wake actions icons to syndie pyjamas hat (which is more logical than just an egg)

About the PR

Media

[x] I have added screenshots/videos to this PR showcasing its changes ingame, or this PR does not require an ingame showcase

Changelog

:cl: mhamster

tweak: Changed sleep action icon from egg to syndie pyjama

I feel like ZZZ would be the clearest way to signify sleep. But that's just my opinion

this is an existing sprite so no reason not to use it


no-unused-prop-types does not trigger if class does not use any properties

import React from "react";

var Hello = React.createClass({
  propTypes: {
    name: React.PropTypes.string
  },
  render: function() {
    return
Hello Bob
;
  }
});

This is almost identical to the first example in the docs.
$ eslint --no-eslintrc --parser-options="{sourceType: \"module\", ecmaVersion: 6, ecmaFeatures: {jsx:true}}" --plugin react --rule react/no-unused-prop-types:error js/Hello.jsx
# No errors shown.

By contrast, this does trigger:
import React from "react";

var Hello = React.createClass({
  propTypes: {
    name: React.PropTypes.string
  },
  render: function() {
    return
Hello Bob {this.props.foo}
;
  }
});

$ eslint --no-eslintrc --parser-options="{sourceType: \"module\", ecmaVersion: 6, ecmaFeatures: {jsx:true}}" --plugin react --rule react/no-unused-prop-types:error js/Hello.jsx

js/Hello.jsx
  5:11  error  'name' PropType is defined but prop is never used  react/no-unused-prop-types

✖ 1 problem (1 error, 0 warnings)

Both React.createClass and React.PropTypes are officially deprecated as of v15.5; but we should still fix this case.

I will work on it.

@ljharb , do you usually close issues after fix was merged to master or released or you follow some other process?

Ah, when the PR says fixes #x github does it for me :-)' (on merge)
Fixed in #1303.

Got it. Will follow that. Thanks!


Diagnostic script

Checks for common issues and prints a full report that should be very useful for debugging issues on our side.

TIL the built-in process report! This is great @raphael-theriault-swi--I gave it a try in my little test setup but it somehow doesn't think solarwinds-apm is installed, will send you more info in slack.


genhtml: No filename specified

I can find the correct way to set lcov output file to the this github action,
I have this configuration:
   - name: Coverage
      working-directory: ${{github.workspace}}/build
      run: make coverage && ls -l && pwd

    - name: Report code coverage
      uses: zgosalvez/github-actions-report-lcov@v1
      with:
        coverage-files: lcov.coverage.info
        minimum-coverage: 90
        artifact-name: code-coverage-report
        github-token: ${{ secrets.GITHUB_TOKEN }}
        working-directory: ${{github.workspace}}/build

The Coverage stage generate the file lcov.coverage.info on the build directory which is the working directory on the github action, but It can't pick it up.
Could anybody point me to the right direction to fix this?
here is a screenshot of the error and a link to the failing runner

Having the same issue here

Any one was able to fix it?

Not yet?

Is lcov.coverage.info in your ${{github.workspace}}/build directory? If so, you're doing it wrong. The coverage-files is relative to your repository. The working-directory is only used to store the generated HTML report.
https://github.com/zgosalvez/github-actions-report-lcov/blob/7b8cc5c3a10c16f6e4fe9c529a82bfb544158e34/src/main.js#L65
Please let me know if this helps. I will update the readme at least. Thank you for your patience. 🙂

Thanks. That helps 👍

@zgosalvez This works, thanks


Handling of implicit genus and subgenus names

A binomial includes a genus. In some mostly zoological cases also a subgenus name.
Should we extract these implicit taxa into the classification if not given explicitly just as we do for a flat hierarchy?
Having explicit genus and subgenus records allows to detect split taxa by simply doing a duplicate detection.

Yes we should!


Doesn’t account for multiple headings with the same text

Describe the bug
Given a Markdown document with headings containing the same text, for example like this contrived one:
## Heading

## Heading

github-slugger will auto-increment and attach an integer to the end of the generated id. So the above Markdown generates the following HTML:

Heading

Heading

This tool uses the stateless version of github-slugger, so does not expect this behaviour and will fail for links to #heading-1 in this case:
To Reproduce

Create a document with headings containing the same text.
Link to one of the later headings (e.g. with a -1 suffix).
Run the link validator.
It will error and flag the -1 link.

Expected behavior
Link anchor slugging logic should match the Astro project’s logic and the validation should pass.
How often does this bug happen?
Every time
System Info
Observed on macOS, but I suspect that’s not relevant.
Additional Context
No response

The fix has been published in v0.4.1.


Keepalived 2.0.10 + vlan handling mechanism

Following a rework of kube-keepalived-vip for a customer, we are contributing back the modifications.
Upgrade to Keepalived 2.0.10 ==> According to keepalived.org, versions 2.x are the go-to versions where bugfixes and improvements happens (no backport).

Custom interface handling ==> Ability to use an other interface rather than the internalIP/externalIP interface.

Vlan handling. ==> Ability to create vlan tagged interfaces complete with subnet mask and gateway route.

This is my first contribution back to an open-source projet, i hope i wrote here everything needed, in any case, 2 additional sections exist in the README (Custom interface and Advanced configuration) explaining what happens.

/assign @bprashanth


Add zazu-dev-doc plugin to zazu plugin directory

Changed plugin doc file title

Changed plugin doc file title


Create readme.md

Company info added after discussion with Florian, Christian and Hans-Henry

Adding a company also requires a logo (PNG) and an entry inside companies.json.


Add size properties collector

Collect size properties from CF_DEFAULT and CF_WRITE.
Don't merge until https://github.com/pingcap/tikv/pull/2096 has been merged.

PTAL @zhangjinpeng1987 @BusyJay

PTAL @zhangjinpeng1987 @BusyJay

PTAL @zhangjinpeng1987 @BusyJay

LGTM

PTAL @zhangjinpeng1987

LGTM


Fix #183

Summary
Fix #183
Enabled state for the view for some reason cause an issue with restoring screens.
As it's not necessary to save view state it will be disabled at the moment.
Test Plan
https://github.com/callstack/react-native-viewpager/issues/183#issuecomment-647553382

Good job 🎉

Good job 🎉


error build/adminguide/routing-is.html

Whenever I try to build this book lately, I get error  build/adminguide/securing-traffic.html or similar errors.
When running with --verbose, it looks as follows:
error  build/adminguide/securing-traffic.html
Is a directory @ io_fread - /app/docs-book-cloudfoundry/output/master_middleman/source/adminguide/

Am I the only one having this issue? I've tried on multiple systems and in Docker containers but it's always the same...

I think this is an issue with Bookbinder because when I try bind local, it works. Closing this one.


plotting bug with plot_components with EEG montage

I am currently working with some biosemi data and I noticed one bug and one enhancement that should be addressed before the release.
the first is the bug: when plotting the IC for the EEG data, only one topo plot is currently being plotted while the rest are left empty.

the second is a feature enhancement: when using ica.plot_components, the plotting function should not default to meg especially when there are no meg channels present in the data. the default ch_type should be None. we should inspect the channel types, and if there are no meg channels, we should plot the EEG, if we have meg only, then plot meg, then if we have both, probably the default should be meg but that can be discussed.

bug fix yes, enhancement time is over. please noe new ENH requests.

@teonlamont I'm plotting ICs for EEG data almost every day. Can you share a snippet of code to reproduce this?

it's a backend pb I suspect. Try starting ipython with --gui=qt or
something along these lines

how do you do that from a PC...? lol

@teonlamont what do you mean? On a PC ipython is supposed to work.

with the flag --gui=qt. there's a separate application that is loaded for ipython

@teonlamont with anaconda there should be a linux terminal emulator.

ok, @dengemann, I will make a separate issue for the ch_type. i think it would be a broader plotting feature to look at for any plot that requires a ch_type as input.

i think it would be a broader plotting feature to look at for any plot that requires a ch_type as input.

yes, I suggested that already a few times during seprate discussions. We should implement it once, carefully and consistently.

@dengemann I will give it a look tomorrow and report back wrt to the linux terminal emulator. I just left the lab. I didn't know if it was our problem or a backend one.

yes, I suggested that already a few times during seprate discussions. We should implement it once, carefully and consistently.

yeah, I remembered it being discussed and i had thought it had been addressed.

open an issue and we'll re-discuss it.


Failed to load Info.plist

When I try to build an app with OpenCV with Xcode, the build succeeds, but the app couldn't be installed, and then such window pops up

If I click Details button, then I get such error
This app could not be installed at this time.
Domain: IXUserPresentableErrorDomain
Code: 1
Failure Reason: Could not install at this time.
Recovery Suggestion: Failed to load Info.plist from bundle at path /Users/mpodlesny/Library/Developer/CoreSimulator/Devices/4D35945D-A969-4578-9BFB-36C284AE832F/data/Library/Caches/com.apple.mobile.installd.staging/temp.B3q0wJ/extracted/FirstReactNativeApp.app/Frameworks/opencv2.framework
Failed to load Info.plist from bundle at path /Users/mpodlesny/Library/Developer/CoreSimulator/Devices/4D35945D-A969-4578-9BFB-36C284AE832F/data/Library/Caches/com.apple.mobile.installd.staging/temp.B3q0wJ/extracted/FirstReactNativeApp.app/Frameworks/opencv2.framework
Domain: MIInstallerErrorDomain
Code: 35
User Info: {
FunctionName = "-[MIBundle _validateWithError:]";
LegacyErrorString = PackageInspectionFailed;
SourceFileLine = 39;
}
Please say how can I overcome this problem. Thanks in advance!

I solved the problem. When you add opencv2.framework to Frameworks, Libraries and Embedded Content set Embed option to Do Not Embed and everything gonna be OK


type 'String' is not a subtype of type 'Enum' in type cast

Error:
Unhandled exception:
type 'String' is not a subtype of type 'TestEnum' in type cast
#0      _safeKeyFromJson (package:fast_immutable_collections/src/imap/imap.dart:1444:14)
#1      new IMap.fromJson.<anonymous closure> (package:fast_immutable_collections/src/imap/imap.dart:414:50)
#2      MapMixin.map (dart:collection/maps.dart:170:28)
#3      new IMap.fromJson (package:fast_immutable_collections/src/imap/imap.dart:413:12)
#4      _$TestClassFromJson (package:bug/main.g.dart:10:18)
#5      new TestClass.fromJson (package:bug/main.dart:25:7)
#6      main (package:bug/main.dart:9:13)

Minimal code:
dependencies:
  flutter:
    sdk: flutter

  fast_immutable_collections: ^7.3.1
  json_annotation: ^4.5.0

dev_dependencies:
  flutter_test:
    sdk: flutter

  build_runner: ^2.1.11
  json_serializable: ^6.2.0

import 'dart:convert';

import 'package:fast_immutable_collections/fast_immutable_collections.dart';
import 'package:json_annotation/json_annotation.dart';

part 'main.g.dart';

void main() {
  TestClass.fromJson(jsonDecode(jsonEncode(const TestClass(
    testField: IMapConst({
      TestEnum.testValue: true,
    }),
  ))));
}

@JsonSerializable()
class TestClass {
  final IMap<TestEnum, bool> testField;

  const TestClass({
    required this.testField,
  });

  factory TestClass.fromJson(Map<String, Object?> json) =>
      _$TestClassFromJson(json);

  Map<String, Object?> toJson() => _$TestClassToJson(this);
}

enum TestEnum {
  testValue,
}

I was just coming in to report this, thanks.  My map's key is an enum as well, so pretty much the exact case.

Hello! Thank you for the report. JSON serialization and deserialization was kindly contributed by @knaeckeKami . Maybe he can help you. Or if you know how to fix this, I'm accepting PRs.
The code that needs to be improved is at the very end of this file:
https://github.com/marcglasberg/fast_immutable_collections/blob/master/lib/src/imap/imap.dart
Meanwhile, I have published version 7.4.1, which improves the error message. It will now be an unsupported error: JSON deserialization of IMap keys of type $type are not supported at the moment.
Also, created a test for that, at the very end of this file:
https://github.com/marcglasberg/fast_immutable_collections/blob/master/json_serializable_e2e_test/test/json_serialization_and_deserialization_test.dart

Hi @marcglasberg
All keys in JSON are strings, so I don't understand why this code is here
Also, why do the _safeKeyFromJson and _safeKeyToJson methods exist?
https://github.com/marcglasberg/fast_immutable_collections/blob/d5aca34b39e697edfedb5245abde013e7733f67e/lib/src/imap/imap.dart#L407-L415

Hm, I don't see a way how this could be handled without a complete overhaul of serialization.
But you can try using json_serializable_fic (https://pub.dev/packages/json_serializable_fic ). It says discontinued because I thought with the addition of the toJson, fromJson methods this package should be obsolete, but I should still work.
There is a test for enum maps (https://github.com/knaeckeKami/json_serializable_immutable_collections/blob/master/builders/json_serializable_fic/test/integration/model_test.dart ).

Hm, actually, I just thought of a way to handle this, but it's very, very hacky. I will try to implement it during this week.


Unroll option in recurrent layers

Forgive me if this has come up before.
It's common in other NN libraries to unroll recurrent layers whenever possible, replacing a theano.scan with bunch of non-scan computations. This can significantly speed up the network, though often at the cost of memory. In general, Theano advises against using scans whenever possible, as performance is greatly hindered on the GPU.
Should we unroll scans in the recurrent layers when we can? One option would to borrow Lasagne's unroll_scan utility function, which conveniently has the same options as the theano.scan option.
Pros:

Potentially much faster recurrent computations (my preliminary experiment saw the training time of one epoch go from ~1300s to ~1050s, which adds up quickly)

Cons

No more truncate_gradient option (does anyone use this?)
Only possible when input_length is provided
More confusing code base

This seems super cool!!! Does it compile faster too? Let me try it.

Pretty cool stuff, I wasn't aware of this.

truncate_gradient (does anyone use this?)

I've used it before. It can be useful in some specific setup. Removing it wouldn't be a huge loss though.

Only possible when input_length is provided

That's a reasonable restriction.

More confusing code base

I find it surprisingly clean!

This unroll scan could be moved to Theano. It was on our todo list, but we
didn't got the time for this. If someone want to help, it would be great.
On Thu, Nov 5, 2015 at 10:17 PM, François Chollet<EMAIL_ADDRESS>wrote:

Pretty cool stuff, I wasn't aware of this.
truncate_gradient (does anyone use this?)
I've used it before. It can be useful in some specific setup. Removing it
wouldn't be a huge loss though.
Only possible when input_length is provided
That's a reasonable restriction.
More confusing code base
I find it surprisingly clean!
—
Reply to this email directly or view it on GitHub
https://github.com/fchollet/keras/issues/954#issuecomment-154272961.

I'm going to copy it into Keras and integrated into the RNNs for now, but it would be great to see it moved upstream.
It's worth noting that it has a slightly different call structure than theano.scan (namely, outputs_info must be a list, and it returns a list, even if it's a singleton). This could be easily amended though.

@stephenroller how much longer does "unrolled" version compile?

My current NN has an input_length of 10, which is super small. So far it hasn't been noticeable for me. Maybe an extra 5-10s of compile time, on top of what's already ~50s? It should grow substantially with input_length. Right now the ~20% time saved in training time seems well worth it to me though (one takes a minute or so, the other takes hours), but the trade-off is definitely there.

I was just curious, for long runs it won't matter and for prototyping you can use normal scan.

truncate_gradient gone for good here? https://github.com/fchollet/keras/commit/47ed18a3af8be20dce91286a331d4671074ee0ca

Can anyone please clarify the conclusion of this? And how to use truncate_gradient option (to perform Truncated BPTT) in keras? truncate_gradient in theano should have something equivalent for keras, right? I would really appreciate some help here.


Added "Learn Microservices with Spring Boot" book

What does this PR do?
Add resource(s) | Remove resource(s) | Add info | Improve repo
For resources
Description
Added "Learn Microservices with Spring Boot" book
Why is this valuable (or not)?
Learn Microservices with Spring Boot
A Practical Approach to RESTful Services using RabbitMQ, Eureka, Ribbon, Zuul and Cucumber
How do we know it's really free?
Free PDF
For book lists, is it a book? For course lists, is it a course? etc.
Book
Checklist:

[x] Read our contributing guidelines.
[x] Search for duplicates.
[x] Include author(s) and platform where appropriate.
[x] Put lists in alphabetical order, correct spacing.
[x] Add needed indications (PDF, access notes, under construction).
[x] Used an informative name for this pull request.

Follow-up

Check the status of GitHub Actions and resolve any reported warnings!

This appears not to be an authorized distribution. Please see docs/CONTRIBUTING.md guidelines.


Updating onboarding script to account for possible null user.info.

What's this PR do?
This PR adds a fix to account for the possibility that the user.info returned from Northstar might have responded with an error and thus the value is null. If so, let's not rely on having user.info.id be present and instead handle that a little better.
How should this be reviewed?
If you get a null response from Northstar (you can try and fake it for testing), do the Onboarding experience still show for a logged in user?
Any background context you want to provide?
Used Object.keys() since we're also added the campaign run to the localStorage key. However, these run ids are nested within objects whose keys are the language codes available for a translated run. To help avoid errors with adding the campaign run ID, I'm nabbing the first item with a key (typically en) and grabbing that run ID.
Relevant tickets
Fixes #6884
Checklist

[ ] Tested on staging.

@DFurnes @deadlybutter
cc: @jessleenyc

@deadlybutter decided to best go the simple route. good call. we can wait and see if campaign runs are necessary if Onboarding is a feature that sticks ;)

sweets
🚂 🚋 🚋


reusing to revive a CNC machine

Hey,
I have a relatively big CNC machine in the basement, all the hardware is present but I've given up on configuring it correctly with linuxcnc.
The drivers are Leadshine Microstep m542 s
They already have some neat cables so I was thinking all I would have to do to make this work is change some pin numbers.
Will pyCNC automatically detect that I don't have a heatbed, etc.? How does calibration work, to make sure it doesn't run out of bounds?

PyCNC knows only pin number which should be triggered if command for heatbed is provided. So calibration and anything else should work without heatbed.
Calibration simply moves steppers slowly and check if endpoint switch was triggered. If so, stops moving. Endpoint switches pin can be configured in a config.


chore: remove unused and improve contributor docs

:tada: This PR is included in version 1.2.9 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Socks5 proxy setting and subcommands tab completion.

I'm not sure this tool supports the following features:

According to the document, it seems that only http[s] proxies are supported:

HTTP_PROXY, HTTPS_PROXY, ALL_PROXY: The proxy to use for all HTTP/HTTPS requests.

So, I would like to know whether it can also support socks5 proxy, say, socks5://<IP_ADDRESS>:16668

Subcommands tab completion.

Regards,
Zhao

We do support tab completion. The instructions are basically the same as in Ruff: https://docs.astral.sh/ruff/configuration/#shell-autocompletion.

I don't think we support SOCKS5 at all (we don't have the reqwest crate feature turned on). I don't think they support a SOCKS5_PROXY environment variable, but if you're interested in it I'd check in upstream.

If you are willing to consider, both SOCKS5 and SOCKS5H should be supported.

Can you share more about the use case?

My native proxy is of type Socks5, and in my specific region, using such a proxy to download dependency packages is more efficient.


Is there a reason not to use the dbt-labs/codegen macros?

It seems that some of the code in dbt-invoke could be replaced with calls to the macros in dbt-labs/codegen? This would centralize the logic for creating the properties files into a single place (dbt-labs/codegen) and allow for additional features as they are added upstream.

Closing due to inactivity. Happy to reopen if there is further input.


Gulp is broken in v7.7.4

I ran a node script that had only one line:
var gulp = require('gulp');

What were you expecting to happen?
For it to not crash
What actually happened?
It crashed with:
Error: Cannot find module 'internal/fs'
    at Function.Module._resolveFilename (module.js:470:15)
    at Function.Module._load (module.js:418:25)
    at Module.require (module.js:498:17)
    at require (internal/module.js:20:19)
    at evalmachine.<anonymous>:18:20
    at Object.<anonymous> (/project/node_modules/vinyl-fs/node_modules/graceful-fs/fs.js:11:1)
    at Module._compile (module.js:571:32)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:488:32)
    at tryModuleLoad (module.js:447:12)

Please post a sample of your gulpfile (preferably reduced to just the bit that's not working)
(posted above)
What version of gulp are you using?
3.9.1
What versions of npm and node are you using?
$ node -v
v7.7.4

Can you try running npm update and npm rebuild?

3.x doesn't work in node 7. Don't use unstable versions of node.

node 7 has been officially released since last October. It's not a LTS release, but not "unstable".


Pino Instrumentation not working with @opentelemetry/sdk-trace-base

What version of OpenTelemetry are you using?
1.6.0
What version of Node are you using?
v20.5.0
What did you do?

Tried Integrating @opentelemetry/instrumentation-pino for traces context injection in my fastify app with code:

const { PinoInstrumentation } = require('@opentelemetry/instrumentation-pino')
const { registerInstrumentations } = require('@opentelemetry/instrumentation')

// Register the Pino instrumentation
registerInstrumentations({
    instrumentations: [new PinoInstrumentation()]
})

const logLevel = process.env.LOG_LEVEL ?? 'debug'
const env = process.env.NODE_ENV ?? 'development'

const baseLogOptions = {
    level: logLevel,
    formatters: {
        level(label) {
            return { level: label }
        }
    }
}

if (env !== 'production') {
    baseLogOptions.transport = {
        target: 'pino-pretty',
        options: {
            colorize: true,
            translateTime: 'yyyy-mm-dd HH:MM:ss'
        }
    }
}

const pino = require('pino')

const logger = pino(baseLogOptions)

module.exports = logger

TraceProvider Initialization using @opentelemetry/sdk-trace-base precedes the above code:
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc'
import { BasicTracerProvider, BatchSpanProcessor } from '@opentelemetry/sdk-trace-base'

const traceExporter = new OTLPTraceExporter({
            concurrencyLimit: 10,
            compression: 'gzip'
        })

const provider = new BasicTracerProvider()

spanProcessor = new BatchSpanProcessor(traceExporter)

provider.addSpanProcessor(spanProcessor)
provider.register()

What did you expect to see?

Successful injection of trace_id & span_id in logs

What did you see instead?

No trace_id & span_id
Log messages:

opentelemetry/api: Registered a global for diag v1.6.0.
@opentelemetry/api: Registered a global for trace v1.6.0.
@opentelemetry/api: Registered a global for propagation v1.6.0.
@opentelemetry/instrumentation-pino Module pino has been loaded before @opentelemetry/instrumentation-pino so it might not work, please initialize it before requiring pino
Applying patch for<EMAIL_ADDRESS>[2023-10-20 08:50:41] INFO (90804): Server listening at http://[::1]:3000
[2023-10-20 08:50:41] INFO (90804): Server listening at http://<IP_ADDRESS>:3000

Additional context

Pino is loaded at only 1 place (1st code snippet)
Does pino-instrumentation works only with NodeTracerProvider or with any TraceProvider?

The BasicTracerProvider is not intended to be used by end users. It's just the common base for web and node.
NodeTracerProvider adds a node.js specific context manager (AsyncHooks or AsyncLocalStore based depending on node.js version in use).
The BasicTracerProvider installs no context manger and as a result the NoopContextManager is used which always returns ROOT_CONTEXT.
pino instrumentation (and a lot other instrumentations) get the current span from current context via ContextManager. As ROOT_CONTEXT holds no span pino instrumentation doesn't add traceId/spanId.
So in short if you want to use an instrumentation install a context manager.
If you create spans by manually calling tracer.startSpan() and pass the span/context around programmatically between all these tracer.startSpan() calls and you enrich the logs also programmatically (because you have the current span in hand anyway) no ContextManager is needed.

Thank you so much for the clear explanation @Flarna
It was really helpful.
I manually create spans using tracer.startSpan() to avoid the load of auto-instrumentation.
Can we still get the benefits of ContextManager with manual instrumentation?
Is there a base documentation that clearly explains these constructs for a person with limited experience?

Can we still get the benefits of ContextManager with manual instrumentation?

ContextManger doesn't care about who/how a span is created. But the creator of the span is also responsible to set is active span matching to the application logic using bind() and/or with()  APIs along with setSpan() or by using the "all in one helper" startActiveSpan().
In fact if you want to use pino instrumentation along with your manual spans you have to use ContextManager as this is the interface between your code and the instrumentation.

Is there a base documentation that clearly explains these constructs for a person with limited experience?

I fear that we are far away of perfect docs, in special for persons with limited experience. Help to improve this is welcome. OTel (or more general monitoring) is not a simple topic. In special the context propagation topic is complicated, language dependent and hard to understand/do right. I doubt simple/easy answers will arrive soon.

Any recommendations to debug this problem?

Review startup code of your application.
Consider to move OTel initialization code into a dedicated file and preload it using the -r command line option.
Set a breakpoint at the beginning of the very first pino file (pino.js) and look at the callstack

Thank you so much for all the help @Flarna
Once I understand these constructs a little better, I will do my part by updating the docs to make it friendlier for people with limited open telemetry experience.

@Flarna with your guidance & help, I was finally able to configure traces & logs
Following is my reference documentation for the same.

Could you please do a quick review of it?
Do suggest any recommendations to improve the implementation further?

Thank you so much for all the help 🙏
OpenTelemetry

OpenTelemetry

Traces

Configuration Steps

Create a Trace Provider
Configure Trace Exporter
Registering Instrumentations
Pre Loading Instrumentation Libraries

Logs

Pino Log Transport to Log Exporter
Pino Logs Transformation to OTLP Log Format

Connecting Traces & Logs
Debugging Traces & Logs

Traces

Traces provides mechanism to understand the flow of execution of a process or workflow

Tracing is configured in the module traceConfig.cjs.

Configuration Steps
It's a 3 step process:

Create a Trace Provider
Configure a Trace Exporter
Register Instrumentations

Create a Trace Provider

Recommeded trace provider for any node app is NodeTraceProvider
It is created with a Resource and ContextManager

Resource identifies the service
Context Manager is responsible for Trace Context Management (trace_id, span_id) for each request

It gives tracers() which are then used to generate trace context for a request

const resource = Resource.default().merge(
    new Resource({
        [SemanticResourceAttributes.SERVICE_NAME]: env.serviceName(),
        [SemanticResourceAttributes.SERVICE_VERSION]: env.appVersion(),
        [SemanticResourceAttributes.DEPLOYMENT_ENVIRONMENT]: env.envName()
    })
);

const contextManager = new AsyncLocalStorageContextManager().enable();

provider = new NodeTracerProvider({
    resource: resource,
    contextManager: contextManager
});

// A. Exporter Code Follows

provider.register();

// B. Instrumentation Code Follows

Configure Trace Exporter

The traces generated above could be exported to a backend or collector using a exporter
We have used OTLPTraceExporter as it is vendor agnostic
It can exports traces to:

Directly to the Backend - eg. New Relic
A Collector Service - eg. OpenTelemetry Collector in Kubernetes
Refer diagram to get an idea

For both cases, NO changes are required from in code
Set the OTEL_EXPORTER_OTLP_ENDPOINT environment variable to backend or collector service url appropriately

// A. Exporter Code Follows
    const traceExporter = new OTLPTraceExporter({
        concurrencyLimit: 10,
        compression: 'gzip'
    })

    spanProcessor = new BatchSpanProcessor(traceExporter, {
        maxQueueSize: 1000, // The maximum queue size. After the size is reached spans are dropped.
        scheduledDelayMillis: 1000 // The interval between two consecutive exports
    })

    provider.addSpanProcessor(spanProcessor)

Registering Instrumentations

We could instrument traces manually or automatically

Manual instrumentation needs following steps done programtically:

Trace Creation using startActiveSpan() or startSpan() with bind() or with() Apis
Trace Context Management
Trace Propagation

Auto-Instrumentation is a simplar alternative:

It is done using registerInstrumentations method in traceConfig.cjs

// B. Instrumentation Code Follows
registerInstrumentations({
        instrumentations: [
            new PinoInstrumentation(),
            new HttpInstrumentation({
                requestHook: (span, request) => {
                    span.updateName(`${request.method} ${request.url}`)
                    span.setAttribute('request.uuid', request.id)
                    span.setAttribute(SemanticAttributes.HTTP_USER_AGENT, request.headers['user-agent'])
                    span.setAttribute(SemanticAttributes.HTTP_HOST, request.hostname)
                },
                responseHook: (span, response) => {
                    span.setAttribute(SemanticAttributes.HTTP_STATUS_CODE, response.statusCode)
                },
                headersToSpanAttributes: ['user-agent', 'http.url', 'http.method', 'http.host', 'http.client_ip']
            }),
            new FastifyInstrumentation()
        ]
    })

Following Instrumentations are used:

HttpInstrumentation:

It adds tracing record with following information - span_id, trace_id & trace_flags to each incoming request
Additional Attributes are added to span as well in its requestHook & responseHook

PinoInstrumentation:

Pino is the default logger used in Fastify
This instrumentation adds context of a request to the corresponding log statement

Pre Loading Instrumentation Libraries

Instrumentation libraries work only if they are pre-loaded before importing the attaching libary eg. PinoInstrumentation attachs to pino. So, it is essential to ensure that this instrumentation is loaded before pino is imported
To ensure this, tracerConfig.cjs is loaded as a module in npm scripts

    "scripts": {
        "dev": "NODE_ENV=dev nodemon --inspect=<IP_ADDRESS>:9229 -r ./app/utils/otel/traceConfig.cjs app/index.js"
    },

Note the extension of file is .cjs and not .js.
While the default module system for app is ES Module, Pre-loaded modules are only supported for CommonJS modules
So, this & it's dependent modules are converted to CommonJS Module format

Logs

Structures logs are also called Events

pino which is the default logging library for fastify, by default supports JSON logs

These logs need to be forwarded to a backend

This could be done in multiple ways. eg.

Save logs to a file & use Fluentd or LogStash to forward it to a backend or collector

A log exporter is used here in otel/logs.js

    const loggerExporter = new OTLPLogExporter()

    const resource = Resource.default().merge(
        new Resource({
            [SemanticResourceAttributes.SERVICE_NAME]: env.serviceName(),
            [SemanticResourceAttributes.SERVICE_VERSION]: env.appVersion(),
            [SemanticResourceAttributes.DEPLOYMENT_ENVIRONMENT]: env.envName()
        })
    )

    // To start a logger, you first need to initialize the Logger provider.
    loggerProvider = new LoggerProvider({
        resource: resource
    })

    // Add a processor to export log record
    loggerProvider.addLogRecordProcessor(new BatchLogRecordProcessor(loggerExporter))

    // logging
    otelLogger = loggerProvider.getLogger('fastify-logger', env.appVersion())

OTLPLogExporter is vendor agnostic & can send logs to a collector or a backend like New Relic

Connecting this exporter with pino is a 2 steps process:

Transporting Logs to Log Exporter
Transforming Logs

Pino Log Transport to Log Exporter

pino supports transport option to forward logs to a backend
To transport logs to OTLPLogExporter, a custom transport is created in app/utils/otel/pinoTransport.js using pino-abstract-transport

import build from 'pino-abstract-transport'

// Otel logger Configuration as shown above...
otelLogger = loggerProvider.getLogger('fastify-logger', env.appVersion())

// Creating a transport for pino
export default async function (_opts) {
    return build(async source => {
        for await (const logObject of source) {
            otelLogger.emit(mapPinoLogsToOtelLogRecord(logObject))
        }
    })
}

The above pino transport is then integrated in pino logger in app/utils/logger.cjs

const transportTargets = []

if (env.isOtelEnabled()) {
    // Add the otelTransport to the transportTargets array
    transportTargets.push({
        target: './otel/logs.js',
        level: env.logLevel()
    })
}

const transport = pino.transport({
    targets: transportTargets
})

With above code, logs would be send to OTLPLogExporter

Pino Logs Transformation to OTLP Log Format

Backends expect the log in format as defined in OTLP Log Protocol
This means pino logs needs to be transformed into this format at the time of export
This is done in app/utils/otels/logs.js using mapPinoLogsToOtelLogRecord which is configured in log transport function created above

// Pino Log Transport
export default async function (_opts) {
    return build(async source => {
        for await (const logObject of source) {
            otelLogger.emit(mapPinoLogsToOtelLogRecord(logObject))
        }
    })
}

function mapPinoLogsToOtelLogRecord(pinoData) {
    // Transformation Code

    // OTLP Log Record Format
    const logRecord = {
        timestamp: time,
        severityNumber: severityNumber,
        severityText: severityText,
        body: message,
        attributes: {
            ...serializedAttributes,
            level // used by NewRelic to filter logs by level
        }
    }

    return logRecord
}

Connecting Traces & Logs

Traces & Logs are connected using trace_id & span_id - the trace context

HttpInstrumentation automatically adds trace context to each incoming request
PinoInstrumentation adds this context to each log statement

But New Relic does not support this out of the box

To connect traces with logs, sync is required in following configuration variable:

service.name - Should be same as service.name in traces. Done using Resource object in LoggerProvider & NodeTraceProvider
const resource = Resource.default().merge(
    new Resource({
        [SemanticResourceAttributes.SERVICE_NAME]: env.serviceName(),
    })
)

trace.id & span.id in logs. OTLPExporter adds trace_id & span_id to each log statement

Covert it into expected fields in mapPinoLogsToOtelLogRecord function

// Following Configuration required to connect traces with logs in NewRelic
if (attributes.span_id && env.isOtelBackendNewRelic()) {
    serializedAttributes['span.id'] = attributes.span_id
    serializedAttributes['trace.id'] = attributes.trace_id
}

Post the above changes, traces & logs would be connected in New Relic UI

@varmehta are you planning on adding this info to the documentation? I think this could help other users in the same situation.

@dyladan I surely can do that, but which documentation?
As the above documents impact multiple packages.

If you use the NodeTracerProvider it's not needed to created the AsyncLocalStorageContextManager and pass it as argument because the NodeTracerProvider does this internally on default.
Manually creating it is only needed if you use the BasicTracerProvider.

@Flarna thank you. I'll remove that.

Can anyone assigned this task to @varmehta since this person already provided the detailed documentation above. It's a waste of time for other contributors to read everything in this issue and realized it's about to be solved. Thank you!


PubSubMessage ContentType is always application/cloudevents+json (even if payload is not cloudevent)

/area runtime
0.11.3
When sending a payload to pub sub that is not a cloud event I expect no change of content-type.
When sending a payload application/json (service bus explorer) the app endpoint is invoked with the payload but with content-type "application/cloudevents+json".
Seems like PubSubMessage is always of content-type application/cloudevents+json regardless of payload.

I'd hesitate a guess that this change is required. @youngbupark if you agree I can quickly open a PR for this?

Comparing the HTTP publish to the gRPC publish impl - the HTTP publish assumes JSON whereas the gRPC implementation handles text/plain and application/json - is it worth making them consistent too?

Related: components-contrib/506, components-contrib/509

This is by design

This is by design


river.state.last.active_begin not working as expected

I'm trying to build an incremental index by "tailing" a table, but what seems to be happening is that the river is using the current time for the river.state.last.active_begin variable, rather than the time the river last ran.  I have the following config:
{
    "type" : "jdbc",
    "jdbc" : {
        "url" : "jdbc:mysql://<IP_ADDRESS>:3306/xxx",
        "user" : "xxx",
        "password" : "xxx",
        "sql" : [
            {
                "statement" : "select * from XXX where REQUESTDTS > ?",
                "parameter" : [ "$river.state.last_active_begin" ]
            }
        ],
        "index" : "xxx_index",
        "type" : "xxx_type",
        "schedule" : "0 0-59 0-23 ? * *"
    }
}

But viewing my mysql-query.log I can see that the ? is being replaced by the current timestamp, not the time the river last ran.  This of course means that it's looking for data that hasn't been added yet, and isn't seeing any data that was added in the last interval.
Am I configuring this wrong?

Even I'm facing the same issue. river.state.last.active_begin is not working as expected.

Team, can someone help out on this.

@jprante could you help us out on this issue. As all the scheduled rivers are not working hence resulting in not getting new data into Elasticsearch. Do suggest us if there is any alternative to overcome this issue. Your early help is highly appreciated.

The schedules are working, i.e. the river runs on the scheduled intervals.  The problem is that it doesn't pull new data.  The other issue I had is that I was hoping that river.state.last.active_begin would be zero on first execution of a new river, so that it would get all data up to that point, and then start importing incremental data.  It seems to always be the current timestamp, which obviously will never work.

Yes schedules are working. But the problem is only with river.state.last.active_begin, as we get the data which is greater than the last river began which is infact not working. Hence new data is not being synced to Elasticsearch.
Hoping anyone from team would provide a solution / alternative for this.

The river.state variable issue will be fixed in the next days.  This is a spare time project, please understand the time I can spend on issues is very limited. Patches, bugfixes, etc. are warmly welcome.

I think the issue is that in scr/main/java/org/xbib/elasticsearch/river/jdbc/strategy/simple/SimpleRiverFlow.java at line 193 the code:
	riverContext.setRiverState(riverState.setCounter(counter)
                .setLastActive(currentTime != null ? currentTime : new DateTime(0), null)
                .setCurrentActive(new DateTime()));

is always going to set the lastActive to the currentTime in the beforeFetch.  It seems like this should be in the afterFetch function, and that beforeFetch should maybe be using riverContext.getRiverState().getLastActiveBegin() instead of riverContext.getRiverState().getCurrentActiveBegin(). I don't see currentTime used anywhere else in the beforeFetch function, so you can probably just replace this, but I may be missing something here.  I am not confident enough in my understanding of the way the river works to try and make the changes myself, sorry about that.
Maybe try replacing this:
	DateTime currentTime = riverContext.getRiverState().getCurrentActiveBegin();
        riverContext.setRiverState(riverState.setCounter(counter)
                .setLastActive(currentTime != null ? currentTime : new DateTime(0), null)
                .setCurrentActive(new DateTime()));

with this:
	DateTime lastActiveBeginTime = riverContext.getRiverState().getLastActiveBegin();
        riverContext.setRiverState(riverState.setCounter(counter)
                .setLastActive(lastActiveBeginTime != null ? lastActiveBeginTime : new DateTime(0), null)
                .setCurrentActive(new DateTime()));

This is new code from a contribution which is unreleased yet https://github.com/jprante/elasticsearch-river-jdbc/commit/9a2fdba56db361d0ae29ce56cfdfdb9d8abc30dc and where I hope this fixes some issues. I'm not sure it works in all cases. More eyeballs on the code are welcome too :)

I don't think the code in that contribution will work.  It looks like it will always set the lastActive to the current time in the beforeFetch.

I tested this out locally.  With the changes I suggested above it seems to work as expected.  One thing to note is that each time the river starts it will go back to the beginning of time and get everything in the table, the riverState lastActiveBegin is does not appear to be persisted anywhere, this could be a problem for users with lots of older data.

@soacom The currentTime is maybe misunderstood here. currentTime is the time got from the last currentActiveBegin state, not the current time. We could rename it to be more explicit. (s/currentTime/currentActive/ for example).
See #489 , which is working for me.
CurrentActiveBegin is an intermediate state just made to initialize the LastActiveBegin that is set in the afterFetch too (I don't get why btw).
So basically we have :
1st run :
currentActiveBegin = null => last_active_begin set to 0 which implies a first dump of the table.
currentActiveBegin set to new DateTime()
next run:
currentActiveBegin not null => last_active_begin  set to the last currentActiveBegin
currentActiveBegin set to new DateTime()
etc...

we can use this
{
"type": "jdbc",
"jdbc": {
"driver": "com.mysql.jdbc.Driver",
"url": "jdbc:mysql://localhost:3306/myspringjdbcdb",
"user": "root",
"password": "root",
"sql": [
{
"statement": "SELECT id AS _id ,comments.* FROM comments where mytimestamp >= DATE_SUB(?, INTERVAL 5 SECOND)",
"parameter": [
"$river.state.last_active_begin"
]
}
],
"index": "profile",
"type": "comments",
"bulk_size": 100,
"max_bulk_requests": 30,
"bulk_timeout": "10s",
"flush_interval": "5s",
"schedule": "0/5 0-59 0-23 ? * *"
}
}
"statement": "SELECT id AS _id ,comments.* FROM comments where mytimestamp >= DATE_SUB(?, INTERVAL 5 SECOND)",
this 5 is the schedule interval depend on   "schedule": "0/5 0-59 0-23 ? * *"


Use FileManager for file operations

An idea originating from #40 is to use FileManager for file operations instead of using commands like
• try! bash("mkdir '\(path)'")
• try! bash("rm -rf '\(path)'")
I benchmarked the rm -rf command against FileManager.default.removeItem(atPath:) and noticed that for folders only containing a few files FileManager.default.removeItem(atPath:) is  roughly 2 times faster than rm -rf (0.06s vs 0.025s). However, when I copied a large iOS project into the folder whose deletion I benchmarked, rm -rf performed better (average of 1.6s) vs. FileManager.default.removeItem(atPath:) (average of 1.9s).
Speed-wise, rm -rf is probably the better approach, yet style-wise I prefer FileManager.
What are your thoughts on this?

I see it the same way as you do, but I'd give performance the higher priority here if the difference is considerable enough (meaning at least 15% faster). As I read your analysis, the more files there are, the bigger the performance gain is with rm -rf, so I'd opt for keeping it that way for now. But I wonder where the performance gain comes from – maybe we can replicate the same behavior rm -rf is doing under the hoods?

It could be that swift is performing some safety checks at runtime but we could have a look at the source of stdlib which will at least contain the linux version if it is bridged to NS libraries on macOS.

Closing this, as Accio is now deprecated.


Radix sort

Pull for submission.

Coverage remained the same at 100.0% when pulling e152a63c5ca61ecc5601f0cda7b3bf08531db20a on radix_sort into 725c40a6eeb62e5c4039881327eed303342b5d03 on master.


BUG: runtime error caused by read_xxx with relative path

Describe the bug
In the interactive environment, the work dir of a local cluster is determined during initialization. If the current dir of IPython changed, loading a dataframe with a relative path may result in a FileNotFound error.
To Reproduce
In [1]: import xorbits

In [2]: xorbits.init()
2023-03-14 21:20:14,070 xorbits._mars.deploy.oscar.local 22259 WARNING  Web service started at http://<IP_ADDRESS>:45351

In [3]: cd /path/to/my/dataset

In [4]: import xorbits.pandas as pd

In [5]: trips = pd.read_parquet("data.parquet")  # will cause a FileNotFoundError since Xorbits is initalized under another directory

In [6]: trips

Expected behavior
Xorbits should better take the relative path and make it an absolute path.

Take


Is there a way to unsort the datatable to its original order after clicking the sort arrows to manipulate the table?

I'm submitting a ...  (check one with "x")
[ ] bug report => search github for a similar issue or PR before submitting
[x ] feature request
[ ] support request => Please do not submit support request here, post on Stackoverflow or Gitter

Current behavior
When the table is sorted by various columns there is no way to unsort them in one click back to the original sort order.
Expected behavior
With a single click, one should be able to restore the original order of the table,.

Table version:
15.0.2

Angular version:
Angular CLI: 8.0.4
Node: 11.6.0
OS: darwin x64
Angular:
...

Package                      Version
@angular-devkit/architect    0.800.4
@angular-devkit/core         8.0.4
@angular-devkit/schematics   8.0.4
@schematics/angular          8.0.4
@schematics/update           0.800.4
rxjs                         6.4.0

Browser: [all | Chrome XX | Firefox XX | IE XX | Safari XX | Mobile Chrome XX | Android X.X Web Browser | iOS XX Safari | iOS XX UIWebView | iOS XX WKWebView ]

Language: [all | TypeScript X.X | ES6/7 | ES5]

Earlier on the third click, data used to reset to the original state i.e. unsorted but that behavior is no longer present. Do we have solution for this?


New version: JuliennedArrays v0.2.1

Registering package: JuliennedArrays
Repository: https://github.com/bramtayl/JuliennedArrays.jl
Created by: @bramtayl
Version: v0.2.1
Commit: 2ed165360b9152c5abcb6fa888d9678dab57a047
Reviewed by: @bramtayl
Reference: https://github.com/bramtayl/JuliennedArrays.jl/issues/18

Can you add
[compat]
julia = "1"

to the project file and retrigger the bot?


Fix accidental reference to global "value" variable

Hi,
We just encountered a bug today where moment was incorrectly parsing year and month fields. The bug was introduced while upgrading to moment v2.16.0.
We eventually tracked it down to another library (modular-scale) which was setting a global variable called value to a number. The isNumber function in moment is then mistakenly referencing this global value instead of the input value it gets passed. This pull request fixes the problem.
All the best,
David

We have tests for this, can you add the one that would have failed.

Aaah, its always safe to say typeof bullshit, because that is the proper way to check if something is defined :)

Merged in 93d1a675cb4879e85173be35422e9d88836a9b36

Sorry about that! I'll do some looking into why jshint didn't catch this for us.

I know why it didnt caught it. Because its a check to see if its defined :) which it isn't. I dont get how come the tests pass and it still fails under some scenario.

@ichernev It only failed if value was defined in the global scope by another library, which naturally our tests didn't cover.
Some other code in the global scope:
var value = 1;

Our code:
typeof value === 'number' ... // always true!

The tests still passed because the typeof check was just a quick check to cover the 90% case. The toString check covers all the cases we care about, but I believe it's more expensive. Since typeof value === 'number' is always false when value is undefined, our tests were just defaulting to the toString check every time.


feat(AIP-203): add resource name IDENTIFIER enforcement

Enforces AIP-203 guidance added in https://github.com/aip-dev/google.aip.dev/pull/1201 that the resource's name field must have google.api.field_behavior = IDENTIFIER.

This breaks the lint for me.
I have an identifier, but it still fails on linting.

@sam-utila can you open an issue instead of commenting on the PR? Also please include the proto that is failing.
worth noting this hasn't been released yet so you are using main which is probably not advisable.

No problem, considering I can always put new linting rules on disabled.
Just wondering that sometimes IDENTIFIERs can also be OUTPUT_ONLY and this is something that is now not encouraged by this AIP - haven't sinked in yet :-) .

I too am interested in the implication for name fields currently marked as OUTPUT_ONLY. I guess semantically IDENTIFIER doesn't preclude a behavior that is OUTPUT_ONLY ("Server ignores it if set as input"). However now we don't have a language for https://google.aip.dev/203#output-only scenario 3 where the name is the immutable field.


Issue rerun failures

Dear allan,
I would like to submit a PR for the issue when pytest_testrail run with pytest-rerunfailures.
At run time, the last success testcase will be the first submitted to testrail --> makes the test result for that particular test case ( as it will choose the fail run as last result)..
You can refer to #100.
If there is anything need to be done or checked, please tell me.
Thank you!

I believe you can run tests on your local before submitting it.

Thanks Peter, my bad.


Import Statements are not working even after complete/partial sync in IntelliJ

Description of the bug:
After doing complete sync and partial sync using Bazel plugin in IntelliJ still many import statements are not working in the Java classes and i am working in red. I am using macOS M1 chipset.
IntelliJ version: 2022.3.3
Bazel Plugin version: 2<IP_ADDRESS>.1-api-version-223
macOS version: M1 chipset (Ventura 13.5.1)
What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Import bazel + java project
Do the partial/complete sync of project from bazel plugin in intellij
Open any Java class and see if all the import statements are getting resolved or not.

Which Intellij IDE are you using? Please provide the specific version.
2022.3.3
What programming languages and tools are you using? Please provide specific versions.
Java
What Bazel plugin version are you using?
2<IP_ADDRESS>.1-api-version-223
Have you found anything relevant by searching the web?
Nope, did not find any concrete solution to this.
Any other information, logs, or outputs that you want to share?
I am not sure what to share here. You can tell me what you need and i will share.

I was not able to reproduce this with https://github.com/bazelbuild/intellij/tree/master/examples/java/greetings_project. Can you make sure that the sync succeeded? Also if you can send a small project where this problem happens, it will be easier to investigate.


Python 3 requires that relative imports are explicit

@sam-github @refack @bnoordhuis @trott

Closing in favor of #5


Saving the nimble object for later analysis

Hello and thank you for creating such an amazing tool for MCMC (and other stuffz).
I'm still a nimble beginner and I like working with models and data which require many iterations and computing time. Therefore I usually let the computer run for a few days and after it's done, I like to save the results to some file so I can analyze them later.
This worked with Stan and others. However, with nimble I ran the MCMC with the ·$run() function and saved the object to a .rds. I later tried to recover the samples with ·$mvSamples from the object loaded with readRDS() and a null pointer is returned.
Is this a bug? Intended? Or am I just doing something wrong? I'm also surprised that the rds can take up to 500MB on disk. This means that the samples must be there, somewhere, right?
Thanks a bunch,
Guido A. Moreira

Hi Guido, you need to get the samples back into an R object (rather than an R interface to what is actually a C++ object) and then save them, as one currently cannot save the C++ objects.
Here's some basic code
my_mcmc$run(niter)
samples <- as.matrix(my_mcmc$mvSamples)
# now save 'samples' to whatever output file format you prefer

We're actively working on an overhaul of the compilation system that will make it possible to save and reload the C++ objects, but it will still be a while until that is available.


Need set default inspectState value

I hope there is a setting to show whether the highlight should be displayed， so that I can always show the Toolbar and click Start when needed.
Now every time I refresh the page, inspectState will be initialized to inspecting, causing me to turn it off every time.
Probably like this.

That sounds like a bug/shouldn't happen. Can you try installing<EMAIL_ADDRESS>and confirm if the issue still exists?

Yes, I'm sure
The problem may come from here
https://github.com/aidenybai/react-scan/pull/145/files#diff-a7c6854aa81d0610f815f93f3d6c1d2c97ccb595c331f56da301e08428bbc9bdL136

Gotcha, will update when fix it live


Make validation case sensitive

This pulls out just the case sensitivity changes in #62

I would keep the changes in crawler-user-agents.json, but not in the validation code. Nothing prevents you of using crawler-user-agents.json in a case-sensitive way. Correct?

I would keep the changes in crawler-user-agents.json, but not in the validation code. Nothing prevents you of using crawler-user-agents.json in a case-sensitive way. Correct?

It makes very little sense to me that you would want the validation to be less strict. People could choose to use them in a case insensitive manner (and suffer the false-positive consequences that I've been mentioning), but the patterns themselves should match the instances in a case sensitive fashion by default. Why let people submit patterns that rely on case insensitivity and then do not function in many users code?
By defaulting to sensitivity in the validation script, you're just making sure that both use cases are supported.
Also, I don't think I can understate how many false positives using these in a case insensitive fashion will cause because of Pinterest being in the user agent for their mobile app, but pinterest being in their bot UA.

you're probably right. since we now have an active community on crawler-user-agents, a case-change would likely be detected fast.

I rebased and fixed the failing test case here.

thanks!


keybase client v5.9.0-20211217212642 causing / mount issues under Ubuntu 18.04, Fedora 34

With the new version of the keybase ( 5.9.0-20211217212642.29bfd9d39f ) that @heronhaye released last week, it's causing a very worrisome issue (which I initially thought to be a hard drive problem).
I've observed it now on two different unrelated Ubuntu 18.04 systems. Initially I didn't realize it was keybase until it happened on the second system and I thought it might be mount/fuse related.
Symptoms
Intermittently the / mount in the OS stops working. Running ls / hangs and doesn't return. Launching nautilus hangs and doesn't launch. Other applications (in my case a java app called SmartGit) that try to read from / also hang. Other mounts appear ok, for example ls /boot works fine. Apps which don't do anything in the / mount (for example my browser which I guess works entirely in my home directory, and home is a distinct mount, works fine)
Solution
After trying some different solutions heronhaye seems to have fixed this with the v5.9.3 release which you can update to via the normal means : https://keybase.io/docs/the_app/install_linux
Big thanks to heronhaye!
Note : If you'd used the workaround below while this was being worked out and put the keybase package on hold in apt make sure to run
sudo apt-mark unhold keybase

Workaround
When this happened on the second Ubuntu machine and I realized it might be keybase, I ran keybase ctl stop and immediately / became available, nautilus would launch, everything was fixed.
I was able to discover the previous version number at https://prerelease.keybase.io/linux_binaries/deb/index.html
Then either
sudo apt install keybase=5.8.1-20210930160723.fefa22edc1

or fetch the deb package listed on https://prerelease.keybase.io/linux_binaries/deb/index.html and install it
wget https://s3.amazonaws.com/prerelease.keybase.io/linux_binaries/deb/keybase_5.8.1-20210930160723.fefa22edc1_amd64.deb
sudo dpkg -i keybase_5.8.1-20210930160723.fefa22edc1_amd64.deb

Then run this to tell apt to hold and not upgrade keybase to 5.9.0
sudo apt-mark hold keybase

Once this issue is fixed you can unhold keybase with
sudo apt-mark unhold keybase

MAKE SURE NOT TO FORGET THAT YOU PUT KEYBASE ON HOLD AS OTHERWISE IT WILL NEVER GET UPDATES AGAIN

Looks like a duplicate of #24364.

@orgcontrib #24364 is a report from 2020 on Keybase 5.5.2 (not a new issue that began in 5.9.0 and which can be worked around by downgrading to 5.8.1).
Also #24364 refers to an inability to use keybase mounts, not symptoms that involve hanging any process that tries to take actions on the / mount (not the keybase mount)

I have the same issue on Fedora 34 with package keybase-<IP_ADDRESS>11217212642.29bfd9d39f-1.x86_64

I've now observed this on a third Ubuntu 18.04 machine. In this case it manifested during an unattended upgrade when grub ran to rebuild the menu after a kernel upgrade it hung. I had to follow the steps above to stop keybase, downgrade it and hold it at 5.8.1

I just ran into the same problem on an Ubuntu Server 20.04 LTS server after a routine upgrade.  /keybase is maybe-sorta-kinda mounted though it should not be (I use a private mount under ~/.config/keybase/kbfs/ instead).  It's hanging up df, lsof, and pretty much anything else that queries FS state.  Also ls /keybase (separated from the previous sentence to clarify).
Can confirm that running keybase ctl stop as non-root worked.
Running the given sudo apt command to install v5.8.1-20210930160723.fefa22edc1 did not work.  "E: Version '5.8.1-20210930160723.fefa22edc1' for 'keybase' was not found"
It would be nice if we could browse the S3 bucket, but whatever.  Downloading the .deb worked, but installing it did not, or at least it seems like it didn't work:
oot@exocortex:~# dpkg -i ~drwho/keybase_5.8.1-20210930160723.fefa22edc1_amd64.d
eb
dpkg: warning: downgrading keybase from 5.9.0-20211217212642.29bfd9d39f to 5.8.1
-20210930160723.fefa22edc1
(Reading database ... 179816 files and directories currently installed.)
Preparing to unpack .../keybase_5.8.1-20210930160723.fefa22edc1_amd64.deb ...
Unpacking keybase (5.8.1-20210930160723.fefa22edc1) over (5.9.0-20211217212642.29bfd9d39f) ...
Setting up keybase (5.8.1-20210930160723.fefa22edc1) ...
mkdir: cannot create directory ‘/keybase’: File exists
chown: cannot access '/keybase': Transport endpoint is not connected
chmod: cannot access '/keybase': Transport endpoint is not connected
dpkg: error processing package keybase (--install):
 installed keybase package post-installation script subprocess returned error ex it status 1
Processing triggers for mime-support (3.64ubuntu1) ...
Processing triggers for hicolor-icon-theme (0.17-2) ...
Processing triggers for shared-mime-info (1.15-1) ...
Errors were encountered while processing:
 keybase

Mountpoint /keybase did not exit.  Attempting the installation again resulted in its creation and "Device or resource busy" when trying to remove it.  keybase.mount is showing up as loaded, active, and mounted in the output of systemtl, however.  After some fiddling, I got systemctl stop keybase.mount to work and then I could manually install the .deb package.  Marked as "held."
Starting my isolated Keybase processes with systemctl --user worked.

Can confirm that running keybase ctl stop as non-root worked.

Yes, I believe it did

Running the given sudo apt command to install v5.8.1-20210930160723.fefa22edc1 did not work

Yes, this also didn't work for me on subsequent machines. I'll update the description on the issue to capture that

Downloading the .deb worked, but installing it did not, or at least it seems like it didn't work:

I got the same output that you see there @virtadpt but it did resolve the problem for me and Keybase now works and is running v5.8.1
Are you saying that after doing the dpkg -i ~drwho/keybase_5.8.1-20210930160723.fefa22edc1_amd64.deb Keybase wasn't working and you had to do further steps to get it working? I ask because on the 3 systems I've done this on, merely running the downgrade was all that was needed and Keybase and KBFS was working after that (despite the error messages shown during the downgrade that you mention)

Can someone experiencing this issue please send a log (keybase log send) when experiencing this issue on the latest version? Also, does it happen after a system restart?

@gene1wood Yes, I had to do it several times before it worked.  The key seems to have been sudo systemctl stop keybase.mount several times a few minutes apart before 0) the process trying to mount /keybase finally stopped and 1) dpkg terminated and unlocked the package database.  Once I got it successfully installed it worked.

@heronhaye I can send one right now if you like.  Things seem to be doing pretty well right now on my box.
Once I got it successfully installed, it did not happen when I restarted the system.

@virtadpt Please do, thanks. Post the log id here.

For anyone in this situation, does the following help?
sudo fusermount -uz /keybase
run_keybase

If so, I believe we have located the issue and should be able to push out a fix shortly.

@heronhaye Keybase log ID: 6f9b6814a3d454639a8ffc1c

Can someone experiencing this issue please send a log (keybase log send) when experiencing this issue on the latest version? Also, does it happen after a system restart?

@heronhaye You got it, chief. Log id: cb9e9c66959430fcdfc6b21c Yes, it occurs after a system restart and also when just logging out and back in again.
As to the fusermount command fixing the issue, I'm afraid it didn't, however issuing systemctl --user stop keybase-redirector.service && sleep 3; systemctl --user start keybase-redirector.service appears to get everything working okay. Oddly just issuing a restart to systemd for the service doesn't, perhaps because it needs a longer interval before re-execution? This one's puzzling, to be sure.

@RogueScholar @virtadpt Would you mind trying a test build to see if it fixes the redirector issue? It may require either sudo fusermount -uz /keybase or a computer reboot to release the mountpoint before restarting Keybase after installation. Thanks.
.deb: https://s3.amazonaws.com/tests.keybase.io/linux_binaries/deb/index.html
.rpm: https://s3.amazonaws.com/tests.keybase.io/linux_binaries/rpm/index.html
Please download the installer with the git commit 95a3939b3a. You can verify the signature from the sig file on the binary with the public key from https://book.keybase.io/docs/server/our-code-signing-key. Thanks. Let me know if you run into any issues, and if this solves the problem.

Hi, we believe we have fixed this issue. Please try upgrading Keybase. Thank you.
You will likely have to reboot your computer after the upgrade.

@heronhaye Thank you I'll upgrade and try it out. Is the fix for this problem found in #24789 ?

Yes. We did not realize that setreuid modified the suid bit; however, in previous Go versions, the calling goroutine took place in a different OS thread, so the bug did not manifest. In the latest Go version, the goroutine was scheduled on the same OS thread so the modified suid bit persisted.

I'll give it a try tomorrow.  Probably going to be working another late nighter tonight.

So on Ubuntu 18.04 I upgraded to the new version, and the problem manifested during the upgrade and apt hung
Preparing to unpack .../06-keybase_5.9.0-20220120174718.95a3939b3a_amd64.deb ...
Unpacking keybase (5.9.0-20220120174718.95a3939b3a) over (5.8.1-20210930160723.fefa22edc1) ...

...

Setting up keybase (5.9.0-20220120174718.95a3939b3a) ...
Autorestarting Keybase via systemd for gene.
Restarted existing root redirector via systemd.

Progress: [ 88%] [###########################################.................]

I can confirm the problem is happening as in another terminal if I do an ls / ls hangs.
I can unhang things by running sudo fusermount -uz /keybase as suggested though it looks like this doesn't unhang the apt update.
I'm not saying that this means the problem persists into the new version 5.9.0-20220120174718.95a3939b3a, just that the upgrade process hangs.
Looking at the process tree the step in the post_install.sh where it hangs is when it does chown root:root /keybase
root     20212  0.0  0.3<PHONE_NUMBER>84 pts/0   S+   11:03   0:06  |   |       \_ apt -y upgrade
root     28831  0.0  0.0  27868 12132 pts/1    Ss+  11:09   0:00  |   |           \_ /usr/bin/dpkg --status-fd 151 --configure --pending
root     17117  0.0  0.0  12888  3152 pts/1    S+   11:13   0:00  |   |               \_ /bin/bash /var/lib/dpkg/info/keybase.postinst configure 5.8.1-20210930160723.fefa22edc1
root     17119  0.0  0.0  13020  3368 pts/1    S+   11:13   0:00  |   |                   \_ bash /opt/keybase/post_install.sh
root     20389  0.0  0.0  15960  2440 pts/1    S+   11:13   0:00  |   |                       \_ chown root:root /keybase

I killed the chown process and the apt update completed with
/opt/keybase/post_install.sh: line 26: 20389 Terminated              chown root:root "$rootmount"

...

W: Operation was interrupted before it could finish

I then quit keybase in the GUI, confirmed it was not running in the process list and then ran
sudo apt install --reinstall keybase

which completed successfully. After that I had two new icons in the Ubuntu status bar which I've never seen before

I did another apt upgrade thinking maybe these new icons relate to an update issue but apt didn't identify any packages that needed updating (which makes sense).
I'll try rebooting and see if those icons go away.

@heronhaye Ya, I can confirm this new version doesn't seem to fix the issue. On version 5.9.0-20220120174718.95a3939b3a on Ubuntu 18.04, immediately on boot doing an ls / hangs. Running sudo fusermount -uz /keybase unhangs it.
@heronhaye Should we open a new issue as this and #24749 are currently closed?

Sorry folks, continuing to investigate.

@gene1wood Unfortunately I'm having trouble reproducing this bug on either Ubuntu 18.04 or 21.10.
First question, what are the permissions on /keybase from ls -l /?
Second, if you are able to, could you first try to disable the redirector with sudo keybase --use-root-config-file ctl redirector --disable. After this and rebooting, is your system and Keybase functional again, with the exception of the /keybase mountpoint, which should be alternately accessible from $XDG_RUNTIME_DIR/keybase/kbfs?
Thanks for your detailed reports so far.

First question, what are the permissions on /keybase from ls -l /?

So when it's working and not exhibiting the problem it shows as this
dr-xr-xr-x   1 root root          0 Jan 24 07:32 keybase

When the problem occurs I can't do an ls of /. After I run sudo fusermount -uz /keybase the permissions show differently, something with a lot of ? instead of rwx.

could you first try to disable the redirector

Sure, I'll reboot without running that, confirm that the problem occurs, then run it and reboot and see if it still occurs.
For example when I booted today running 5.9.0-20220120174718.95a3939b3a I haven't had a problem. I'm unsure what the conditions are that trigger the issue.

Strange. Let me know if you manage to run into it again. Thanks again for the detailed reports. I'll post in the other thread as well.

I'm on Fedora 35, with keybase-<IP_ADDRESS>20120174718.95a3939b3a-1.x86_64
I tried the procedure
sudo fusermount -uz /keybase
run_keybase

and the run_keybase command never finished
Journalctl tells me
janv. 25 22:57:17 mymachine systemd[2283]: Starting Mark boot as successful...
janv. 25 22:57:17 mymachine systemd[2283]: Finished Mark boot as successful.
janv. 25 22:57:22 mymachine Keybase[3312]: Quit through before-quit
janv. 25 22:57:22 mymachine Keybase[3312]: Quit the app
janv. 25 22:57:22 mymachine Keybase[3312]: Exec (null) not available for platform: darwin != linux
janv. 25 22:57:22 mymachine Keybase[3312]: Done with ctlstop
janv. 25 22:57:22 mymachine Keybase[3312]: exiting app
janv. 25 22:57:22 mymachine Keybase[3312]: browser window killed
janv. 25 22:57:22 mymachine Keybase[3312]: browser window killed
janv. 25 22:57:22 mymachine Keybase[3312]: [3312:0125/225722.033102:FATAL:gpu_data_manager_impl_private.cc(445)] GPU process isn't usable. Goodbye.
janv. 25 22:57:22 mymachine kernel: show_signal: 107 callbacks suppressed
janv. 25 22:57:22 mymachine kernel: traps: Chrome_IOThread[3466] trap int3 ip:557a1a4b9649 sp:7f8a64b8aba0 error:0 in Keybase[557a17e56000+6068000]
janv. 25 22:57:22 mymachine audit: BPF prog-id=63 op=LOAD
janv. 25 22:57:22 mymachine audit: BPF prog-id=64 op=LOAD
janv. 25 22:57:22 mymachine audit: BPF prog-id=65 op=LOAD
janv. 25 22:57:22 mymachine systemd[1]: Started Process Core Dump (PID 5821/UID 0).
janv. 25 22:57:22 mymachine audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-coredump@1-582>
janv. 25 22:57:22 mymachine systemd[1]: run-user-1000-keybase-kbfs.mount: Deactivated successfully.
janv. 25 22:57:22 mymachine sh[5835]: fusermount: entry for /run/user/1000/keybase/kbfs not found in /etc/mtab
janv. 25 22:57:22 mymachine systemd[2283]: Reloading.
janv. 25 22:57:22 mymachine systemd[2283]: Stopping Keybase core service...
janv. 25 22:57:22 mymachine systemd-coredump[5825]: [🡕] Process 3312 (Keybase) of user 1000 dumped core.

                                               Found module linux-vdso.so.1 with build-id: 5adf6bc52f5c51e10c5d1229124d27eebdae3b02
                                               Found module libdbusmenu-glib.so.4 with build-id: 6bc3a1183dab693921aaa6b6c6eb265c59e02bd5
                                               Found module libdbusmenu-gtk3.so.4 with build-id: c7b725aae4ca29ca64de268a33435a6cc7704650
                                               Found module libindicator3.so.7 with build-id: 4b6c12c50f3b52923760df92f3be559a69c0e6d2
                                               Found module libappindicator3.so.1 with build-id: 52a34c2096ee72837f18aabefff6c34e759d46d9
                                               Found module libudev.so.1 with build-id: 4a00048d488bd5a48980577f23fcaced551223c3
                                               Found module libdconfsettings.so with build-id: f07853974014f14e04029f524a32d4aea9558c1d
pleinty of stacktrace
janv. 25 22:57:22 mymachine systemd[2283]: keybase.gui.service: Main process exited, code=killed, status=5/TRAP
janv. 25 22:57:22 mymachine systemd[2283]: keybase.gui.service: Failed with result 'signal'.
janv. 25 22:57:22 mymachine systemd[2283]: keybase.gui.service: Unit process 5808 (Keybase) remains running after unit stopped.
janv. 25 22:57:22 mymachine systemd[2283]: keybase.gui.service: Unit process 5816 (Keybase) remains running after unit stopped.
janv. 25 22:57:22 mymachine systemd[2283]: keybase.gui.service: Unit process 5817 (Keybase) remains running after unit stopped.
janv. 25 22:57:22 mymachine systemd[2283]: keybase.gui.service: Consumed 5.227s CPU time.
janv. 25 22:57:22 mymachine systemd[1]<EMAIL_ADDRESS>Deactivated successfully.

@heronhaye Ok, just had to wait and the problem recurred.
On this boot the problem manifested, either it didn't start until 25 minutes into using the system or it started at boot and I didn't notice.
I produced a set of logs and sent them while the problem was occurring Log ID : d1c8a8d206f5e43d5cb8761c
When I run sudo keybase --use-root-config-file ctl redirector --disable I get this :
$ sudo keybase --use-root-config-file ctl redirector --disable
Redirector configuration updated.
▶ ERROR Failed to delete mountpoint at /keybase: remove /keybase: device or resource busy
▶ ERROR If KBFS is not being used, run `# pkill -f keybase-redirector`, delete /keybase and try again.
▶ ERROR remove /keybase: device or resource busy

I then ran sudo fusermount -uz /keybase to unhang my system
Running sudo keybase --use-root-config-file ctl redirector --disable a second time
I get now
$ sudo keybase --use-root-config-file ctl redirector --disable
Redirector configuration updated.
Redirector mount deletion successful.
Please run `# pkill -f keybase-redirector` to stop the redirector for all users.

After a few days, I haven't seen the problem (but I also don't have the /keybase/ mount). I've re-enabled it just now (so I can access KBFS) by running
$ sudo keybase --use-root-config-file ctl redirector --enable
Redirector configuration updated.
Redirector mount created.
Please run `run_keybase` to start the redirector for each user using KBFS.

and I'll report back here if the problem begins occurring again

@heronhaye And the symptom manifested again (on version 5.9.0-20220120174718.95a3939b3a), hanging processes that do anything with /.
I stopped keybase and it unhung everything.
Is there something I can do when this next occurs to get you the details you need to determine the cause of this? When I next see things hang because of keybase, what should I run to help?

Just like to confirm that I get the same "upgrade hangs" or also reinstall hangs issue on Ubuntu 20.04. However so far I have not noticed any of the mounting issues.

Seems to have been fixed for me with the latest v5.9.1release.

Following what @dietmarw said above, I removed the hold on keybase
sudo apt-mark unhold keybase

And did an sudo apt upgrade on Ubuntu 18.04
It looks like the v5.9.1 .deb package hasn't been made available yet : https://prerelease.keybase.io/linux_binaries/deb/index.html
as when I upgrade from 5.8.1-20210930160723.fefa22edc1 I got 5.9.0-20220120174718.95a3939b3a
and the same hang occurred again until I stopped keybase
@heronhaye Do you know when your release of v5.9.1 from a few weeks back will be packaged for Debian/Ubuntu and uploaded to https://prerelease.keybase.io/linux_binaries/deb/ ?

I'm sorry, my mistake. I also got "only" 5.9.0-20220120174718.95a3939b3a from the update channel. But that was a newer version than the one with which my problem occurred. So I removed the lock and the update succeeded this time. So I'm not actually on v5.9.1. Sorry for the confusion but the latest version above fixed it for me.

@gene1wood, @dietmarw, and all others having issues, please try out the latest 5.9.3 test build:
.deb: https://s3.amazonaws.com/tests.keybase.io/linux_binaries/deb/index.html
.rpm: https://s3.amazonaws.com/tests.keybase.io/linux_binaries/rpm/index.html
A restart might be required before/after installing.
You can verify the signature on the binary against our code signing key at https://book.keybase.io/docs/server/our-code-signing-key.
I'm hoping this will resolve the issues, especially with ls / hanging. If you are still having issues, let me know. Otherwise I will promote this test build to a release in a few days. Thanks for your patience.

Sure thing.
I confirmed the code signature by running gpg -v keybase_5.9.3-20220208204733.d72d1e68b3_amd64.deb.sig and comparing the fingerprint of the signing key to the one on the doc page you linked.
I've upgrade from 5.9.0-20220120174718.95a3939b3a to 5.9.3-20220208204733.d72d1e68b3 successfully (no errors). I'll continue running this new version and will report back here if the problem manifests.

@heronhaye Working so far with 5.9.3.

Anyone else test out this build?

I've now been running this build, 5.9.3-20220208204733.d72d1e68b3 on Ubuntu 18.04 for 8 days now and the problem has not manifested once. With versions 5.9.0-20211217212642.29bfd9d39f and 5.9.0-20220120174718.95a3939b3a I feel confident the problem would have manifested already. I'm feeling like 5.9.3 was the fix.

Okay, thanks for testing. I'm going to release this build shortly.

Thanks for your patience. Please upgrade to 5.9.3 via your package manager or https://keybase.io/docs/the_app/install_linux and let me know if you still run into any issues.

Sorry for the silence but since my issue was already solved in the 5.9.0-20220120174718.95a3939b3a version but now I can confirm that the 5.9.3 still is working fine for me.

v5.9.3 is working for me, also.  Only weird thing is that I don't see it in the output of df -h anymore (though it can be found in the output of mount).

It's not working for me
Jun  9 14:47:40 v systemd[8222]: Starting Keybase core service... Jun  9 14:47:40 v systemd[8222]: Started Keybase core service. Jun  9 14:47:40 v systemd[8222]: Starting Keybase Filesystem service... Jun  9 14:47:40 v systemd[8222]: Starting Keybase Root Redirector for KBFS...  Jun  9 14:47:40 v keybase-redirector[116572]: Mount error, exiting cleanly: /keybase is non-empty Jun  9 14:47:40 v systemd[8222]: Started Keybase Filesystem service. Jun  9 14:47:40 v systemd[8222]: Starting Keybase GUI... Jun  9 14:47:40 v systemd[8222]: Started Keybase GUI. Jun  9 14:47:40 v Keybase[116617]: Version: 5.9.3-20220216215910+c82d65a685 Jun  9 14:47:40 v Keybase[116617]: (node:116617) electron: The default of contextIsolation is deprecated and will be changing from false to true in a future release of Electron.  See https://github.com/electron/electron/issues/23506 for more information Jun  9 14:47:40 v Keybase[116617]: (node:116617) electron: The default of contextIsolation is deprecated and will be changing from false to true in a future release of Electron.  See https://github.com/electron/electron/issues/23506 for more information Jun  9 14:47:40 v<EMAIL_ADDRESS>unable to update icon for Keybase1 Jun  9 14:47:41 v Keybase[116617]: [116617:0609/144741.729489:FATAL:gpu_data_manager_impl_private.cc(445)] GPU process isn't usable. Goodbye.  Jun  9 14:47:41 v systemd[8222]: keybase.gui.service: Unit process 116624 (Keybase) remains running after unit stopped. Jun  9 14:47:41 v systemd[8222]: keybase.gui.service: Unit process 116625 (Keybase) remains running after unit stopped. Jun  9 14:47:41 v systemd[8222]: keybase.gui.service: Unit process 116627 (Keybase) remains running after unit stopped. Jun  9 14:47:41 v systemd[8222]: keybase.gui.service: Unit process 116658 (Keybase) remains running after unit stopped.

@Mykola-Veryha am I correct that this is an issue with the GUI and not KBFS? We will attempt to address this in the next release.

I'm also having sporadic issues with this after restarting my desktop (ubuntu 22.04)

Ya, at some point after Feb 2022 when I was working with @heronhaye on this issue, the problem started happening again. It occurs across multiple Ubuntu machines, running 20.04 and 22.04. It's insidious enough (as it causes all kinds of havoc on a computer when writes get locked) that I now (sadly) routinely shutdown Keybase when my machine boots. I've also migrated most things off of KBFS to instead use Syncthing. I think my plan at this point is to either try adding the KEYBASE_NO_KBFS environment variable to my system to be able to use Keybase without KBFS or just uninstall it from the machines that I have it on.
@heronhaye Are you having trouble reproducing this? If you need any additional info, or a screen recording of it happening, or specific logs or command output when it does occur, let me know.

I've done the following to try prevent this issue on my end,  I hope this helps someone else.

Disable keybase on systemctl. This prevents keybase running on startup

systemctl --user disable keybase.service

Create a reset_keybase script

/root/scripts/reset_keybase.sh

#!/usr/bin/env bash

/usr/bin/fusermount -uz /keybase

(and remember to make script executable)

Add the reset_keybase script to the sudoers.d directory (allows non-root user to run this script as sudo)

Don't put periods in file name

/etc/sudoers.d/reset_keybase

Cmnd_Alias RESET_KEYBASE_CMDS = /root/scripts/reset_keybase.sh

alexiswl ALL=(ALL) NOPASSWD: RESET_KEYBASE_CMDS

Create script for user

scripts/reset_keybase_on_reboot.sh

#!/usr/bin/env bash

set -euo pipefail

# Wait a minute for everything to be set up
/usr/bin/sleep 60

# Run fusermount
# This wont trigger a password since we have permissions to run this script
sudo /root/scripts/reset_keybase.sh

# Now reset keybase
/usr/bin/run_keybase

and remember to make executable.

Create user crontab

crontab -e

And add the following line
@reboot nohup /home/alexiswl/scripts/reset_keybase_on_reboot.sh 1>/dev/null 2>&1 &

I get fairly frequent hangs in Ubuntu 22 with Keybase, where various apps will seize up while trying to open files, dialogs, etc.  The command df /keybase will also hang.  Stopping Keybase with keybase ctl stop unblocks them and allows them to continue.
This may be related to the fact that the Keybase menu is unresponsive on login, and always crashes when I launch the app.
Once I've launched the app things seem to go smoothly.
Looking at @alexiswl 's workaround above, combined with my symptoms, it seems likely that Keybase isn't starting correctly.
Keybase itself isn't completely stuck though.  I can get a red notification dot on the Keybase menu after login, even though the menu doesn't respond to clicks and crashes on launch.
rb@kiwi:~$ uname -a
Linux kiwi 6.2.0-39-generic #40~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Nov 16 10:53:04 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
rb@kiwi:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.3 LTS
Release:	22.04
Codename:	jammy
rb@kiwi:~$ dpkg-query -l '*keybase*'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version                         Architecture Description
+++-==============-===============================-============-==========================================
ii  keybase        6.2.3-20231016183016.06cb935ee3 amd64        The Keybase Go client, filesystem, and GUI

I think my plan at this point is to either try adding the KEYBASE_NO_KBFS environment variable to my system to be able to use Keybase without KBFS or just uninstall it from the machines that I have it on.

Following this suggestion, I edited /usr/lib/systemd/user/keybase.service to read:
# Added KEYBASE_NO_KBFS=1 to suppress broken /keybase mount.  See <https://github.com/keybase/client/issues/24764#issuecomment-1821788495>.  RB 2023-12-20.
Environment=KEYBASE_SERVICE_TYPE=systemd KEYBASE_NO_KBFS=1

and for good measure I did:
systemctl --user disable kbfs

and now operations on /keybase fail immediately, which is better than hanging
rb@kiwi:~$ df /keybase
df: /keybase: Transport endpoint is not connected

Also, Keybase no longer crashes on launch of the GUI app.

@rptb1 's description above of the symptoms are exactly what I experience on all of my Ubuntu desktop machines.
@heronhaye I worry, because this issue that I opened in 2021 has a title potentially implying that the problem is only with Ubuntu 18.04 and with an older version of Keybase, that it's being overlooked by you and other devs at Zoom because it looks like an issue with an out of date OS. I'll say that this problem is easy to reproduce across any modern Ubuntu system.
I'll update the title of the issue in hopes of avoiding this confusion.

Another issue I've found is that anytime I type / into an IDE, GitHub Copilot will also freeze trying to check what possible folders I'm trying to list and autocomplete.  This in-turn freezes the IDE!
I've now resorted to just restarting keybase as soon as this starts happening.
I now run a cron every minute with
if timeout 2 ls / 1>/dev/null 2>&1; then
  exit
fi

...reset_keybase_commands (see my previous comments in the thread)

Weirdly the above doesn't always work.
timeout 2 ls /

will return the top mount list,
but the below will freeze
ls /

The following is fine
bash -c 'ls /'

While this will hang
bash -ic 'ls /'

I have updated my comment accordingly

I was able to disable the redirector with the instructions at: https://github.com/keybase/client/issues/24764#issuecomment-1021737083
Note that the redirector makes /keybase available but the kbfs is also available without the redirector at ~/keybase.  My intention is to leave the redirector disabled and just use ~/keybase.

Still happening with keybase version 6.3.1 on Ubuntu 24.04.

This issue came up again over the weekend and I initially thought I had a hardware failure.
The solution I found was to edit the last line in ~/.config/autostart/keybase_autostart.desktop as follows:
Exec=env KEYBASE_AUTOSTART=1 run_keybase -f

This disables KBFS, which I believe to be some sort of Dropbox alternative, and which I don't use.

Still happening in Keybase 6.4.0 under Ubuntu 22.04.
Perhaps you could remove the claim that this problem is fixed from the issue post?

@rptb1 I've updated the issue description to capture that it hasn't been fixed.


What are "frame pointers"? The documentation needs to explain what they are, and how they can be used

It is not clear what "Frame pointers" are, especially when they are briefly mentioned here in the "Symbols and Symbol Files" article here:
https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/symbols-and-symbol-files
and here:
https://docs.microsoft.com/en-us/cpp/build/reference/oy-frame-pointer-omission?view=vs-2017
Does it mean anything related to how Windows handle the memory allocation, which also mentioned frame pointers here.
https://docs.microsoft.com/en-us/cpp/build/stack-usage?view=vs-2017#alloca
In that instance, it is said _alloca() requires the use of frame pointers, which is still vague for us to understand what they are, and how it can be used with _alloca(). Are there any "gotchas" we need to be aware of when working with "frame pointers"?
Thanks.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: bbd14f34-9a79-51c8-5691-0af44764db45
Version Independent ID: fc84eed6-cfc6-aa7d-1ac7-f3cb8278572d
Content: /Oy (Frame-Pointer Omission)
Content Source: docs/build/reference/oy-frame-pointer-omission.md
Product: visual-cpp
GitHub Login: @corob-msft
Microsoft Alias: corob

@tommai78101 Most of the time, you don't need to worry about frame pointers. You probably only need to know what they are and how and whether they are used if you write assembly code, because it affects how you write your function prologs and epilogs. You need to enable frame pointers if you use alloca, setjmp/longjmp, or structured exception handling. And if your code will be debugged by other people, then frame pointers will make their lives easier.
What's a frame pointer? Every time you call a non-trivial function (that is, one that might call another function, or cause an exception), the compiler generates a stack frame, an area allocated on the stack to be used by the called function. Depending on the architecture, it may have space set aside to store non-volatile register contents, local variables, and space allocated by alloca(), as well as space for the parameters of functions it calls. A special-purpose, architecture-specific register is used as the stack pointer, and it always points at the top of the stack. A frame pointer is another register-based pointer that's used to figure out where the start of the stack frame is. It's not strictly necessary, since the compiler generates code to remove the stack frame when the call returns, or when a longjmp or an exception occurs, and it creates function prolog and epilog code that does the right thing needed to unwind the stack at each point in the function. When a frame pointer is used, it points to either the beginning of or a specified offset into the stack frame, depending on architecture and calling convention. Like the stack pointer, the frame pointer also uses an architecture-specific register (EBP on x86, RBP on x64, r29 on ARM or ARM64). Having the frame pointer in the register and stored at fixed locations in the stack frame allows for rapid traversal of the stack, whether for debugging or exception unwinding, as each frame pointer stored in the stack frame either points to the previous frame, or to a spot that's at a fixed offset from the previous frame. When there are no frame pointers, it's a little more work for a debugger, but it may save a small bit of time and space in the code by not having to save and restore the frame register, and lets functions use that register for something else. That's why it's an optimization option. By default, the x64 ABI doesn't use frame pointers, but the other architectures do.

I see. Thanks for the long explanation.
But, if possible, can that description be added somewhere in the article? It helps to explain why it's an optimization option, and that is something I was looking for more information on when I had been curious about the inclusions in some projects.


Zeroes gives an "-inf" value.

The docs include a way to calculate SPI for zeroes but was it not implemented in this package?
Zeros gives an "-inf" value for SPI calculation

@wsor330 Are you referring to the paper by Lloyd-Hughes and Saunders? I don't believe they specify a way to handle zeros.

This issue is fixed with the new version 1.0

This issue is fixed with the new version 1.0