Serde Serialize/Deserialize implementation?

I've found myself wanting to dump some BitVecs into JSON; it'd be nice if BitVec implemented Serde's Serialize/Deserialize (likely behind a crate feature flag).

I'm definitely on board with this; I've had it as a vague to-eventually-do item but as I've never used serde before I don't really know what that entails or how to do it correctly.
I am currently in the middle of a rewrite for the 0.10/1.0 release; I had hoped to finish it before my work was interrupted this winter but I didn't quite make it, and I haven't picked it back up yet.
Something about which I'm unsure would be how to serialize the bit slices - I'm leery of JSON numerics, but a byte array is probably a better format than a string of literal 1 and 0 characters. I'd be delighted to know what your expected behavior is as a guide on how to move forward.
Thanks for your interest and your request!
Alexander Payne

From: Darius Jahandarie<EMAIL_ADDRESS>Sent: Thursday, February 28, 2019 14:47
To: myrrlyn/bitvec
Cc: Subscribed
Subject: [myrrlyn/bitvec] Serde Serialize/Deserialize implementation? (#4)
I've found myself wanting to dump some BitVecs into JSON; it'd be nice if BitVec implemented Serde's Serialize/Deserialize (likely behind a crate feature flag).

You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmyrrlyn%2Fbitvec%2Fissues%2F4&data=02|01||3cd012e8ea5b47eb51d108d69dc6627e|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|636869872705221632&sdata=iFUZnzBiE8rPigxqxXyr7A8%2FnYG4Ge1EB8LxZ9DueC8%3D&reserved=0, or mute the threadhttps://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAFpZVYHwto2SWxC0iahEJa-yd3-lLVFUks5vSE6FgaJpZM4bXzmE&data=02|01||3cd012e8ea5b47eb51d108d69dc6627e|84df9e7fe9f640afb435aaaaaaaaaaaa|1|0|636869872705231640&sdata=4xBVAeGDv3LlJuO8pRKSJWSpe9xaL9DIeChAPEyDGJ0%3D&reserved=0. https://github.com/notifications/beacon/AFpZVUzF0_EGhwi7kW8XclPZaWrvXQKpks5vSE6FgaJpZM4bXzmE.gif

Good to hear! I’m no expert on Serde itself, but in terms of the JSON serialization, I manually played around with a couple things.
The first attempt was a JSON array of bools, but this ended up being too inefficient (in terms of on-the-wire representation and eventual in-browser memory representation). I’m not sure if Serde exposes a way to provide multiple serializations, but if it does, also allowing users to pick the underlying int Vec as the serialization might be nice.
FWIW, in my case, I sort of gave up entirely and summarized the BitVec (each entry representing a second) into a Vec (each entry representing how many seconds were 1 in a given minute). I.e., chunking by 60 and counting 1s. For the purpose of my frontend, this level of granularity was all that was required.
Unfortunately this all doesn’t really help answer the question. Though, if only one behavior could be picked, the array of bools is probably the most intuitive one at least.

I think I have a reasonable concept of how to accomplish this.
Fundamentally, a BitSlice is composed of two structures:

the handle: a (magically packed) structure of pointer (useless), element count (necessary), head position (necessary), and tail position (necessary)

the type parameters C: Cursor and T: Bits also need to be de/serialized somehow

the data: a sequence of arbitrary T elements.

The following RON is probably a reasonable representation:
BitSlice {
 cursor: "BigEndian" | "LittleEndian" | other?,
 storage: "u8" | "u16" | "u32" | "u64",
 elements: usize,
 head: u8,
 tail: u8,
 data: [
  number,
  number,
 ],
}

I am not entirely sure how the cursor and storage parameters will work, but I'll cross that bridge later. In the meantime, this is the data model I'm going to try out and see how it goes.

@djahandarie does commit 19734399 address what you want?

bitvec 0.11.0 contains a serde feature which enables serialization of BitSlice in all environments, and de/serialization of BitBox and BitVec when an allocator exists.


Sending HUP does not reopen log files

Currently using logrotate to rotate the log files for resque-pool but sending HUP does reopen the log files - instead the manager keeps logging to the renamed log file, in this case resque.log.1.

Interesting that this is still open from 2011.  I had a bunch of servers running resque-pool that kept running out of disk space even though logrotate is sending a HUP to the masters after rotate.
I was going through the differences between what HUP does vs what I expected it to do and I realized that all resque-pool is re-opening $stdout and $stderr.  If you have additional loggers inside of your application they will never get re-opened.
I went into Unicorn and looked through what they are doing there and apparently it loops through the ObjectSpace and looks for open files that are in append/write-only mode and re-opens those.  I think this is probably a better solution.
Has anyone attempted to do something similar in resque-pool?  I am about to take a run at doing something similar to Unicorn's logrotate approach so I can try that out in production.
@tekin Did you ever get this resolved some other way?

My gut instinct is that running through ObjectSpace and managing everyone else's append files is both overkill and dangerous.  I'd rather just be responsible with my own logfiles and let other software handle their own (e.g. register additional HUP trap to manage the rest).
BUT... if Unicorn is doing it, then it must be a good idea and work well for a bunch of people.  I trust Eric Wong's unix-fu.  Send in a PR that (very) closely mirrors unicorn's code and I'll probably merge it.

Merged to master now.  Next rc gem push (0.6.0.rc1?) should include this.  Thanks @PatrickTulskie!

Awesome. Glad I could help!


DO NOT MERGE Port timeout JS code from "Help with fees"

This ports over most of the code for the modal "your session is about to
expire" dialog from the "Help with fees" app.
Do not merge
This PR is just for sharing around, there's more work to be done. I've only ported the code over mostly verbatim – but it still needs a fair bit of tuning.

Superseded by #271.


TypeScript types not working due to typo in package.json

Hello,
Thanks for releasing a TypeScript version! I'm excited to have a typed interface available.
However, the types field in package.json currently references a non-existent file as of the 2.0.0 release:
  "types": "./build/src/index.t.ds",

I believe the correct file is:
  "types": "./build/src/index.d.ts",

(Note the swapped letters in the file extension).
This causes an error when trying to import the module in a TypeScript project: TS7016: Could not find a declaration file for module '@google-cloud/storage'.
Thanks!

🤦♂

Thanks for the quick response!

@JustinBeckwith Hi, thanks for the quick fix, when shall we expect a new patched version on NPM?

@jakubdrabik This fix is on NPM now with 2.0.1.


Use -I rather than -isystem for LLVM include directory

Building on Arch Linux, using gcc 6.1.1 and LLVM 3.7.1, fails with
the error:
/usr/include/c++/6.1.1/cmath:45:23: fatal error: math.h: No such file or directory
 #include_next <math.h>

Replacing -isystem with -I alters the include file search path and avoids
the problem.
This fixes #797

@sylvanc this blew up pretty hardcore. so the same issue exists. we'll need another solution for #797.

All of these errors are conversion warnings dressed up in -Werror. Is this something perhaps fixable or perhaps false positives?  It seems odd that the code would generate these regardless of vendor.

This was fixed in #824, so no need for this PR anymore.

@Earnestly, I think I've figured out the issue, and they do appear to be false positive errors.  The errors/warnings in question are caused by -Wunused-parameter (from -Wextra) and -Wconversion.
But, from the Gnu C preprocessor docs:

The header files declaring interfaces to the operating system and runtime libraries often cannot be written in strictly conforming C. Therefore, GCC gives code found in system headers special treatment. All warnings, other than those generated by ‘#warning’ (see Diagnostics), are suppressed while GCC is processing a system header. Macros defined in a system header are immune to a few warnings wherever they are expanded. This immunity is granted on an ad-hoc basis, when we find that a warning generates lots of false positives because of code in macros defined in system headers.

Normally, only the headers found in specific directories are considered system headers. These directories are determined when GCC is compiled. There are, however, two ways to make normal headers into system headers. [One is -isystem and the other is a #pragma.]

Therefore, -isystem silences GCC about the extra warnings/errors but when used on a system include directory breaks the ordering needed by #include_next.
This is just to document what is going on here.

That's pretty interesting; I wouldn't have guessed the situation with system headers and GCC's leniency towards them.
Thank you very much for concluding this, TIL \o/


PUT to submit FormData

How are you
how to use the PUT to submit FormData
excuse me

What's your questions exactly?

I think he means that PUT doesn't have constructingBodyWithBlock

For general help, please ask on StackOverflow.


Deleted wallet_seq_no Api call from Rust and Java wrapper.

What about removing of this call from iOS wrapper?

I can not check for iOS what after deleting all tests work.

It is hard. I can not check for iOS what all tests work after deleting seqNo.


DB configuration options should be available via ENV

All database config options, instead of just python config parameters, should also be possible to set via an environment variables.

yup, all database configuration options can be set via env variables


upgrade: drivelist to v5.0.22

https://github.com/resin-io-modules/drivelist/pull/168

Change-Type: patch
Changelog-Entry: Fix increased CPU usage because of perl regular expression in macOS.
Fixes: https://github.com/resin-io/etcher/issues/1288
Signed-off-by: Juan Cruz Viotti<EMAIL_ADDRESS>

Minor nit, but could you change it to Fix occasional increased ...?

Done!


Encode Url

Encode the special sanitize chars

XSS protection isn't just adding as much encoding/sanitization/whatever you can come up with on top of the code. In fact, double escaping can backfire and sometimes result in vulnerabilities.


Improve error message when the user forgets to specify environments

Fixes #216

Verified new error message is now

Error: could not find enough structure properties to display, but there are atom properties. Please provide the environment list to display them

providing an environment list is enforcing to provide a cutoff

Thanks for checking this!


Redo Invertebrate View Hiding Unset Values

Only values which are set should be displayed.

Handled in branch styling/


Running --minimal smoke-tests in Ubuntu 22.04

Hi, I'm trying to use the smoke tests in https://github.com/dotnet/installer/tree/release/6.0.1xx/src/SourceBuild/tarball/content/test/Microsoft.DotNet.SourceBuild.SmokeTests as building tests for dotnet packaging in Ubuntu 22.04 using tag 6.0.103-source.build. I know this is kind of legacy as per https://github.com/dotnet/source-build/issues/1654, but by now is what I'm executing.
I got some problems executing it from the building. First was the smoke-test.sh file, I had to correct the artifacts folder:
--- a/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/smoke-test.sh
+++ b/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/smoke-test.sh
@@ -7,7 +7,7 @@ VERSION_PREFIX=6.0
 # See https://github.com/dotnet/source-build/issues/579, this version
 # needs to be compatible with the runtime produced from source-build
 DEV_CERTS_VERSION_DEFAULT=6.0.0-preview.6.21355.2
-ARTIFACTS_DIR="$SCRIPT_ROOT/../../../../../../../artifacts/"
+ARTIFACTS_DIR="$SCRIPT_ROOT/../../../../artifacts/"
 executingUserHome=${HOME:-}

 export DOTNET_CLI_TELEMETRY_OPTOUT=1

After that, If I execute manually it (./assets/smoke-tests/smoke-test.sh --dotnetDir /build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet --projectOutput --archiveRestoredPackages --targetRid ubuntu.22.04-x64) is all OK -log attached  all-tests.log - (except for me because of the part where it's downloading stuff to be able to run the tests).
So, let's go to the problems:

Trying the options minimal, excluding omnisharptest and onlinetest into smoke-test.sh to see if it can only test local stuff... It doesn't. Executing ./assets/smoke-tests/smoke-test.sh --dotnetDir {Directory.GetParent(DotNetHelper.DotNetPath)} --projectOutput --archiveRestoredPackages --targetRid {Config.TargetRid} --minimal --excludeOnlineTests --excludeOmniSharpTests all resolved is   ./assets/smoke-tests/smoke-test.sh --dotnetDir /build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet --projectOutput --archiveRestoredPackages --targetRid ubuntu.22.04-x64 --minimal --excludeOnlineTests --excludeOmniSharpTests :

[...]
build/dotnet6-KxJTNR/dotnet6-6.0.103/t

est/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet/sdk/6.0.103/MSBuild.dll -maxcpucount -property:PublishTrimmed=true -property:PublishReadyToRun=true -property:SelfContained=True -property:
_CommandLineDefinedSelfContained=true -property:RuntimeIdentifier=linux-x64 -property:_CommandLineDefinedRuntimeIdentifier=true -restore -target:Publish -verbosity:m /bl:/build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0407084134/logs/CSharp_classlib_local_nohttps_publish-self-contained-R2R-portable.binlog ./CSharp_classlib.csproj
Determining projects to restore...
Restored /build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0407084134/CSharp_classlib/CSharp_classlib.csproj (in 120 ms).
CSharp_classlib -> /build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0407084134/CSharp_classlib/bin/Debug/net6.0/linux-x64/CSharp_classlib.dll
Optimizing assemblies for size, which may change the behavior of the app. Be sure to test after publishing. See: https://aka.ms/dotnet-illink
CSharp_classlib -> /build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0407084134/CSharp_classlib/bin/Debug/net6.0/linux-x64/publish/
finished language C#, type classlib
starting language C#, type xunit
running new
The template "xUnit Test Project" was created successfully.
running restore
/build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet/sdk/6.0.103/MSBuild.dll -nologo -maxcpucount -target:Restore -verbosity:m /bl:/build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0407084134/logs/CSharp_xunit_local_nohttps_restore.binlog ./CSharp_xunit.csproj
Determining projects to restore...
/build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0407084134/CSharp_xunit/CSharp_xunit.csproj : error NU1101: Unable to find package Microsoft.NET.Test.Sdk. No packages exist with this id in source(s): source-built-packages
/build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0407084134/CSharp_xunit/CSharp_xunit.csproj : error NU1101: Unable to find package xunit. No packages exist with this id in source(s): source-built-packages
/build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0407084134/CSharp_xunit/CSharp_xunit.csproj : error NU1101: Unable to find package xunit.runner.visualstudio. No packages exist with this id in source(s): source-built-packages
/build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0407084134/CSharp_xunit/CSharp_xunit.csproj : error NU1101: Unable to find package coverlet.collector. No packages exist with this id in source(s): source-built-packages
Failed to restore /build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0407084134/CSharp_xunit/CSharp_xunit.csproj (in 169 ms).
make[1]: *** [debian/rules:132: override_dh_install] Error 1
make[1]: Leaving directory '/build/dotnet6-KxJTNR/dotnet6-6.0.103'
make: *** [debian/rules:30: binary] Error 2

Therefore I can't use this as a workaround for the next point @MichaelSimons :( (although I'm working on how to download this stuff while creating the tarball... no good progress on this yet).

Executing the VERBOSE=1 ./build.sh --run-smoke-test allowing to download requisites (as we see it worked manually previously we suppose it has to work... it doesn't):

VERBOSE=1 ./build.sh --run-smoke-test
Found bootstrap SDK 6.0.102, bootstrap Arcade 6.0.0-beta.21609.4, bootstrap SourceLink 1.1.0-beta-21480-02
Microsoft (R) Build Engine version 17.0.0+c9eb9dd64 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

/build/dotnet6-aZycfo/dotnet6-6.0.103/.dotnet/sdk/6.0.102/MSBuild.dll /bl:/build/dotnet6-aZycfo/dotnet6-6.0.103/artifacts/log/Debug/BuildTests_0330135715.binlog /clp:v=m /fileLoggerParameters:LogFile=/build/dotnet6-aZycfo/dotnet6-6.0.103/artifacts/logs/BuildTests_0330135715.log /flp:v=detailed /t:RunSmokeTest /build/dotnet6-aZycfo/dotnet6-6.0.103/build.proj
[SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.Sdk. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
[SourceBuiltSdkResolver] Looking for SDK Microsoft.DotNet.Arcade.Sdk. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
[SourceBuiltSdkResolver] Overriding Microsoft.DotNet.Arcade.Sdk with 'ARCADE'
[SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadAutoImportPropsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
[SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.ILLink.Tasks. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
[SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadManifestTargetsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'

  Welcome to .NET 6.0!
  ---------------------
  SDK Version: 6.0.102

  ----------------
  Installed an ASP.NET Core HTTPS development certificate.
  To trust the certificate run 'dotnet dev-certs https --trust' (Windows and macOS only).
  Learn about HTTPS: https://aka.ms/dotnet-https
  ----------------
  Write your first app: https://aka.ms/dotnet-hello-world
  Find out what's new: https://aka.ms/dotnet-whats-new
  Explore documentation: https://aka.ms/dotnet-docs
  Report issues and find source on GitHub: https://github.com/dotnet/core
  Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
  --------------------------------------------------------------------------------------
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.Sdk. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadAutoImportPropsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.ILLink.Tasks. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadManifestTargetsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
    Determining projects to restore...
    Restored /build/dotnet6-aZycfo/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/Microsoft.DotNet.SourceBuild.SmokeTests.csproj (in 5 sec).
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.Sdk. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadAutoImportPropsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.ILLink.Tasks. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadManifestTargetsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
    Microsoft.DotNet.SourceBuild.SmokeTests -> /build/dotnet6-aZycfo/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/Microsoft.DotNet.SourceBuild.SmokeTests.dll
  Test run for /build/dotnet6-aZycfo/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/Microsoft.DotNet.SourceBuild.SmokeTests.dll (.NETCoreApp,Version=v6.0)
  Microsoft (R) Test Execution Command Line Tool Version 17.0.0
  Copyright (c) Microsoft Corporation.  All rights reserved.

  Starting test execution, please wait...
  A total of 1 test files matched the specified pattern.
  [xUnit.net 00:39:36.88]     Microsoft.DotNet.SourceBuild.SmokeTests.SmokeTests.SmokeTestsScript [FAIL]
    Failed Microsoft.DotNet.SourceBuild.SmokeTests.SmokeTests.SmokeTestsScript [39 m 36 s]
EXEC : error Message:  [/build/dotnet6-aZycfo/dotnet6-6.0.103/build.proj]
     Assert.Equal() Failure
  Expected: 0
  Actual:   127
    Stack Trace:
       at Microsoft.DotNet.SourceBuild.SmokeTests.SmokeTests.SmokeTestsScript() in /build/dotnet6-aZycfo/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/SmokeTests.cs:line 43
    Standard Output Messages:
   Executing: ./assets/smoke-tests/smoke-test.sh --dotnetDir /build/dotnet6-aZycfo/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet --projectOutput --archiveRestoredPackages --targetRid ubuntu.22.04-x64

  Results File: /build/dotnet6-aZycfo/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/TestResults/_Camelot-TITAN_2022-03-30_13_57_28.trx

  Failed!  - Failed:     1, Passed:     3, Skipped:     0, Total:     4, Duration: 39 m 33 s - /build/dotnet6-aZycfo/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/Microsoft.DotNet.SourceBuild.SmokeTests.dll (net6.0)
/build/dotnet6-aZycfo/dotnet6-6.0.103/build.proj(98,5): error MSB3073: The command "/build/dotnet6-aZycfo/dotnet6-6.0.103/.dotnet/dotnet test ./test/Microsoft.DotNet.SourceBuild.SmokeTests --logger:trx -c Release" exited with code 1.
make[1]: *** [debian/rules:129: override_dh_install] Error 1
make[1]: Leaving directory '/build/dotnet6-aZycfo/dotnet6-6.0.103'
make: *** [debian/rules:30: binary] Error 2

How can it be possible if the command stated here Executing: ./assets/smoke-tests/smoke-test.sh --dotnetDir /build/dotnet6-aZycfo/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet --projectOutput --archiveRestoredPackages --targetRid ubuntu.22.04-x64 worked manually before ? I put some prints out in the SmokeTest.Cs file for  executeResult.StdOut, executeResult.StdErr and executeResult.ExitCode (to see where the 1 code comes from) and got the following:
testing-smoke-0406114624/CSharp_console/bin/Debug/net6.0/publish/
Microsoft (R) Build Engine version 17.0.0+c9eb9dd64 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

/build/dotnet6-OIjAuR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet/sdk/6.0.103/MSBuild.dll -maxcpucount -property:SelfContained=True -property:_CommandLineDefinedSelfContained=true -property:RuntimeIdentifier=ubuntu.22.04-x64 -property:_CommandLineDefinedRuntimeIdentifier=true -restore -target:Publish -verbosity:m /bl:/build/dotnet6-OIjAuR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/assets/smoke-tests/testing-smoke-0406114624/logs/CSharp_console_local_nohttps_publish-self-contained-ubuntu.22.04-x64.binlog ./CSharp_console.csproj
Determining projects to restore...
StdOut: /build/dotnet6-OIjAuR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet/sdk/6.0.103/NuGet.targets(130,5): error : The local source '/build/dotnet6-OIjAuR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/artifacts/obj/x64/Release/blob-feed/packages/' doesn't exist. [/build/dotnet6-OIjAuR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/assets/smoke-tests/testing-smoke-0406114624/CSharp_console/CSharp_console.csproj] [/build/dotnet6-OIjAuR/dotnet6-6.0.103/build.proj]
StdErr:
ExitCode: 1
Failed Microsoft.DotNet.SourceBuild.SmokeTests.SmokeTests.SmokeTestsScript [6 s]
EXEC : error Message:  [/build/dotnet6-OIjAuR/dotnet6-6.0.103/build.proj]
Assert.Equal() Failure
Expected: 0
Actual:   1
Stack Trace:
at Microsoft.DotNet.SourceBuild.SmokeTests.SmokeTests.SmokeTestsScript() in /build/dotnet6-OIjAuR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/SmokeTests.cs:line 47
Standard Output Messages:
Executing: ./assets/smoke-tests/smoke-test.sh --dotnetDir /build/dotnet6-OIjAuR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet --projectOutput --archiveRestoredPackages --targetRid ubuntu.22.04-x64 --minimal --excludeOnlineTests --excludeOmniSharpTests

Results File: /build/dotnet6-OIjAuR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/TestResults/_Camelot-TITAN_2022-04-06_11_46_24.trx

Failed!  - Failed:     1, Passed:     3, Skipped:     0, Total:     4, Duration: 6 s - /build/dotnet6-OIjAuR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/Microsoft.DotNet.SourceBuild.SmokeTests.dll (net6.0)
/build/dotnet6-OIjAuR/dotnet6-6.0.103/build.proj(98,5): error MSB3073: The command "/build/dotnet6-OIjAuR/dotnet6-6.0.103/.dotnet/dotnet test ./test/Microsoft.DotNet.SourceBuild.SmokeTests --logger:trx -c Release" exited with code 1.
make[1]: *** [debian/rules:131: override_dh_install] Error 1
make[1]: Leaving directory '/build/dotnet6-OIjAuR/dotnet6-6.0.103'
make: *** [debian/rules:30: binary] Error 2

It seems another wrong artifact folder set elsewhere : /build/dotnet6-OIjAuR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet/sdk/6.0.103/NuGet.targets(130,5): error : The local source '/build/dotnet6-OIjAuR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/artifacts/obj/x64/Release/blob-feed/packages/' doesn't exist.  ... could it be a possibility?
Thanks in advance for looking into this.

@mirespace, I have a couple suggestions for the two different attempts you tried.

Can you provide clarity on what path you are executing ./assets/smoke-tests/smoke-test.sh from?  It appears that you maybe invoking the script directly out of the source directory.  This isn't support as you are discovering.  Try invoking it out of the build output directory /test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/assets/smoke-tests/smoke-test.sh.  When doing so, revert your change to the artifacts variable.
That said, I don't suspect this is the cause of the failure.  Can checked to see if the packages being loaded exist in /build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/../../../../artifacts/obj/x64/Release/blob-feed/packages/?  From the log, this is where it is looking for the Microsoft.NET.Test.Sdk, xunit, xunit.runner.visualstudio, and coverlet.collector packages

In the /build/dotnet6-OIjAuR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/artifacts/obj/x64/Release directory there should be a testing-smoke*/logs directory for each run.  Can you share the logs for the a run the demonstrates the problem?
When attempting to run the tests via the build.sh script, did you revert you changes to modify the artifacts variable.

Hi @MichaelSimons! I run all the process from the scratch again.

The complete command is <<PKGBUILDDIR>>/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/smoke-test.sh --dotnetDir /<<PKGBUILDDIR>>/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet --projectOutput --archiveRestoredPackages --targetRid ubuntu.22.04-x64 --minimal --excludeOnlineTests --excludeOmniSharpTests: is executing from the top of the source tree building source (dotnet6- as temporal folder for building):

(jammy-amd64-sbuild)root@Camelot-TITAN:/build/dotnet6-4woFhD/dotnet6-6.0.103# /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/smoke-test.sh --dotnetDir /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet --projectOutput --archiveRestoredPackages --targetRid ubuntu.22.04-x64 --minimal --excludeOnlineTests --excludeOmniSharpTests

I didn't realize when open the issue that it fails because the  bin folder is not present:
/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/smoke-test.sh: line 485: /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet/dotnet: No such file or directory

because it would be done by ./build.sh --run-smoke-test (I guess this is a step before invoking the smoke-test.sh script).
so, directly, it can't be called.

I reverted the change to the artifacts variable this time, and I executed from the top of the source tree the build command:

(jammy-amd64-sbuild)root@Camelot-TITAN:/build/dotnet6-4woFhD/dotnet6-6.0.103# VERBOSE=1 ./build.sh --run-smoke-test

It failed a first time for the xUnit.net test:
(jammy-amd64-sbuild)root@Camelot-TITAN:/build/dotnet6-4woFhD/dotnet6-6.0.103# VERBOSE=1 ./build.sh --run-smoke-test
Found bootstrap SDK 6.0.102, bootstrap Arcade 6.0.0-beta.21609.4, bootstrap SourceLink 1.1.0-beta-21480-02
Microsoft (R) Build Engine version 17.0.0+c9eb9dd64 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

/build/dotnet6-4woFhD/dotnet6-6.0.103/.dotnet/sdk/6.0.102/MSBuild.dll /bl:/build/dotnet6-4woFhD/dotnet6-6.0.103/artifacts/log/Debug/BuildTests_0413105044.binlog /clp:v=m /fileLoggerParameters:LogFile=/build/dotnet6-4woFhD/dotnet6-6.0.103/artifacts/logs/BuildTests_0413105044.log /flp:v=detailed /t:RunSmokeTest /build/dotnet6-4woFhD/dotnet6-6.0.103/build.proj
[SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.Sdk. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
[SourceBuiltSdkResolver] Looking for SDK Microsoft.DotNet.Arcade.Sdk. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
[SourceBuiltSdkResolver] Overriding Microsoft.DotNet.Arcade.Sdk with 'ARCADE'
[SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadAutoImportPropsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
[SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.ILLink.Tasks. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
[SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadManifestTargetsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'

  Welcome to .NET 6.0!
  ---------------------
  SDK Version: 6.0.102

  ----------------
  Installed an ASP.NET Core HTTPS development certificate.
  To trust the certificate run 'dotnet dev-certs https --trust' (Windows and macOS only).
  Learn about HTTPS: https://aka.ms/dotnet-https
  ----------------
  Write your first app: https://aka.ms/dotnet-hello-world
  Find out what's new: https://aka.ms/dotnet-whats-new
  Explore documentation: https://aka.ms/dotnet-docs
  Report issues and find source on GitHub: https://github.com/dotnet/core
  Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
  --------------------------------------------------------------------------------------
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.Sdk. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadAutoImportPropsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.ILLink.Tasks. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadManifestTargetsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
    Determining projects to restore...
    Restored /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/Microsoft.DotNet.SourceBuild.SmokeTests.csproj (in 4.2 sec).
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.Sdk. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadAutoImportPropsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.ILLink.Tasks. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadManifestTargetsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
    Microsoft.DotNet.SourceBuild.SmokeTests -> /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/Microsoft.DotNet.SourceBuild.SmokeTests.dll
  Test run for /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/Microsoft.DotNet.SourceBuild.SmokeTests.dll (.NETCoreApp,Version=v6.0)
  Microsoft (R) Test Execution Command Line Tool Version 17.0.0
  Copyright (c) Microsoft Corporation.  All rights reserved.

  Starting test execution, please wait...
  A total of 1 test files matched the specified pattern.
  [xUnit.net 00:00:40.08]     Microsoft.DotNet.SourceBuild.SmokeTests.SmokeTests.SmokeTestsScript [FAIL]
    Failed Microsoft.DotNet.SourceBuild.SmokeTests.SmokeTests.SmokeTestsScript [39 s]
EXEC : error Message:  [/build/dotnet6-4woFhD/dotnet6-6.0.103/build.proj]
     Assert.Equal() Failure
  Expected: 0
  Actual:   1
    Stack Trace:
       at Microsoft.DotNet.SourceBuild.SmokeTests.SmokeTests.SmokeTestsScript() in /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/SmokeTests.cs:line 43
    Standard Output Messages:
   Executing: ./assets/smoke-tests/smoke-test.sh --dotnetDir /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet --projectOutput --archiveRestoredPackages --targetRid ubuntu.22.04-x64 --minimal --excludeOnlineTests --excludeOmniSharpTests

  Results File: /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/TestResults/_Camelot-TITAN_2022-04-13_10_50_56.trx

  Failed!  - Failed:     1, Passed:     3, Skipped:     0, Total:     4, Duration: 36 s - /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/Microsoft.DotNet.SourceBuild.SmokeTests.dll (net6.0)
/build/dotnet6-4woFhD/dotnet6-6.0.103/build.proj(98,5): error MSB3073: The command "/build/dotnet6-4woFhD/dotnet6-6.0.103/.dotnet/dotnet test ./test/Microsoft.DotNet.SourceBuild.SmokeTests --logger:trx -c Release" exited with code 1.

And, no, the Microsoft.NET.Test.Sdk, xunit, xunit.runner.visualstudio, and coverlet.collector packages are not in /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/assets/smoke-tests/../../../../../../../artifacts/obj/x64/Release/blob-feed/packages/. So, yes, that's the reason of failing, but... if the parameters are --minimal and --excludeOnlineTests everything needed to run the test shouldn't already be there and not need to be downloaded?

The only error I see in that files is related with the artifacts folder location:

(jammy-amd64-sbuild)root@Camelot-TITAN:/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests# ack error testing-smoke-041*/logs/*.log
/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet/sdk/6.0.103/NuGet.targets(130,5): error : The local source '/artifacts/obj/x64/Release/blob-feed/packages/' doesn't exist. [/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0413110747/CSharp_console/CSharp_console.csproj]
(jammy-amd64-sbuild)root@Camelot-TITAN:/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests#

complete file is  (indeed, only this log has been generated):
(jammy-amd64-sbuild)root@Camelot-TITAN:/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests# cat testing-smoke-0413110747/logs/smoke-test.log
    running new

Welcome to .NET 6.0!
---------------------
SDK Version: 6.0.103

----------------
Installed an ASP.NET Core HTTPS development certificate.
To trust the certificate run 'dotnet dev-certs https --trust' (Windows and macOS only).
Learn about HTTPS: https://aka.ms/dotnet-https
----------------
Write your first app: https://aka.ms/dotnet-hello-world
Find out what's new: https://aka.ms/dotnet-whats-new
Explore documentation: https://aka.ms/dotnet-docs
Report issues and find source on GitHub: https://github.com/dotnet/core
Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
--------------------------------------------------------------------------------------
The template "Console App" was created successfully.
    new succeeded
    running restore
/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet/sdk/6.0.103/MSBuild.dll -nologo -maxcpucount -target:Restore -verbosity:m /bl:/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0413110747/logs/CSharp_console_local_nohttps_restore.binlog ./CSharp_console.csproj
  Determining projects to restore...
  Restored /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0413110747/CSharp_console/CSharp_console.csproj (in 71 ms).
    restore succeeded
    running build
Microsoft (R) Build Engine version 17.0.0+c9eb9dd64 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet/sdk/6.0.103/MSBuild.dll -consoleloggerparameters:Summary -maxcpucount -restore -verbosity:m /bl:/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0413110747/logs/CSharp_console_local_nohttps_build.binlog ./CSharp_console.csproj
  Determining projects to restore...
  All projects are up-to-date for restore.
  CSharp_console -> /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0413110747/CSharp_console/bin/Debug/net6.0/CSharp_console.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:01.02
    build succeeded
    running run
Hello, World!
    run succeeded
    running multi-rid-publish
Microsoft (R) Build Engine version 17.0.0+c9eb9dd64 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet/sdk/6.0.103/MSBuild.dll -maxcpucount -property:SelfContained=False -property:_CommandLineDefinedSelfContained=true -restore -target:Publish -verbosity:m /bl:/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0413110747/logs/CSharp_console_local_nohttps_publish-fx-dep.binlog ./CSharp_console.csproj
  Determining projects to restore...
  All projects are up-to-date for restore.
  CSharp_console -> /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0413110747/CSharp_console/bin/Debug/net6.0/CSharp_console.dll
  CSharp_console -> /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0413110747/CSharp_console/bin/Debug/net6.0/publish/
Microsoft (R) Build Engine version 17.0.0+c9eb9dd64 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet/sdk/6.0.103/MSBuild.dll -maxcpucount -property:SelfContained=True -property:_CommandLineDefinedSelfContained=true -property:RuntimeIdentifier=ubuntu.22.04-x64 -property:_CommandLineDefinedRuntimeIdentifier=true -restore -target:Publish -verbosity:m /bl:/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0413110747/logs/CSharp_console_local_nohttps_publish-self-contained-ubuntu.22.04-x64.binlog ./CSharp_console.csproj
  Determining projects to restore...
/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet/sdk/6.0.103/NuGet.targets(130,5): error : The local source '/artifacts/obj/x64/Release/blob-feed/packages/' doesn't exist. [/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0413110747/CSharp_console/CSharp_console.csproj]

[Triage] Closing as the recently changes to the smoke-tests removed the script.  The ability to run tests in an offline mode is tracked with https://github.com/dotnet/source-build/issues/2844.


Unable to start substitute 😡🤬

I get this message: unable to start substitute,
My tweakcompatible deleted substitute and made my cydia’s sources not to index anymore
I installed it again but now it’s just TRASH.

Restore your iPhone and you will be good to go, besides of that don't write about problems you can fix yourself here. Moreover, use the bug reporting template and try solving such a petty problem yourself.
Please close @Cryptiiiic


added token claims

Added claims for the access/refresh token and other misc info sent back from Spotify.
Helpful for Spotify API requests if stored against the authenticated user.

Lets talk about this further in #250 before we continue with this patch.


pytest-codspeed v3.0.0

Checklist

[x] Used a personal fork of the feedstock to propose changes
[ ] ~Bumped the build number (if the version is unchanged)~
[x] Reset the build number to 0 (if the version changed)
[ ] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x] Ensured the license file is being packaged.

Closes #1

@conda-forge-admin, please rerender

Hi! This is the friendly automated conda-forge-linting service.
I wanted to let you know that I linted all conda-recipes in your PR (recipe/meta.yaml) and found some lint.
Here's what I've got...
For recipe/meta.yaml:

noarch packages can't have selectors. If the selectors are necessary, please remove noarch: python.

For recipe/meta.yaml:

PyPI default URL is now pypi.org, and not pypi.io. You may want to update the default source url.

@conda-forge-admin, please rerender

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe/meta.yaml) and found it was in an excellent condition.

Hi! This is the friendly automated conda-forge-webservice.
I tried to rerender for you but ran into some issues. Please check the output logs of the GitHub actions workflow below for more details. You can also ping conda-forge/core for further assistance or you can try rerendering locally.
The following suggestions might help debug any issues:

Is the recipe/{meta.yaml,recipe.yaml} file valid?
If there is a recipe/conda-build-config.yaml file in the feedstock make sure that it is compatible with the current global pinnnings.
Is the fork used for this PR on an organization or user GitHub account? Automated rerendering via the webservices admin bot only works for user GitHub accounts.

This message was generated by GitHub actions workflow run https://github.com/conda-forge/conda-forge-webservices/actions/runs/11632129111.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe/meta.yaml) and found it was in an excellent condition.
I do have some suggestions for making it better though...
For recipe/meta.yaml:

ℹ noarch: python recipes should usually follow the syntax in our documentation for specifying the Python version.

For the host section of the recipe, you should usually use python {{ python_min }} for the python entry.
For the run section of the recipe, you should usually use python >={{ python_min }} for the python entry.
For the test.requires section of the recipe, you should usually use python {{ python_min }} for the python entry.
If the package requires a newer Python version than the currently supported minimum version on conda-forge, you can override the python_min variable by adding a Jinja2 set statement at the top of your recipe (or using an equivalent context variable for v1 recipes).

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe/meta.yaml) and found it was in an excellent condition.

@conda-forge-admin, please rerender


Backbone done

I think I have done the final damage to make the move to backbone. I'm sure it can be tightened up some but it should be in pretty good shape. We could break models/views into more files but if that happens we probably want to use require.js.
This needs to be given a good test before going into master. I also think there has been enough done in master that it will be a messy merge. There is also a d&d change that should probably be retained but won't merge in without help.

btw, I closed the pull request for drag and drop to prevent conflicts with changes like this and to give me a little more time to polish up that script).  I'll redo a pull request later after this goes through.

thx bgrins. appreciate that.


Python connectivity polling thread not shut down

Here is a short test case that takes about 10 seconds to finish (when it should take no time):
import grpc
import threading
import time

def test():
    channel = grpc.insecure_channel('a:2')
    channel.subscribe(lambda s: s)
    time.sleep(2)

def main():
    t = threading.Thread(target=test)
    t.daemon = True
    t.start()
    t.join()

main()

I think this is because _poll_connectivity is blocked on completion_queue.poll().
Here is verbose log indicating the leaked channel:
I0928 11:58:16.165395342   23091 ev_epoll_linux.c:85]        epoll engine will be using signal: 36
D0928 11:58:16.165433960   23091 ev_posix.c:106]             Using polling engine: epoll
I0928 11:58:16.165445984   23091 init.c:193]                 grpc_init(void)
I0928 11:58:16.173584449   23092 metadata_array.c:42]        grpc_metadata_array_init(array=0x7fb25c132470)
I0928 11:58:16.174708966   23092 metadata_array.c:42]        grpc_metadata_array_init(array=0x7fb25b7a6d80)
I0928 11:58:16.175100315   23092 channel_create.c:219]       grpc_insecure_channel_create(target=0x7fb25e79689c, args=0x7fb25c144ba0, reserved=(nil))
I0928 11:58:16.175141727   23092 completion_queue.c:139]     grpc_completion_queue_create(reserved=(nil))
I0928 11:58:16.175201420   23092 completion_queue.c:139]     grpc_completion_queue_create(reserved=(nil))
I0928 11:58:16.175326281   23093 channel_connectivity.c:53]  grpc_channel_check_connectivity_state(channel=0x7fb2540054c0, try_to_connect=0)
D0928 11:58:16.175346906   23093 connectivity_state.c:92]    CONWATCH: 0x7fb254005608 client_channel: get IDLE
poll
I0928 11:58:16.175570405   23093 channel_connectivity.c:196] grpc_channel_watch_connectivity_state(channel=0x7fb2540054c0, last_observed_state=0, deadline=gpr_timespec { tv_sec:<PHONE_NUMBER>, tv_nsec: 375560998, clock_type: 1 }, cq=0x7fb254045f40, tag=0x7fb25c1a7fa0)
D0928 11:58:16.175584477   23093 connectivity_state.c:109]   CONWATCH: 0x7fb254005608 client_channel: from IDLE [cur=IDLE] notify=0x7fb24c000d88
I0928 11:58:16.175594006   23093 completion_queue.c:333]     grpc_completion_queue_next(cc=0x7fb254045f40, deadline=gpr_timespec { tv_sec:<PHONE_NUMBER>, tv_nsec: 375591672, clock_type: 1 }, reserved=(nil))
I0928 11:58:16.375905029   23093 completion_queue.c:394]     RETURN_EVENT[0x7fb254045f40]: QUEUE_TIMEOUT
I0928 11:58:16.375930727   23093 completion_queue.c:333]     grpc_completion_queue_next(cc=0x7fb254045f40, deadline=gpr_timespec { tv_sec:<PHONE_NUMBER>, tv_nsec: 575928767, clock_type: 1 }, reserved=(nil))
I0928 11:58:16.375947513   23093 completion_queue.c:252]     grpc_cq_end_op(exec_ctx=0x7fb25b7a4460, cc=0x7fb254045f40, tag=0x7fb25c1a7fa0, error={"created":"@1475078296.375939554","description":"Timed out waiting for connection state change","file":"src/core/ext/client_config/channel_connectivity.c","file_line":142}, done=0x7fb25d147fa0, done_arg=0x7fb24c000e30, storage=0x7fb24c000ed8)
E0928 11:58:16.375957054   23093 completion_queue.c:254]     Operation failed: tag=0x7fb25c1a7fa0, error={"created":"@1475078296.375939554","description":"Timed out waiting for connection state change","file":"src/core/ext/client_config/channel_connectivity.c","file_line":142}
I0928 11:58:16.375965519   23093 completion_queue.c:394]     RETURN_EVENT[0x7fb254045f40]: OP_COMPLETE: tag:0x7fb25c1a7fa0 ERROR
event 2
I0928 11:58:16.376007658   23093 channel_connectivity.c:196] grpc_channel_watch_connectivity_state(channel=0x7fb2540054c0, last_observed_state=0, deadline=gpr_timespec { tv_sec:<PHONE_NUMBER>, tv_nsec: 575999021, clock_type: 1 }, cq=0x7fb254045f40, tag=0x7fb25c1a7fa0)
D0928 11:58:16.376019659   23093 connectivity_state.c:109]   CONWATCH: 0x7fb254005608 client_channel: from IDLE [cur=IDLE] notify=0x7fb24c001598
I0928 11:58:16.376027681   23093 completion_queue.c:333]     grpc_completion_queue_next(cc=0x7fb254045f40, deadline=gpr_timespec { tv_sec:<PHONE_NUMBER>, tv_nsec: 576026149, clock_type: 1 }, reserved=(nil))
I0928 11:58:16.576335549   23093 completion_queue.c:394]     RETURN_EVENT[0x7fb254045f40]: QUEUE_TIMEOUT
I0928 11:58:16.576405074   23093 completion_queue.c:333]     grpc_completion_queue_next(cc=0x7fb254045f40, deadline=gpr_timespec { tv_sec:<PHONE_NUMBER>, tv_nsec: 776401187, clock_type: 1 }, reserved=(nil))
I0928 11:58:16.576435751   23093 completion_queue.c:252]     grpc_cq_end_op(exec_ctx=0x7fb25b7a4460, cc=0x7fb254045f40, tag=0x7fb25c1a7fa0, error={"created":"@1475078296.576419322","description":"Timed out waiting for connection state change","file":"src/core/ext/client_config/channel_connectivity.c","file_line":142}, done=0x7fb25d147fa0, done_arg=0x7fb24c001710, storage=0x7fb24c0017b8)
E0928 11:58:16.576462310   23093 completion_queue.c:254]     Operation failed: tag=0x7fb25c1a7fa0, error={"created":"@1475078296.576419322","description":"Timed out waiting for connection state change","file":"src/core/ext/client_config/channel_connectivity.c","file_line":142}
I0928 11:58:16.576470755   23093 completion_queue.c:394]     RETURN_EVENT[0x7fb254045f40]: OP_COMPLETE: tag:0x7fb25c1a7fa0 ERROR
event 2
I0928 11:58:16.576538519   23093 channel_connectivity.c:196] grpc_channel_watch_connectivity_state(channel=0x7fb2540054c0, last_observed_state=0, deadline=gpr_timespec { tv_sec:<PHONE_NUMBER>, tv_nsec: 776528120, clock_type: 1 }, cq=0x7fb254045f40, tag=0x7fb25c1a7fa0)
D0928 11:58:16.576549085   23093 connectivity_state.c:109]   CONWATCH: 0x7fb254005608 client_channel: from IDLE [cur=IDLE] notify=0x7fb24c0014f8
I0928 11:58:16.576560819   23093 completion_queue.c:333]     grpc_completion_queue_next(cc=0x7fb254045f40, deadline=gpr_timespec { tv_sec:<PHONE_NUMBER>, tv_nsec: 776559236, clock_type: 1 }, reserved=(nil))
I0928 11:58:16.776960187   23093 completion_queue.c:394]     RETURN_EVENT[0x7fb254045f40]: QUEUE_TIMEOUT
I0928 11:58:16.777006096   23093 completion_queue.c:333]     grpc_completion_queue_next(cc=0x7fb254045f40, deadline=gpr_timespec { tv_sec:<PHONE_NUMBER>, tv_nsec: 977002575, clock_type: 1 }, reserved=(nil))
I0928 11:58:16.777026819   23093 completion_queue.c:252]     grpc_cq_end_op(exec_ctx=0x7fb25b7a4460, cc=0x7fb254045f40, tag=0x7fb25c1a7fa0, error={"created":"@1475078296.777018118","description":"Timed out waiting for connection state change","file":"src/core/ext/client_config/channel_connectivity.c","file_line":142}, done=0x7fb25d147fa0, done_arg=0x7fb24c001900, storage=0x7fb24c0019a8)
E0928 11:58:16.777038003   23093 completion_queue.c:254]     Operation failed: tag=0x7fb25c1a7fa0, error={"created":"@1475078296.777018118","description":"Timed out waiting for connection state change","file":"src/core/ext/client_config/channel_connectivity.c","file_line":142}
I0928 11:58:16.777046842   23093 completion_queue.c:394]     RETURN_EVENT[0x7fb254045f40]: OP_COMPLETE: tag:0x7fb25c1a7fa0 ERROR
event 2
I0928 11:58:16.777105362   23093 channel_connectivity.c:196] grpc_channel_watch_connectivity_state(channel=0x7fb2540054c0, last_observed_state=0, deadline=gpr_timespec { tv_sec:<PHONE_NUMBER>, tv_nsec: 977086067, clock_type: 1 }, cq=0x7fb254045f40, tag=0x7fb25c1a7fa0)
D0928 11:58:16.777129089   23093 connectivity_state.c:109]   CONWATCH: 0x7fb254005608 client_channel: from IDLE [cur=IDLE] notify=0x7fb24c001618
I0928 11:58:16.777137282   23093 completion_queue.c:333]     grpc_completion_queue_next(cc=0x7fb254045f40, deadline=gpr_timespec { tv_sec:<PHONE_NUMBER>, tv_nsec: 977135693, clock_type: 1 }, reserved=(nil))
I0928 11:58:16.977432674   23093 completion_queue.c:394]     RETURN_EVENT[0x7fb254045f40]: QUEUE_TIMEOUT
I0928 11:58:16.977479252   23093 completion_queue.c:333]     grpc_completion_queue_next(cc=0x7fb254045f40, deadline=gpr_timespec { tv_sec:<PHONE_NUMBER>, tv_nsec: 177475755, clock_type: 1 }, reserved=(nil))
I0928 11:58:16.977501173   23093 completion_queue.c:252]     grpc_cq_end_op(exec_ctx=0x7fb25b7a4460, cc=0x7fb254045f40, tag=0x7fb25c1a7fa0, error={"created":"@1475078296.977491819","description":"Timed out waiting for connection state change","file":"src/core/ext/client_config/channel_connectivity.c","file_line":142}, done=0x7fb25d147fa0, done_arg=0x7fb24c001a90, storage=0x7fb24c001b38)
E0928 11:58:16.977512707   23093 completion_queue.c:254]     Operation failed: tag=0x7fb25c1a7fa0, error={"created":"@1475078296.977491819","description":"Timed out waiting for connection state change","file":"src/core/ext/client_config/channel_connectivity.c","file_line":142}
I0928 11:58:16.977521488   23093 completion_queue.c:394]     RETURN_EVENT[0x7fb254045f40]: OP_COMPLETE: tag:0x7fb25c1a7fa0 ERROR
event 2
I0928 11:58:16.977570995   23093 channel_connectivity.c:196] grpc_channel_watch_connectivity_state(channel=0x7fb2540054c0, last_observed_state=0, deadline=gpr_timespec { tv_sec:<PHONE_NUMBER>, tv_nsec: 177560091, clock_type: 1 }, cq=0x7fb254045f40, tag=0x7fb25c1a7fa0)
D0928 11:58:16.977581658   23093 connectivity_state.c:109]   CONWATCH: 0x7fb254005608 client_channel: from IDLE [cur=IDLE] notify=0x7fb24c0015d8
I0928 11:58:16.977590119   23093 completion_queue.c:333]     grpc_completion_queue_next(cc=0x7fb254045f40, deadline=gpr_timespec { tv_sec:<PHONE_NUMBER>, tv_nsec: 177588583, clock_type: 1 }, reserved=(nil))
I0928 11:58:17.178106142   23093 completion_queue.c:394]     RETURN_EVENT[0x7fb254045f40]: QUEUE_TIMEOUT
I0928 11:58:17.178153924   23093 completion_queue.c:333]     grpc_completion_queue_next(cc=0x7fb254045f40, deadline=gpr_timespec { tv_sec:<PHONE_NUMBER>, tv_nsec: 378150786, clock_type: 1 }, reserved=(nil))
I0928 11:58:17.178175281   23093 completion_queue.c:252]     grpc_cq_end_op(exec_ctx=0x7fb25b7a4460, cc=0x7fb254045f40, tag=0x7fb25c1a7fa0, error={"created":"@1475078297.178166267","description":"Timed out waiting for connection state change","file":"src/core/ext/client_config/channel_connectivity.c","file_line":142}, done=0x7fb25d147fa0, done_arg=0x7fb24c001b70, storage=0x7fb24c001c18)
E0928 11:58:17.178186168   23093 completion_queue.c:254]     Operation failed: tag=0x7fb25c1a7fa0, error={"created":"@1475078297.178166267","description":"Timed out waiting for connection state change","file":"src/core/ext/client_config/channel_connectivity.c","file_line":142}
I0928 11:58:17.178195215   23093 completion_queue.c:394]     RETURN_EVENT[0x7fb254045f40]: OP_COMPLETE: tag:0x7fb25c1a7fa0 ERROR
event 2
I0928 11:58:17.178244820   23093 channel_connectivity.c:196] grpc_channel_watch_connectivity_state(channel=0x7fb2540054c0, last_observed_state=0, deadline=gpr_timespec { tv_sec:<PHONE_NUMBER>, tv_nsec: 378235101, clock_type: 1 }, cq=0x7fb254045f40, tag=0x7fb25c1a7fa0)
D0928 11:58:17.178275360   23093 connectivity_state.c:109]   CONWATCH: 0x7fb254005608 client_channel: from IDLE [cur=IDLE] notify=0x7fb24c001c68
I0928 11:58:17.178283305   23093 completion_queue.c:333]     grpc_completion_queue_next(cc=0x7fb254045f40, deadline=gpr_timespec { tv_sec:<PHONE_NUMBER>, tv_nsec: 378281708, clock_type: 1 }, reserved=(nil))
I0928 11:58:17.378649343   23093 completion_queue.c:394]     RETURN_EVENT[0x7fb254045f40]: QUEUE_TIMEOUT
I0928 11:58:17.378693082   23093 completion_queue.c:333]     grpc_completion_queue_next(cc=0x7fb254045f40, deadline=gpr_timespec { tv_sec:<PHONE_NUMBER>, tv_nsec: 578690312, clock_type: 1 }, reserved=(nil))
I0928 11:58:17.378711557   23093 completion_queue.c:252]     grpc_cq_end_op(exec_ctx=0x7fb25b7a4460, cc=0x7fb254045f40, tag=0x7fb25c1a7fa0, error={"created":"@1475078297.378703464","description":"Timed out waiting for connection state change","file":"src/core/ext/client_config/channel_connectivity.c","file_line":142}, done=0x7fb25d147fa0, done_arg=0x7fb24c001d40, storage=0x7fb24c001de8)
E0928 11:58:17.378719624   23093 completion_queue.c:254]     Operation failed: tag=0x7fb25c1a7fa0, error={"created":"@1475078297.378703464","description":"Timed out waiting for connection state change","file":"src/core/ext/client_config/channel_connectivity.c","file_line":142}
I0928 11:58:17.378728196   23093 completion_queue.c:394]     RETURN_EVENT[0x7fb254045f40]: OP_COMPLETE: tag:0x7fb25c1a7fa0 ERROR
event 2
I0928 11:58:17.378776655   23093 channel_connectivity.c:196] grpc_channel_watch_connectivity_state(channel=0x7fb2540054c0, last_observed_state=0, deadline=gpr_timespec { tv_sec:<PHONE_NUMBER>, tv_nsec: 578767061, clock_type: 1 }, cq=0x7fb254045f40, tag=0x7fb25c1a7fa0)
D0928 11:58:17.378798751   23093 connectivity_state.c:109]   CONWATCH: 0x7fb254005608 client_channel: from IDLE [cur=IDLE] notify=0x7fb24c001808
I0928 11:58:17.378817192   23093 completion_queue.c:333]     grpc_completion_queue_next(cc=0x7fb254045f40, deadline=gpr_timespec { tv_sec:<PHONE_NUMBER>, tv_nsec: 578815619, clock_type: 1 }, reserved=(nil))
I0928 11:58:17.579089856   23093 completion_queue.c:394]     RETURN_EVENT[0x7fb254045f40]: QUEUE_TIMEOUT
I0928 11:58:17.579140479   23093 completion_queue.c:333]     grpc_completion_queue_next(cc=0x7fb254045f40, deadline=gpr_timespec { tv_sec:<PHONE_NUMBER>, tv_nsec: 779138417, clock_type: 1 }, reserved=(nil))
I0928 11:58:17.579158378   23093 completion_queue.c:252]     grpc_cq_end_op(exec_ctx=0x7fb25b7a4460, cc=0x7fb254045f40, tag=0x7fb25c1a7fa0, error={"created":"@1475078297.579150586","description":"Timed out waiting for connection state change","file":"src/core/ext/client_config/channel_connectivity.c","file_line":142}, done=0x7fb25d147fa0, done_arg=0x7fb24c001eb0, storage=0x7fb24c001f58)
E0928 11:58:17.579166506   23093 completion_queue.c:254]     Operation failed: tag=0x7fb25c1a7fa0, error={"created":"@1475078297.579150586","description":"Timed out waiting for connection state change","file":"src/core/ext/client_config/channel_connectivity.c","file_line":142}
I0928 11:58:17.579174617   23093 completion_queue.c:394]     RETURN_EVENT[0x7fb254045f40]: OP_COMPLETE: tag:0x7fb25c1a7fa0 ERROR
event 2
I0928 11:58:17.579223312   23093 channel_connectivity.c:196] grpc_channel_watch_connectivity_state(channel=0x7fb2540054c0, last_observed_state=0, deadline=gpr_timespec { tv_sec:<PHONE_NUMBER>, tv_nsec: 779214143, clock_type: 1 }, cq=0x7fb254045f40, tag=0x7fb25c1a7fa0)
D0928 11:58:17.579243908   23093 connectivity_state.c:109]   CONWATCH: 0x7fb254005608 client_channel: from IDLE [cur=IDLE] notify=0x7fb24c0014b8
I0928 11:58:17.579262697   23093 completion_queue.c:333]     grpc_completion_queue_next(cc=0x7fb254045f40, deadline=gpr_timespec { tv_sec:<PHONE_NUMBER>, tv_nsec: 779261079, clock_type: 1 }, reserved=(nil))
I0928 11:58:17.779758969   23093 completion_queue.c:394]     RETURN_EVENT[0x7fb254045f40]: QUEUE_TIMEOUT
I0928 11:58:17.779806297   23093 completion_queue.c:333]     grpc_completion_queue_next(cc=0x7fb254045f40, deadline=gpr_timespec { tv_sec:<PHONE_NUMBER>, tv_nsec: 979803517, clock_type: 1 }, reserved=(nil))
I0928 11:58:17.779828193   23093 completion_queue.c:252]     grpc_cq_end_op(exec_ctx=0x7fb25b7a4460, cc=0x7fb254045f40, tag=0x7fb25c1a7fa0, error={"created":"@1475078297.779819122","description":"Timed out waiting for connection state change","file":"src/core/ext/client_config/channel_connectivity.c","file_line":142}, done=0x7fb25d147fa0, done_arg=0x7fb24c002020, storage=0x7fb24c0020c8)
E0928 11:58:17.779840007   23093 completion_queue.c:254]     Operation failed: tag=0x7fb25c1a7fa0, error={"created":"@1475078297.779819122","description":"Timed out waiting for connection state change","file":"src/core/ext/client_config/channel_connectivity.c","file_line":142}
I0928 11:58:17.779848438   23093 completion_queue.c:394]     RETURN_EVENT[0x7fb254045f40]: OP_COMPLETE: tag:0x7fb25c1a7fa0 ERROR
event 2
I0928 11:58:17.779907708   23093 channel_connectivity.c:196] grpc_channel_watch_connectivity_state(channel=0x7fb2540054c0, last_observed_state=0, deadline=gpr_timespec { tv_sec:<PHONE_NUMBER>, tv_nsec: 979887962, clock_type: 1 }, cq=0x7fb254045f40, tag=0x7fb25c1a7fa0)
D0928 11:58:17.779931854   23093 connectivity_state.c:109]   CONWATCH: 0x7fb254005608 client_channel: from IDLE [cur=IDLE] notify=0x7fb24c001658
I0928 11:58:17.779939889   23093 completion_queue.c:333]     grpc_completion_queue_next(cc=0x7fb254045f40, deadline=gpr_timespec { tv_sec:<PHONE_NUMBER>, tv_nsec: 979938352, clock_type: 1 }, reserved=(nil))
I0928 11:58:17.980208164   23093 completion_queue.c:394]     RETURN_EVENT[0x7fb254045f40]: QUEUE_TIMEOUT
I0928 11:58:17.980254747   23093 completion_queue.c:333]     grpc_completion_queue_next(cc=0x7fb254045f40, deadline=gpr_timespec { tv_sec:<PHONE_NUMBER>, tv_nsec: 180251855, clock_type: 1 }, reserved=(nil))
I0928 11:58:17.980276852   23093 completion_queue.c:252]     grpc_cq_end_op(exec_ctx=0x7fb25b7a4460, cc=0x7fb254045f40, tag=0x7fb25c1a7fa0, error={"created":"@1475078297.980267467","description":"Timed out waiting for connection state change","file":"src/core/ext/client_config/channel_connectivity.c","file_line":142}, done=0x7fb25d147fa0, done_arg=0x7fb24c002170, storage=0x7fb24c002218)
E0928 11:58:17.980288421   23093 completion_queue.c:254]     Operation failed: tag=0x7fb25c1a7fa0, error={"created":"@1475078297.980267467","description":"Timed out waiting for connection state change","file":"src/core/ext/client_config/channel_connectivity.c","file_line":142}
I0928 11:58:17.980297219   23093 completion_queue.c:394]     RETURN_EVENT[0x7fb254045f40]: OP_COMPLETE: tag:0x7fb25c1a7fa0 ERROR
event 2
I0928 11:58:17.980359202   23093 channel_connectivity.c:196] grpc_channel_watch_connectivity_state(channel=0x7fb2540054c0, last_observed_state=0, deadline=gpr_timespec { tv_sec:<PHONE_NUMBER>, tv_nsec: 180349111, clock_type: 1 }, cq=0x7fb254045f40, tag=0x7fb25c1a7fa0)
D0928 11:58:17.980402421   23093 connectivity_state.c:109]   CONWATCH: 0x7fb254005608 client_channel: from IDLE [cur=IDLE] notify=0x7fb24c001218
I0928 11:58:17.980411424   23093 completion_queue.c:333]     grpc_completion_queue_next(cc=0x7fb254045f40, deadline=gpr_timespec { tv_sec:<PHONE_NUMBER>, tv_nsec: 180409747, clock_type: 1 }, reserved=(nil))
I0928 11:58:18.177442602   23092 completion_queue.c:541]     grpc_completion_queue_shutdown(cc=0x7fb254047320)
I0928 11:58:18.177490205   23092 completion_queue.c:333]     grpc_completion_queue_next(cc=0x7fb254047320, deadline=gpr_timespec { tv_sec:<PHONE_NUMBER>854775807, tv_nsec: 0, clock_type: 1 }, reserved=(nil))
I0928 11:58:18.177499915   23092 completion_queue.c:394]     RETURN_EVENT[0x7fb254047320]: QUEUE_SHUTDOWN
I0928 11:58:18.177509352   23092 completion_queue.c:561]     grpc_completion_queue_destroy(cc=0x7fb254047320)
I0928 11:58:18.177514612   23092 completion_queue.c:541]     grpc_completion_queue_shutdown(cc=0x7fb254047320)
I0928 11:58:18.180852605   23093 completion_queue.c:394]     RETURN_EVENT[0x7fb254045f40]: QUEUE_TIMEOUT
I0928 11:58:18.181502776   23091 metadata_array.c:47]        grpc_metadata_array_destroy(array=0x7fb25c132470)
I0928 11:58:18.182613728   23091 metadata_array.c:47]        grpc_metadata_array_destroy(array=0x7fb25b7a6d80)
I0928 11:58:18.182979920   23091 init.c:198]                 grpc_shutdown(void)
I0928 11:58:18.183024436   23091 completion_queue.c:252]     grpc_cq_end_op(exec_ctx=0x7fff67b836f0, cc=0x7fb254045f40, tag=0x7fb25c1a7fa0, error={"created":"@1475078298.182997820","description":"Shutting down timer system","file":"src/core/lib/iomgr/timer.c","file_line":383}, done=0x7fb25d147fa0, done_arg=0x7fb24c002350, storage=0x7fb24c0023f8)
E0928 11:58:18.183036620   23091 completion_queue.c:254]     Operation failed: tag=0x7fb25c1a7fa0, error={"created":"@1475078298.182997820","description":"Shutting down timer system","file":"src/core/lib/iomgr/timer.c","file_line":383}
D0928 11:58:19.184326417   23091 iomgr.c:101]                Waiting for 1 iomgr objects to be destroyed
D0928 11:58:20.185651776   23091 iomgr.c:101]                Waiting for 1 iomgr objects to be destroyed
D0928 11:58:21.186716792   23091 iomgr.c:101]                Waiting for 1 iomgr objects to be destroyed
D0928 11:58:22.188056493   23091 iomgr.c:101]                Waiting for 1 iomgr objects to be destroyed
D0928 11:58:23.189387562   23091 iomgr.c:101]                Waiting for 1 iomgr objects to be destroyed
D0928 11:58:24.190672565   23091 iomgr.c:101]                Waiting for 1 iomgr objects to be destroyed
D0928 11:58:25.191880335   23091 iomgr.c:101]                Waiting for 1 iomgr objects to be destroyed
D0928 11:58:26.193087628   23091 iomgr.c:101]                Waiting for 1 iomgr objects to be destroyed
D0928 11:58:27.194348205   23091 iomgr.c:101]                Waiting for 1 iomgr objects to be destroyed
D0928 11:58:28.195563886   23091 iomgr.c:120]                Failed to free 1 iomgr objects before shutdown deadline: memory leaks are likely
D0928 11:58:28.195607794   23091 iomgr.c:82]                 LEAKED OBJECT: workqueue:0x7fb24c0010a0 fd=4 0x7fb24c001190
D0928 11:58:28.195654010   23091 metadata.c:238]             WARNING: 1 metadata elements were leaked
D0928 11:58:28.195663918   23091 metadata.c:251]             WARNING: 1 metadata strings were leaked
D0928 11:58:28.195669669   23091 metadata.c:255]             LEAKED: a:2

This is tested on v1.0.0.
I tried changing _moot in _channel.py to shut down the polling thread's completion queue. This was a bit intrusive so I wasn't sure if it is the correct fix.

From reading your output, it looks like the program attempts to exit after 2 seconds, but finds a memory leak and waits ~8s longer for the program to clean it up.
This is likely due to some cyclic reference and objects not being destroyed.  Because shutting down the polling threads completion queue fixes the issue, the cyclic reference is probably somewhere in that area.  I'll take a look

Deferring; there may be something here that we can improve but this doesn't seem high-enough priority to target to 1.1.0.

Is this related to #9688? If yes, as a user, I'd say this is VERY high priority.

I would almost guarantee it's related and what's more this basically spells out the solution to both. They're relying on python objects to be deleted because in the del methods they do cleanup of c objects via reference counts. In some cases there is a cycle which means the objects won't be cleaned up until the garbage collector runs which is non-deterministic.

I'm also experiencing this issue. I believe every python grpc-server user will experience this.

Something like this would fix the issue, the call needs to be explicitly cleaned up, you cannot rely on __dealloc__ methods from cython to be called since they can be GC dependent.
diff --git a/src/python/grpcio/grpc/_cython/_cygrpc/call.pyx.pxi b/src/python/grpcio/grpc/_cython/_cygrpc/call.pyx.pxi
index 752fb330d0..06439643f1 100644
--- a/src/python/grpcio/grpc/_cython/_cygrpc/call.pyx.pxi
+++ b/src/python/grpcio/grpc/_cython/_cygrpc/call.pyx.pxi
@@ -89,6 +89,11 @@ cdef class Call:
       gpr_free(peer)
     return result

+  def shutdown(self):
+    if self.c_call != NULL:
+      grpc_call_unref(self.c_call)
+      self.c_call = NULL
+
   def __dealloc__(self):
     if self.c_call != NULL:
       grpc_call_unref(self.c_call)
diff --git a/src/python/grpcio/grpc/_server.py b/src/python/grpcio/grpc/_server.py
index cd59b07c04..18f36bf6cb 100644
--- a/src/python/grpcio/grpc/_server.py
+++ b/src/python/grpcio/grpc/_server.py
@@ -154,6 +154,7 @@ def _receive_close_on_server(state):
                     0].received_cancelled:
                 state.client = _CANCELLED
             elif state.client is _OPEN:
+                receive_close_on_server_event.operation_call.shutdown()
                 state.client = _CLOSED
             state.condition.notify_all()
             return _possibly_finish_call(state, _RECEIVE_CLOSE_ON_SERVER_TOKEN)
@@ -709,7 +710,7 @@ def _serve(state):
         # We want to force the deletion of the previous event
         # ~before~ we poll again; if the event has a reference
         # to a shutdown Call object, this can induce spinlock.
-        event = None
+        # event = None

 def _stop(state, grace):

@kpayson64: please reassess - should this be closed because there is no longer a connectivity polling thread or retitled because it's actually about something else?

I think this issue can be closed now that we have removed the background polling thread.


Try to integrate the docs of Grammar

I will try to integrate the docs of Grammar a week later or so. Because Grammar is the most attractive of Perl 6 to me now, but its doc is a little lack. What's worse, they are scattered in several pages.
At first I want to involve both Regex and Grammar, but it's too hard to make it in one pass.
Below are the pods considered to be integrated:
language/grammars and language/grammar_tutorial, these two should be one.
Besides type/Grammar, type/Match and type/Cursor may be involved. Because #1646 supposed what to document below language or type, the current text will be rearranged. Also I hope it be an example when filing document.
Anyway, Grammar is one of the most awesome features supplied by rakudo/Perl 6.
Any suggestion like what you want to be mentioned(such as method make/made) or how it's supposed to be organized is welcome.
Any question/problem is welcome, too.

Maybe @drforr's blogpost can come handy (the Action part needs to be more verbose, though):
http://theperlfisher.blogspot.be/2016/02/from-regular-expressions-to-grammars-pt.html
http://theperlfisher.blogspot.be/2016/02/from-regular-expressions-to-grammars-pt_13.html
http://theperlfisher.blogspot.be/2016/02/from-regular-expressions-to-grammars-pt_20.html
http://theperlfisher.blogspot.be/2016/02/from-regular-expressions-to-grammars-pt_28.html

@nxadm I've glanced over those blogs. It starts with regex, but as mentioned above, I prefer to integrate only docs of grammar and assume those who read grammar docs at least know the basic of regex.
As you say, Actions are important. Users parse text by grammar almost want to perform some actions on the fly. The most common action is make/made, which we talk little about it.
Besides, there's many traps(trap-like feature?) in grammar, especially when it comes to $/.
Anyway, thanks for your reply!

Recently be busy with school things. Close for I cannot accomplish it as promise.


Transaction deferral system

Compiles now - had to a bit of an ugly hack to serialize VerifiedSequencedConsensusTransaction

Should we limit defer to certificates and system transactions only, instead of arbitrary consensus transaction?
It is a bit unclear why would we want to defer other consensus transactions and semantics of this is also unclear.
So instead of deferring a vec of VerifiedSequencedConsensusTransaction, we could make a vec of VerifiedExecutableTransaction. I think some other parts of the code could simplify too after that

I tried that initially, and it turns out to be a much more awkward change, since the point at which we re-inject the deferred transactions is operating on VerifiedSequencedConsensusTransaction. In practice, we would never defer anything that isn't a UserCertificate. I can look into making that larger change if we think it is important and/or less confusing. My thought for how to do that was to partition UserCertificate transactions from all other transactions, and then have separate entry points into the per-epoch store for each type. Willing to do it if we think its important.

I see, I don't think it's that important if it involves extra complexity


How to obtain the radial stress distribution of particles in the swelling model

Hello,Sir
I can only see the stress distribution along the particle surface in the variables of this model. I can not find one variable of the stress distribution inside the particle. Can this model obtain radial stress instead of surface radial stress?
Looking forward to your answer, thank you very much!

cc @weilongai

I'm afraid there are no built-in variables for the radial stress inside particles in PyBaMM, but they can be easily calculated from lithium concentrations. See Eqs. (22-23) in Journal of The Electrochemical Society, 154(10), A910-A916, 2007

Thank you very much for your clarification. Wishing you success in your research and happiness every day！
L Y?
@.***

------------------ 原始邮件 ------------------
发件人:                                                                                "pybamm-team/PyBaMM"                                                                                @.>;
发送时间: 2024年3月11日(星期一) 晚上9:16
@.>;
抄送: "L<EMAIL_ADDRESS>主题: Re: [pybamm-team/PyBaMM] How to obtain the radial stress distribution of particles in the swelling model (Issue #3881)
I'm afraid there are no built-in variables for the radial stress inside particles in PyBaMM, but they can be easily calculated from lithium concentrations. See Eqs. (22-23) in Journal of The Electrochemical Society, 154(10), A910-A916, 2007
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>

@Santiagopeacely It looks like you got an answer for this ticket, so I am going to close this for now. If there is anything more feel free to reopen the ticket.


Add contact tool (part 1)

@yy6linda Can you please contribute to this PR (branch prepare-for-contact) and finalize the contact annotator example? You have write access to the repo so you can work directly inside it.

Now a duplicate of #6


Build Issue

Can't build do to go-counchbase update with package name changes. https://github.com/couchbase/go-couchbase/commit/94213df48fb5692b208a2d4fc7f8c1fab8667cc5#diff-36483a7174bd7c4d70f4e898a4ca0715
Here is the error output
./database.go:103: cannot use rv (type *"github.com/couchbaselabs/go-couchbase".Bucket) as type *"github.com/couchbase/go-couchbase".Bucket in argument to couchbaseutil.UpdateView
Needed to manually update the following lines in the database.go for a successful build
// Alias this because we call our connection couchbase
cb "github.com/couchbaselabs/go-couchbase"
"github.com/couchbaselabs/go-couchbase/util"
to
// Alias this because we call our connection couchbase
cb "github.com/couchbase/go-couchbase"
"github.com/couchbase/go-couchbase/util"

Would you mind sending a PR for this?

resolved by #144


Back end test suite

Challenge anonymous-message-board has an issue.
User Agent is: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0.
Please describe how to reproduce this issue, and include links to screenshots if possible.
I cloned this repo, added this line of code: process.env.NODE_ENV = 'test';
and then did "node server.js".
The server runs, but the tests don't seem to, i just get stuck with the following output:
http://i.imgur.com/diPR6Jy.png
I can't see what the test parameters are or how to pass them, there's no output.

@Em-Ant do you know what might be causing this?

I'm closing this issue as stale since it hasn't been active lately. If you think this is still relevant to the newly updated platform, please explain why, then reopen it.


Add release notes section to navigation bar

First introduced in 0.7

thanks, @drich10 !

Thank you for the quick merge!


Adds MNIST example Jupyter/Colab notebook.

As part of issues #479 and #231.
For reviewing the Colab, see
https://colab.sandbox.google.com/github/andsteing/flax/blob/mnist/linen_examples/mnist/mnist.ipynb

Codecov Report

Merging #493 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #493   +/-   ##
=======================================
  Coverage   74.29%   74.29%
=======================================
  Files          55       55
  Lines        3976     3976
=======================================
  Hits         2954     2954
  Misses       1022     1022

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 76d580d...8305394. Read the comment docs.

I like the way this PR visualizes wrong predictions but I don't really love a few things here:

The whole initialization with copying the MNIST example out is not something we would expect our users to do, why are we doing it here?
The actual model and training code which is what people most want to tinker with isn't available here. (And there's no clear way to modify the code if you want to tinker)

I would think it's more helpful to our users to instead supply a maximally simple MNIST Colab that they can directly tinker with, maybe something like the annotated MNIST version (https://flax.readthedocs.io/en/latest/annotated_mnist.html).
WDYT?

I like the way this PR visualizes wrong predictions but I don't really love a few things here:

The whole initialization with copying the MNIST example out is not something we would expect our users to do, why are we doing it here?

When this code is run in Jupyter, then no initialization is performed.
On the other hand, when the Colab is opened directly via the
https://colab.sandbox.google.com/github/andsteing/flax/blob/mnist/linen_examples/mnist/mnist.ipynb
link then the files in the same directory are not available ($CWD is in a newly created VM).
So if we want to make the notebooks run on Colab as opposed to only with Jupyter, then we need to do some setup. On the positive side, this setup should be the same for every notebook and it's only 9 LOC.

The actual model and training code which is what people most want to tinker with isn't available here. (And there's no clear way to modify the code if you want to tinker)

I would think it's more helpful to our users to instead supply a maximally simple MNIST Colab that they can directly tinker with, maybe something like the annotated MNIST version (https://flax.readthedocs.io/en/latest/annotated_mnist.html).
WDYT?

I disagree : I think tinkering with the .py files should be done in your favourite text editor where you have syntax highlighting etc, instead of in a Colab cell. The Colab should only contain the glue code to let you interactively run different parts of the example code and inspect the inputs/outputs.
Of course nothing prevents you from copying in code from the example into a cell and execute it from there. But I would rather avoid us making this copy and then having to keep it in sync.
Having a maximally simple MNIST Colab is a different story that we can put into the docs separately. This PR is about a model Colab that should be put into every existing example folder.

I disagree : I think tinkering with the .py files should be done in your favourite text editor where you have syntax highlighting etc, instead of in a Colab cell. The Colab should only contain the glue code to let you interactively run different parts of the example code and inspect the inputs/outputs.

OK, maybe -- I see that point. But then how would one do this with this Colab? Is there a way to edit the MNIST code locally and run it in Colab? If that's the case that's great -- and should probably be documented.
If we want people to use this from Jupyter only, then that's fine and I wouldn't call these Colabs. But that seems suboptimal for people wanting to tinker with models on GPU and TPU?
I guess I'd like to better understand the expected user flow of these notebooks

Having a maximally simple MNIST Colab is a different story that we can put into the docs separately. This PR is about a model Colab that should be put into every existing example folder.

Sounds good.

OK, maybe -- I see that point. But then how would one do this with this Colab? Is there a way to edit the MNIST code locally and run it in Colab? If that's the case that's great -- and should probably be documented.

Yes, you can edit it directly in Colab UI:

Of course you can also develop on your local machine and then push/pull via a branch in your forked Flax repo.

If we want people to use this from Jupyter only, then that's fine and I wouldn't call these Colabs. But that seems suboptimal for people wanting to tinker with models on GPU and TPU?

I think it's great having code run both in Jupyter and Colab, especially for easy access to accelerators.

I guess I'd like to better understand the expected user flow of these notebooks

Should I create some new documentation for this? Maybe in examples/README.md or should we add additional documentation under docs/examples/colab.rst?
Then we could link that new documentation in the beginning of every notebook.

Oh, I didn't know about that editing ability. But then where are those files saved? Is there a risk that with Colab kernel restart you lose your changes?

Yes they're gone with the VM. We could link Drive to persist files. But then I feel the setup code becomes more cumbersome.
Maybe we should not encourage people explicitly to use the Colab UI after all (I think it's a nice feature to play around with, but unexperienced users might be confused by the VM persistence semantics).
I think the workflow should be like:

Develop on your local machine with your favourite editor & interact in Jupyter notebook.
Once you're happy with your changes and want to try something with a GPU/TPU : push to your forked flax repo & load in Colab.
If you really want, you could change files in Colab editor & push them back. But I feel that's getting complicated.

But maybe that's more of an advanced usecase.
I still think it's nice to have Colab because it allows people to play around with the example without installing anything.

I still think it's nice to have Colab because it allows people to play around with the example without installing anything.

I agree with that sentiment, but I'm not convinced (yet?) that this approach will give users that, so I'd rather we discuss this more before we add Colabs like this to all example.

Reminder: Colab can be reviewed at
https://colab.sandbox.google.com/github/andsteing/flax/blob/mnist/linen_examples/mnist/mnist.ipynb
(=> Suggested actions : Inspect, clear outputs, run all, ...)
Hi Avital
Thanks for the time reviewing this thoroughly and all your valuable input!
Please see the updated notebook - I think that I addressed many of your concerns:

There's a link to the annotated_mnist_example._ipynb notebook that is precisely the MNIST example with all the code inlined. I agree this is an important usecase, but I don't think it makes sense to replicate that code here as well. BTW : any idea why the file ends in ._ipynb instead of the more standard .ipynb?
I explain the "Flax Notebook Workflow" prominently in the beginning of the notebook. (Note that everything works just fine in Jupyter[lab] as well)
The editor now opens mnist_lib.py automatically and the code can directly be modified. Very easy. I strongly believe that it is preferable having a horizontal split on all but 90-dgree-rotated screens. Also, it has the advantage that no cell needs to be re-run after updating the code.
Persisting all code in your personal Google Drive is as simple as selecting yes in the drop down.
Tensorboard is shown inline (additional to the previous "upload to tensorboard.dev").
The Colab now defaults to the GPU runtime and does a simple loop over a hyper parameter for illustration purposes.

Note that I do NOT think that we should have an identical notebook for every example, but we should rather think carefully in every case what brings most benefit to the user. I think that the notebooks will be a main figurehead for the examples - especially with the Colab links figuring prominently on the README.md.
Some minor open questions:
a. Do you agree mount_gdrive should default to yes? The current setup has the advantage to run end-to-end without any user interaction (authenticating for Drive requires manually copying a auth token).
b. Similar question and reasoning for upload_data dropdown.
c. Do you agree we should commit the notebook with the outputs? It makes it rather large (~10M) mainly because of TensorBoard - that won't display when viewed on Github (link), but it shows fine with on Colab (link). Also, including the outputs might discourage users from actually running it (?).
Cheers, Andreas

Thanks @andsteing -- this really looks much better!
Honestly this flow where you set up local editing with Google Drive with one click, and then allowing editing example code (or Flax core code) without running any cells is really innovative. It's better than any other Jupyter or Colab flow that I know of (when I use Jupyter[Lab] locally I typically need to go back and forth between an IDE/Editor and the notebook tab)
I think this template really does support all of the "user journeys" we discussed -- both for someone new to Flax trying to learn and tinker, as well as advanced users wanting to iterate on accelerators either locally or in Colab. And we could use this to also make custom Colabs for fine-tuning and other interesting downstream tasks.
This PR LGTM. I suspect there are a few small improvements we could make as people start using these notebooks more. Which brings me to my main question:
How do we plan to keep this Colab and others up-to-date when we want to change some of the "skeleton" setup (e.g. the Google Drive mounting, autoreload configuration, code cloning and copying)
A first possible part of the answer is: "can we extract part of this skeleton code into a mini-library within Flax? like flax_notebooks.py"? (This file would be somehow loaded first, imported, and not impacted by the auto-reload/Google Drive logic)
If we're not able to extract all of the skeleton code into a library, then we may want to consider this through the prism of @jheek's formal examples/skeleton proposal (that sets up best practices/patterns and nothing else).
Then we could consider telling people: "if you change anything in these files or in these parts of the files, then you must also get approval for change in the skeleton and have maintainers update their examples accordingly"
This response became long, and we probably need a separate GitHub issue to discuss the examples/skeleton proposal (does that exist?)
I think there's a bug though, (sorry GitHub didn't let me add inline comments on the ipynb file because it's too big) -- you have code that loops over 3 different optimizer momentum values but it doesn't look like you updated the config so I think you're running the same training code for all 3?
To answer your concrete questions,

a. Do you agree mount_gdrive should default to yes? The current setup has the advantage to run end-to-end without any user interaction (authenticating for Drive requires manually copying a auth token).

It's good to not require Google Drive token for initial exploratory runs. (One small downside is that you can still edit mnist_lib.py and you can still press "save" in Colab which would clone the /Colab/ (but not your mnist_lib) into Drive, so users may be surprised. Maybe there's a way to detect that? Like could we chmod the file to be read-only when you're not using Drive? Would Colab then indeed make the files read-only in the editor?)

b. Similar question and reasoning for upload_data dropdown.

Yes, same -- don't have people upload their results unless they know they're doing that.

c. Do you agree we should commit the notebook with the outputs? It makes it rather large (~10M) mainly because of TensorBoard - that won't display when viewed on Github (link), but it shows fine with on Colab (link). Also, including the outputs might discourage users from actually running it (?).

I don't think it's a problem if people don't run the Colab -- if it's good enough to just browse that's a feature not a bug. It's nice to see the TB plots inline in Colab.
The problem is simply size -- 10M per file is indeed quite large and gets close to the limit of what's not good to store directly in Git (as any time anyone clones the repo they'll clone the entire history of edits which means approximately 10M * number_of_edits_per_colab_evet * number_of_colabs. See this where they suggest no more than 1GB per repo. This would put us dangerously close to that limit.
Why is it 10M? Is it because of the scalars stored for metrics or some TB overhead? I do think 10M is too much.

Hi Avital
Thanks for your additional detailed comments!
As for your main question : How can we keep these Colabs in sync? A library would be one approach, but it has the disadvantage of more indirection (i.e. need to open the library files if you want to understand what is going on), and we would still need to keep the magic cells in sync that load the library.
What do you think about the following solution?
We have a skeleton.ipynb notebook some where (e.g. in dev/) and we have some utility sync_notebooks.py that automatically syncs any cells that are identical with a cell in skeleton.ipynb. This way it would suffice to update a single cell in the skeleton.ipynb and all notebooks when one of its cells are updated. We can use the same utility script (with a different flag) to compare cells and fail if there is a difference (for an automated check). If the skeleton.ipynb becomes unwieldy we can still separate logic into a library...
If this looks good to you, I will add the skeleton & utilities when I check in the second notebook as part of #480.
chmodding the files kind of nullifies the purpose of the editor :) Instead, I added a DISCLAIMER to the files opened in the editor to make it very clear whether changes will be persisted:
## WARNING : Editing in VM - changes lost after reboot!!
########################################################

vs.
## Note : Editing in your Google Drive, changes will persist.
#############################################################

Other than fixing the config.momentum = momentum bug, I also separated MNIST-specific variables into their own cell so we can use the setup cell as a skeleton later on, added a standard "Open in Colab" button, and added some explanation as to the merits of running the Colab instead of simply reading it.
Removing the TensorBoard output reduced the overall notebook size from 11.3M to 111k (mostly PNGs) - it would have been nice to include it, but at these costs it's definitely not worth it...
Note: If you want you can review the notebook here : https://github.com/andsteing/flax/pull/3
(Not sure how I can install the ReviewNB app in the google/flax repository...)


CachingDataSetIterator

Caching layer for data set iterator that can speed up DataSet retrieval.

Idea is nice, especially InFile datasetcache.


Update cats-effect, cats-effect-laws, ... to 3.3.14

Updates

org.typelevel:cats-effect
org.typelevel:cats-effect-laws
org.typelevel:cats-effect-testkit

from 3.3.13 to 3.3.14.
GitHub Release Notes - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.typelevel" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "@monthly" },
  dependency = { groupId = "org.typelevel" }
}]

labels: library-update, early-semver-patch, semver-spec-patch, commit-count:1

Codecov Report

Merging #473 (1ac5edc) into main (52a6e37) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #473   +/-   ##
=======================================
  Coverage   91.21%   91.21%
=======================================
  Files          18       18
  Lines         296      296
  Branches        4        4
=======================================
  Hits          270      270
  Misses         26       26

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 52a6e37...1ac5edc. Read the comment docs.


Implement sealed! statically

Motivation

This implements sealed classes statically. I'll be following up with an
implementation of static support for singletons, and then using this to
give better exhaustiveness checks for Opus::Enum.
Test plan

See included automated tests.

My test seems to be flaky, because it relies on the ordering of printing elements within T.any.

https://buildkite.com/sorbet/sorbet/builds/4898#2c7c321a-3538-46aa-837c-5bb9a35cd2b7/121-231
i've flipped this around 3 times now and it keeps coming up wanting it the other direction.

the sheer number of invariants that the new Symbol methods expect

I'm pretty sure I took care to make sure they're maintained, and the implementation heavily relies on them, so I figured it'd be better to fail loud and early. Let the fuzzer prove me wrong 🙂