Stream closed in another thread (IOError)

When running sauce_runner, the command fails with the following error:
Running version 0.150.0
20:27:55 INF Running XCUITest in Sauce Labs
20:27:55 INF Archiving project.app to .ipa
20:28:00 INF Archiving project-e2e-tests-Runner.app to .ipa
20:28:01 INF Checking if /var/folders/bl/wbxjgtzx7j5_mjsmfr3ynlc00000gp/T/project-1311811755.ipa has already been uploaded previously
20:28:01 INF Checksum: 0b1c55dcc101dcd58e9bd4460584222b592ec79d3b370d28ff36742334f372fd
+---------------------------+---------------------------------------------------------------+
|                                       Lane Context                                        |
+---------------------------+---------------------------------------------------------------+
| DEFAULT_PLATFORM          | ios                                                           |
| PLATFORM_NAME             |                                                               |
| LANE_NAME                 | upload_to_sauce                                               |
| KEYCHAIN_PATH             | ~/Library/Keychains/fastlane_tmp_keychain                     |
| ORIGINAL_DEFAULT_KEYCHAIN | "/Users/distiller/Library/Keychains/fastlane_tmp_keychain-db" |
+---------------------------+---------------------------------------------------------------+
#<Thread:0x00007fc871afe240 /Users/distiller/project/.bundle/ruby/3.0.0/gems/fastlane-plugin-saucectl-0.1.4/lib/fastlane/plugin/saucectl/helper/runner.rb:27 run> terminated with exception (report_on_exception is true):
/Users/distiller/project/.bundle/ruby/3.0.0/gems/fastlane-plugin-saucectl-0.1.4/lib/fastlane/plugin/saucectl/helper/runner.rb:28:in `each': stream closed in another thread (IOError)
	from /Users/distiller/project/.bundle/ruby/3.0.0/gems/fastlane-plugin-saucectl-0.1.4/lib/fastlane/plugin/saucectl/helper/runner.rb:28:in `block (2 levels) in system'

This only happens on CI. Following the same steps locally works fine. Coincidentally, we started getting this failure around the time SauceLabs had downtime, but support claims that they resolved all the issues. What's the best way to fix or debug this issue?

@kublaios apologies I just saw this. Are you still having this issue? Seems fine from my side.

@kublaios apologies I just saw this. Are you still having this issue? Seems fine from my side.

No worries. We could not find the root cause. Although we did not change anything in our setup, our test runs were either timing out or not showing up. This happened after SauceLabs had an outage, but they declined that the issue was on their end. Eventually, we moved to another platform.
Thanks for your work here. I'll go ahead and close the issue.


Have to manually run react-native start on Windows each time

Can this be added to the code base so that it starts automatically when I run react-native run-android

Can you provide more context?

Sure.
Can 'react-native start' be bundled with 'react-native run-android/ios' instead of running the commands separately?
Sort of how "adb reverse tcp:8081 tcp:8081" runs automatically with "react-native run-android"?

So this is only necessary on Windows as far as I can tell. Before bundling it with run-*, I guess my question would be why this doesn't work on Windows by default.

Personally I don't know why but I was thinking that if adb reverse could be bundled in, that could too. Preferably it would run after react-native run installs the app on the device

cc @hramos


Rancher Compose Config Variable Mgmt

Rancher needs to be able to support a way to define variables in the compose template itself so they can be replaced with user-specified values when the service is created and started.  This will allow Rancher compose templates to be re-usable with having to duplicate them.

Great feature to deploy services from one template with different passwords, ...

This is not working in below release. Referencing #1868
Component 	Version
Rancher 	v0.41.0-rc1
Cattle 	v0.99.0
User Interface 	v0.55.0
Rancher Compose 	beta/latest

Verified in compose release v0.4.1-rc2


fix onAudioTracks : selected payload

Summary
Fix selected payload of onAudioTracks event
Motivation
The selected payload of the onAudioTracks event was always coming as false, making it impossible to know which audio track is currently selected.
Changes
change trackIndex to 0 for the isTrackSelected call
Test plan

If I remember well I changed that to fix another issue :(
Can you provide the sample to reproduce the issue, I would like to triple check !


Loops

What's about loop support in templates?

Hey @pwFoo, what would you be looking for in loop support? Similarly to #1, I'd consider the effort required to refactor & change the templating logic plus updating the test suite to be more than something I'm able to personally do right now. Happy to help facilitate a PR and work on defining a spec for what it could look like, however.
A couple thoughts off the top of my head:

I dogfooded templater.js in the docs of another personal project of mine, meed, and used a native JS loop to build up the rendered output before writing it to the DOM: https://codepen.io/Pinjasaur/pen/byNoGv (the demo doesn't fully work right now for other reasons: https://github.com/Pinjasaur/meed/issues/1)
One of the features I added previously was for a key in the context object to be a function which is called at runtime and the returned result is used as the value rendered in the template. I'm wondering if this feature could be used to create a loop functionality but defined within the context object versus the template. When I initially added the feature I kept it very minimal: the function has no arguments. But maybe it would make sense to have optional arguments e.g. the template, context object, or similar to how Mustache supports lambdas? https://mustache.github.io/mustache.5.html#Sections

I think ja loop should be fine and avoid bloating up templates.js. thanks!


WIP: docs: added documentation for the API

I've created some small Jekyll plugins to overcome some flaws from the liquid
markup language... The code for the API documentation was inspired by
https://github.com/jexhson/jekyll-swagger, but I ultimately changed it so much
that it does not resemble in any way that plugin.
Moreover, because of the creation of these plugins, rubocop was getting noisier,
so I adapted the configuration from the master branch into this one.
Signed-off-by: Miquel Sabaté Solà<EMAIL_ADDRESS>

This is still WIP because it needs some changes on the API (e.g. extra dots on the description of some endpoints, and such).

Review #1646 before this one.

@vitoravelino this is now ready to be merged


Request to Add Product in Start React

Hello,
I am Maheshwari from team GeekyAnts.
On behalf of Start React, we add open source products which we find helpful to the community & also we provide credits to author itself.
Let me know if you are interested showcase your product in our open source website.
If yes, then I request you to add MIT Licence in your repo.
Looking forward to hear from you.

Hi @maheshwari-1153 . thanks. sounds great!
Is this LICENSE enough or you need specific format?
here is the link:
https://github.com/pavjacko/renative/blob/master/LICENSE


Default CTS arguments don't give good skew results

Description
Expected:

Actual:

To reproduce: untar  cts.tar.gz
./run-me-mock-array-asap7-base.sh
. vars-mock-array-asap7-base.sh
ODB_FILE=results/asap7/mock-array/base/4_1_cts.odb openroad -gui scripts/gui.tcl

Suggested Solution
CTS should automatically pick good defaults
Additional Context
No response

It seems that the timing repair step is degrading the clock skew. Some of timing repair can be pulled into CTS to mitigate this issue. @oharboe, what's the needed timeline for this enhancement?

I would prioritize orderly p

I would guess it is repair_clock_nets rather than repair_timing that is the problem (the later shouldn't touch the clock tree).

I would guess it is repair_clock_nets rather than repair_timing that is the problem (the later shouldn't touch the clock tree).

Correct.


Add file status on API

Will replace #5607 and fix #4313

I don't see anything in diff that should have changed commits in tests or am I missing something?

Tests are failing since you check the repo content in tests now and user2/repo2.git is not initialized.

Codecov Report

Merging #7671 into master will increase coverage by 0.04%.
The diff coverage is 43.47%.

@@            Coverage Diff             @@
##           master    #7671      +/-   ##
==========================================
+ Coverage   41.23%   41.28%   +0.04%
==========================================
  Files         471      471
  Lines       63744    63761      +17
==========================================
+ Hits        26287    26325      +38
+ Misses      34027    34000      -27
- Partials     3430     3436       +6

Impacted Files
Coverage Δ

modules/repofiles/action.go
62.5% <40%> (-0.97%)
:arrow_down:

modules/pull/merge.go
39.47% <40%> (-0.23%)
:arrow_down:

models/action.go
54.25% <46.15%> (-1.19%)
:arrow_down:

models/repo_list.go
73.09% <0%> (+1.01%)
:arrow_up:

routers/repo/view.go
43.25% <0%> (+1.01%)
:arrow_up:

modules/log/event.go
65.64% <0%> (+1.02%)
:arrow_up:

models/unit.go
67.56% <0%> (+5.4%)
:arrow_up:

modules/git/commit.go
57.03% <0%> (+10.26%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d974c13...c7a752b. Read the comment docs.

@lafriks get file status will read git repository, and previous tests will read a non-exist repository, so I change it to an exist repository. I have copied all repositories from integration/gitea-repositories-meta in a previous PR that means all unit tests could really read/write git repositories.
@sapk tests fixed.


New version: Dio.PureCodec version 20240131

This pull request is automatically generated by 🥟 Dumplings in #3267
Microsoft Reviewers: Open in CodeFlow

Hi @SpecterShell,
The package didn't pass a Defender or similar security scan. This might be a false positive and we can rescan tomorrow.
(Automated response - build 684.)

Automatic Validation ended with:

Installation failed with exit code -1978335186 2024-01-31 12:46:08.892 [FAIL] D:\a_work\1\s\external\pkg\src\AppInstallerSharedLib\SHA256.cpp(144)\WindowsPackageManager.dll!00007FFF43DD273F: (caller: 00007FFF43E1AEC4) Exception(1) tid(1698) 8A150040  2024-01-31 05:28:50.061 [FAIL] D:\a_work\1\s\external\pkg\src\AppInstallerCommonCore\Downloader.cpp(208)\WindowsPackageManager.dll!00007FFF43E14424: (caller: 00007FFF43E15C49) Exception(2) tid(1698) 8A15002E  2024-01-31 05:28:50.070 [CLI ] Failed to download, waiting a bit and retry. Url: http://dl-t1.wmzhe.com/13/13163/PureCodec20240131.exe 2024-01-31 05:33:13.233 [FAIL] D:\a_work\1\s\external\pkg\src\AppInstallerSharedLib\SHA256.cpp(144)\WindowsPackageManager.dll!00007FFF43DD273F: (caller: 00007FFF43E1AEC4) Exception(3) tid(1698) 8A150040  2024-01-31 05:40:56.692 [FAIL] D:\a_work\1\s\external\pkg\src\AppInstallerCommonCore\Downloader.cpp(208)\WindowsPackageManager.dll!00007FFF43E14424: (caller: 00007FFF43E15C49) Exception(4) tid(1698) 8A15002E  2024-01-31 05:40:56.702 [CLI ] Caught wil::ResultException: D:\a_work\1\s\external\pkg\src\AppInstallerCommonCore\Downloader.cpp(208)\WindowsPackageManager.dll!00007FFF43E14424: (caller: 00007FFF43E15C49) Exception(4) tid(1698) 8A15002E  Installation failed with exit code -1978335187 2024-01-31 11:23:29.188 [FAIL] Installer failed security check. Url: http://dl-t1.wmzhe.com/13/13163/PureCodec20240131.exe Result: 0x80004005

(Automated response - build 684.)


[Impeller] Perform clipping on the depth buffer.

See this design doc.
Now that we simply replay clips and no longer ever need to store the multisample depth/stencil attachments, we can safely perform clipping on the depth buffer and free up the stencil buffer for Stencil-then-Cover of concave shapes.
Further, since we have a lot more bits to work with in the depth buffer, we can eliminate the need for clip restores altogether.
All of this can be done in a way that synergizes well with optimizing the draw order to minimize overdraw.

Completed by https://github.com/flutter/engine/pull/51219


3355: Allow for RNAseq analysis form GVCF level

https://github.com/bcbio/bcbio-nextgen/issues/3355

Making a fresh pull request with just the 2 small commits. My bcbio fork was out of date.


Expose Import instruction in transactions

Datalogia transactions support Import instruction that take arbitrary DAG-JSON data and ingest it by deriving facts from it
export interface Instantiation {
  [Key: string]: Constant | Constant[] | Instantiation | Instantiation[]
}

export interface FactsSelector {
  entity?: Entity
  attribute?: Attribute
  value?: Constant
}

export type Instruction = Variant<{
  Assert: Fact
  Retract: Fact
  Import: Instantiation
}>

export interface Transaction extends Iterable<Instruction> {}

export interface Transactor<Ok extends {} = {}> {
  transact(transaction: Transaction): Task<Ok, Error>
}

It was not exposed mostly because datalogia does not support ordered lists and we punned on import until #20 would be get fixed. However it seems like it would be a good idea to expose it despite not having a perfect solution yet we could improve importer when we do have one.

This got implemented and landed


Update requirements

Go does not need to be installed bc there is a package for compliance masonry
may need to install node-js
gitbook

Fixed these up in https://github.com/opencontrol/freedonia-compliance/pull/14 ... I think this is good to close! Feel free to reopen if I missed something.


build.sh: Update dependencies

Summary
Update the versions of Alpine, Go, Dart runtime, and Dart protobuf invovles a change of the repo from dart-lang org to google
Changes

Update the version set to DART_PROTOBUF_VERSION
Update the version set to DART_VERSION
Update the version set to GO_VERSION
Update the version set to ALPINE_VERSION
Update the version set to NODE_VERSION

Thanks for the contribution! Not sure how fast it would get merged though, since I left the company and have no more rights to do anything. I don't think anyone in TheThingsIndustries care about this repo.
@johanstokking we talked earlier that you would give me write rights on this repository or even transfer to my namespace, is that still happening?
cc @htdvisser

We are working to sort this out, but things are moving a bit slow due to vacation season here.


Honor pre-trim for tags

Not the fastest way to do it -- I think doing it in the tokenizer is
better. Let me know if this is acceptable.

I'm not sure why https://github.com/Shopify/liquid-c/blob/76b1b8ae5268ecdab4441e4701eafe83b9e0c804/ext/liquid_c/tokenizer.c#L120-L129 isn't making this work already.

I'm not sure why

Turns out, it's an off-by-one that I missed!


Translations seem to cause trouble in multiplayer, disconnecting joining player

See https://forums.factorio.com/viewtopic.php?p=569067#p569067 and possibly the related topic linked in the posts in the topic.

Yeah, this is a known issue. I have some ideas on how to fix it, but I am unable to replicate the issue myself so I can't properly test it. Once I get my laptop in August I'll be able to properly test it and fix it.

Thanks for the detailed analysis @Hornwitser. I have decided to remove the nesting entirely, and do a hardcoded five strings per translation. I don't have the tools to test how much network traffic this generates, but I'm going to send it to a few people to see if it solves their issues.


kube-hunter: init at 0.6.3

Motivation for this change
Tool to search issues in Kubernetes clusters
https://github.com/aquasecurity/kube-hunter
Related to #81418
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 149469 run on x86_64-linux 1

1 package built:

kube-hunter


Check that every strand-seq file is properly paired.

In case a situation like this makes it to the pipeline

Pipeline now checks that paired files differ only in 1 character, and that those characters are 1 or 2


How do users initiate opt-out requests?

@jdwieland8282
Does it mean all publishers website integrate Shared-Id need provide a link which lead users to a Shared-Id opt-out UI? (similar concept as CCPA CMP)

We are trying to prevent such a scenario @wqi1972 , we believe that enforcing that is unworkable for all pubs. Rather, today sharedid leverages the 3rd party cookie to persist optout across domains.  In the future we'll need a replacement for the 3rd party cookie. Browsers have not picked a standard yet, safari's choice seems to be the storage access api, Chrome is exploring trust tokens. I suspect optout will need to support browser detection, then do the appropriate thing.


Add Bash/Linux/Unix-style shortcuts to Command Prompt

Description of the new feature/enhancement
Add support, preferably via an optional toggle, for Bash/Linux/Unix-style console shortcuts. Into the standard windows command prompt. Once you start to incorporate some of them into muscle memory, it becomes obvious that stuff like this is sorely lacking in the ordinary Windows Command Prompt.
Examples of such are (non-exhaustive list)
Ctrl+A = Move text cursor to the beginning of the line.
Ctrl+E = Move text cursor to the end of the line.
Ctrl+B = Move text cursor one step backwards.
Ctrl+U = Cut everything behind the text cursor.
Ctrl+Y = Paste last thing cut in clipboard.
Ctrl+D = Same functionality as the 'Delete'-key
A more exhaustive list can easily be found with simple searches. https://www.howtogeek.com/howto/ubuntu/keyboard-shortcuts-for-bash-command-shell-for-ubuntu-debian-suse-redhat-linux-etc/

Man, as much as I'd love that, we won't be updating cmd.exe anytime soon. If you want a modern, updated shell on Windows, you should try out PowerShell 7.
IIRC, PSReadline (a component of pwsh), already includes some of that functionality (though, it may be bound to other keys by default).

Not that it changes much, but implementing the suggested command-line editing shortcuts would have nothing to do with the CMD shell. It would be new functionality in the cooked read that the console implements for ReadFile and ReadConsole when line-input mode is enabled. This would affect all applications that use the console's cooked read, not just the CMD shell. From what I gather, the console team is not interested in updating the builtin command-line editor with new functionality or configurable behavior such as emacs and vi modes.

You're technically correct, the best kind of correct.

Yeah -- unfortunately if we add those shortcuts to cooked read (conhost's line editor) it'll immediately take them out of the hands of applications that might want to use them. Even so, maybe that's a non-issue.
I wonder if we should do something here? It's just new bindings for existing things -- in most cases (home, end, delete, backspace, kill-word, etc.) that's just adding another binding for the same thing.
In other cases . . . well, we'd have to design a yank buffer... and decide that we're going "full emacs" :smile:
As an aside, I've always appreciated that on macOS, emacs bindings work in all text fields. It took a while to break me of the habit in Windows!

I wonder if we should do something here?

I don't disagree with Mike Griese that this shouldn't change for CMD -- especially if we're talking about radically redesigning its input loop to use ReadConsoleInput instead of ReadConsole. I just wanted to highlight that the more direct way, which doesn't touch CMD's source code, would be to extend the console's command-line editor with new edit modes. This would affect all simple command-line shells (e.g. Python's REPL shell and input function just call ReadConsoleW in Windows), which is both a benefit and a curse. I thought that the console team considered the built-in command-line editor in line-input mode to be a mistake, in which case doubling down to extend it with new functionality would be a poor choice. The alternative is to release a C/C++ readline library that's based on low-level ReadConsoleInput, maybe even incorporate it into the Windows API. A readline library can support "Vi", "Emacs", and classic "Windows" modes, like PSReadLine does.

The alternative is to release a C/C++ readline library that's based on low-level ReadConsoleInput, maybe even incorporate it into the Windows API. A readline library can support "Vi", "Emacs", and classic "Windows" modes, like PSReadLine does.

I"VE ABSOLUTELY HAD THE SAME THOUGHT.
It wouldn't be total madness to do this. At the end of the day though, someone would need to write the code, and I'm thinking that's sitting way farther down no the backlog for now. But I'd really love to have all these applications that are relying on conhost's cooked read to rely on something that we can update without fear of changing cmd.exe


Template fixes for Wagtail 4.2

Based on https://github.com/wagtail/wagtail-airtable/pull/48
Fix the templates to bring them in line with Wagtail 4.2, and fix broken bulk actions.

With the new button styles, a button with an icon doesn't play well with long-running buttons, so I've changed the "Import foo" button to just use the long-running style.
Manual tests performed for Wagtail 4.2 compatibility:

[X] Import snippets from Airtable
[X] Save new snippet to Airtable
[X] Save new snippet with RevisionMixin to Airtable
[X] Import snippets from Airtable to snippet with RevisionMixin

Import works, but revision not created (will open an issue for this)

[X] Import pages from Airtable
[X] Save page to Airtable

I'm not sure why the CI is failing, it looks like the tests are passing.


RecordSet application

We want a simple record set application for displaying a set (maybe only in table form) of records (entities) from an ermrest path expression.

I've started this off but should pass it off sometime soon.
@karlcz if you want to see current status,
https://dev.isrd.isi.edu/~schuler/chaise/recordset/#1/legacy:dataset
It should be able to support filters already like this:
https://dev.isrd.isi.edu/~schuler/chaise/recordset/#1/legacy:dataset/id::gt::100/project::eq::154
But that won't work yet, just pending some finishing touches from @shinyichen on the ermrestjs refactor library.

This now works:
https://dev.isrd.isi.edu/~schuler/chaise/recordset/#1/legacy:dataset/id::gt::100&project::eq::154&id::lt::13000

This is in a pre-MVP state. But as an initial semi-functional "prototype", I'd like to review and close this. Then we should review with Karl and specify the requirements for MVP for this feature.


interactions index: unintended code block

Describe the bug
In the index page for the Interactions guide, there is an unintended code block.

To Reproduce

Go to the page for the Interactions index
You'll see a code block when the guide is listing the different types of Context Menu commands

Expected behavior
The text is formatted correctly to not be a code block.
Additional Context
I haven't tested to see if this issue appears in the master branch, but it definitely does appear in the production branch.
Checklist

[x] This is an issue with the build, not a spelling or grammar mistake.
[x] I have the latest version of the guide.

Quick look at the master branch commits, it does seem to be fixed. However, I don't think the fix was merged to production yet.

Fixed
https://62859beb63300e0008a6c20e--pycord.netlify.app/interactions/


Showing graphs immediately after clicking on skip

This is very unlikely to happen and I do not know if it can be prevented. But if you skip a round (it takes a few seconds until next round appears) and click on show graphs before new round is loaded, you can't do anything in the next round.

We could disable these buttons after clicking on "skip"


(WIP) Publish a more usable package

The objective of this branch is to make it so we can publish a package that is more useful for application developers.  We currently publish an openlayers package that is really only usable if people develop applications with the Closure Library and Closure build tools or use our custom build tool.  This branch proposes that we publish an ol package that can be used by mainstream package managers and module bundlers (e.g. webpack, Rollup, JSPM, or Browserify).
The branch includes a codemod (for use with jscodeshift) that transforms our sources into ES6 modules.  Running make package generates modules in the build/package directory.  The ol package can then be published from this directory.
There are still things to work out, but here is how to use it.
Install the package.  For example with npm:
npm install ol@beta

Import just what you need:
import Map from 'ol/map';
import View from 'ol/view';
import TileLayer from 'ol/layer/tile';
import XYZ from 'ol/source/xyz';

new Map({
  target: 'map',
  layers: [
    new TileLayer({
      source: new XYZ({
        url: 'https://{a-c}.tile.openstreetmap.org/{z}/{x}/{y}.png'
      })
    })
  ],
  view: new View({
    center: [0, 0],
    zoom: 2
  })
});

A few things to note:

module paths are like the the ol.foo.Bar names, except they have / instead of . and are lowercase
boolean defines are global, use _ instead of ., and are uppercase (e.g. OL_DEBUG instead of ol.DEBUG) (update: punting on defines for now)
builds are going to be huge because of how much code we unconditionally require in the library (e.g. +100 kB of unused WebGL code - see #6303 for an alternative) (update: see below for detail on advanced compilation with the Closure Compiler) (update 2: with #6312 and #6314, far less unused code is included in bundles and Uglify minification is getting reasonable)

More complete examples using the ol@beta package:

for use with webpack & Uglify
for use with Rollup & Uglify
for use with Rollup & Closure Compiler
for use with Browserify & Uglify

TODO:

[x] use build flags and compress with Uglify
[x] demonstrate use with Browserify
[x] demonstrate advanced compilation with the Closure Compiler

I think this is a great piece of work. I've not come across jscodeshift before. It takes a while to work out, but once you get used to it (and once you discover tools like http://astexplorer.net/) it's a great way to make bulk changes to code: writing programs to change programs.
But ...
The problem I have with publishing this as a package is that it creates a different api which is not compatible with existing apps - a major version change - without any documentation or examples showing how to use it. If people look in the source, they find comments which also relate to the ol3 api, not this new one. I think this is likely to confuse people. I think the readme should acknowledge this, and include instructions for use: include your 'few things to note' and tell people that this is mainly intended for adapting existing ol3 apps to mainstream build/bundle tools: "take your existing app and change all the global ol references to use imported variable names".
It's also not clear to me how this fits in with the aim of converting to ES6 modules. The codemod can easily be adapted to change to ES6, and I can see how the output from that could be used as the basis for a new improved ol4. If the comments/types were changed as well, you could use advanced-mode Closure on that to create a build similar to the present one, but with much more flexibility, as it would be based on standard ES6 modules rather than Closure's esoteric globals. JSDoc similarly. Node tools are slowly moving to ES6 - Webpack 2 for instance - a trend which I think will accelerate once Node implements V8's ES6 module implementation. Apps can then be modified to use import/export instead of the current globals. But if you go with CJS, at some point this will have to be changed to ES6, and people will have to change their apps twice. Not only that, but CJS is only usable in Node, whereas ES6 is designed to be usable in both browser and server. You comment on build sizes, but this should improve with tools like Rollup and Webpack 2 which can do 'tree-shaking' with ES6 modules. Webpack 2 can also split the code, a major aim for me, though I've not tried this yet to see how it might work in practice. All in all, going with ES6 seems a much better option to me than CJS.

@probins - I've updated the codemod to generate ES6 modules.  Clearly ES6 modules are the way forward.  I'm primarily interested in making OpenLayers easier to use - I don't think there is as much value in a goal like "use ES6 modules."  I've put together one example of how the ol@beta package can be used (right now, not in the future) with Rollup.  We can do the same with Webpack etc. (unfortunately, it is more involved to consume ES6 dependencies with Browserify).  It is also worth considering that our project has dependencies on CommonJS modules (pbf, pixelworks, rbush, vector-tile).  And those dependencies bring in transitive dependencies that are also CommonJS modules.  So again, while we agree that ES6 modules are the future, my goal is to make the project more usable right now.  After fleshing out some additional examples, we'll see if that is successful.
I'll also take a look at generating Closure Compiler builds from the Rollup output.  Though everybody likes to repeat "tree shaking!", we'll need a major restructuring of the library to really benefit from selective named imports (this will be far easier to accomplish with a major release).

I added instructions for using the ol@beta package with webpack@beta here: https://gist.github.com/tschaub/79025aef325cd2837364400a105405b8

If anybody is interested in experimenting with rollup->closure, here is a rollup.config.js that works:
const compile = require('google-closure-compiler-js').compile;

module.exports = {
  entry: 'main.js',
  targets: [
    {dest: 'bundle.js', format: 'iife'}
  ],
  plugins: [
    require('rollup-plugin-node-resolve')(),
    require('rollup-plugin-commonjs')(),
    {
      name: 'closure-compiler-js',
      transformBundle: function(code) {
        const flags = {
          createSourceMap: true,
          jsCode = [{
              src: code
          }]
        };
        const output = compile(flags);
        return {code: output.compiledCode, map: output.sourceMap};
      }
    }
  ]
};

This assumes you have:
npm install --save-dev rollup rollup-plugin-commonjs rollup-plugin-node-resolve google-closure-compiler-js
npm install --save ol@beta

This could be improved with logging as in rollup-plugin-closure-compiler-js - though we can't use processCommonJsModules as in that plugin.  And this is just simple compilation.  Still huge bundle size.

there is a known issue with defines in modules https://github.com/google/closure-compiler/issues/1601 - not sure if this is the same problem. Are they being renamed/mangled? Did you try window.FLAG? Or exporting the symbols?
Closure compiler needs typedefs.js, so would suggest leaving it in src/ and changing codemod to ignore it.

If the comments/types were changed as well, you could use advanced-mode Closure on that to create a build similar to the present one

actually, types shouldn't need changing for the compiler, as types can be any string. Might need changing for JSDoc though.
I totally agree on reducing dependencies. This is same complaint as in https://github.com/openlayers/ol3/issues/2381#issue-37865709

Use in browser
I would suggest changing the codemod to add '.js' to all the import specifiers. This should work the same in Node, and would also work in the browser. I have made this change, added a test directory, and pushed the result to https://github.com/probins/oles6
Then non-map loads like https://rawgit.com/probins/oles6/master/test/testextent.html should work in Edge with experimental JS flag enabled (the only browser which has so far released a usable module loader).
maps do not work, because ol.ext also need to be converted to ES6 modules. A simple map with no vector layers does not need ol.ext, but these are always loaded because of excessive dependencies. If I hack the code to (a) define flags, and (b) remove unwanted dependencies, as in https://github.com/probins/oles6/commit/3a66183656bb5b477fd904ea1c17a6ded93b5215, then a simple map as in https://github.com/probins/oles6/blob/master/test/testes6.js should also work in Edge.
For other browsers, I have converted to System.register
babel --plugins transform-es2015-modules-systemjs es6 -d reg

and pushed to https://github.com/probins/olreg. Then - ta-da - the simple map in https://rawgit.com/probins/olreg/master/test/testreg.html, which uses the SystemRegisterLoader, will work. This is effectively a debug load, so not recommended on a flaky mobile connection, but response times are acceptable on my fibre connection. Run the files through uglify (or Closure simple, or babili, which is looking promising), create a release which can then be cached and loaded via cdn.rawgit.com, and preload dependencies to eliminate round-trips, and this should be usable in production. I still have to test this out, but having independently loadable modules as in my extent example is a major step forward, as it removes the need to load a large one-size-fits-all build upfront.
So, my priorities are:

sort out ol.ext (for example, for rbush, could use build on unpkg.com, keeping this as a global, though it's probably better to remove the UMD wrapper and replace with an ES6 one)
remove unwanted dependencies, making the define flags unnecessary

I've ticketed google/closure-compiler#2217 to reflect the remaining issue with @define.
It is even clearer to me now that we need to stop using @define to configure conditional dependencies - specifically the alternative map renderers and layer-specific renderers.
In addition, I think it would be worth reconsidering our use of ol.DEBUG.  I'm not convinced that we are getting a lot of benefit from it, and since it is awkward to work unless working with the Closure Compiler and the global ol type builds, it would be unfortunate to start including more code behind ol.DEBUG conditions.

Ok, at this point it looks to me like the Compiler cannot successfully compile a Rollup bundle in advanced mode.  I'm seeing an issue where the Compiler is optimizing away an enum but not recognizing the use of all of its properties.  Specifically, _ol_layer_Base_.Property is compiled to a variable declaration that is missing a declarator for the SOURCE property.  All other properties are used by _ol_layer_Base_ methods.  The SOURCE property is used in _ol_layer_Layer_ methods.  Rollup adds a _ol_layer_Base_$1 alias for _ol_layer_Base_, _ol_layer_Layer_ methods use this alias.  The Compiler isn't recognizing that _ol_layer_Base_$1.Property.SOURCE is accessing _ol_layer_Base_.Property.SOURCE.
I've confirmed that the Rollup bundle runs as expected.  I've tried producing a minimal example that demonstrates the Compiler issue, but the optimization strategies are different with smaller, contrived examples.  Though looking at the uncompiled and compiled code for the map example above (minus the defines), it looks clearly like a Compiler issue to me.
I'm guessing that this problem would be resolved by moving these enums to dedicated modules instead of tacking them on to existing constructors.

Ok, a bit of a breakthrough.  This now includes the code from #6308.  After running into rollup/rollup#1189 (which may be considered expected behavior), I've reworked the codemod here so our modules make it through Rollup a bit more intact.  And boom, ADVANCED compilation achieved.
Here is a complete, end-to-end example of how you can build an application with the ol@beta package, Rollup, and the Closure Compiler (in ADVANCED mode).  And with camelaissani/rollup-plugin-closure-compiler-js#4 in, the Rollup config got even simpler.
As noted in the readme, the Compiler is crazy noisy.  Unfortunately, the extraAnnotationNames flag is not supported for the JS version of the compiler.  So we get warnings for our @api annotations.  I'm sure this can be addressed too.
This is starting to feel promising to me.  I'd still like to demonstrate an end-to-end example with Browserify and webpack.  And, it would be great to deal with our defines problem.  In the meantime, I'm going to make this package opinionated about the map renderer (though we still get unnecessary layer renderer code).
Goodbye closure-util.

thanks for all your work on this, @tschaub
The noisy compiles should be reduced by #6294, right? Though I notice that the rollup plugin is not using the latest compiler version, so there may be some additional issues there. The annotation tag warning should be suppressable https://github.com/google/closure-compiler/wiki/Warnings - or maybe better would be to raise an issue for the js version to add this.
I think you should change var Map = in your example. Like var Object/Math/JSON/... =, it will not actually produce an error, but will screw up any following code that tries to use the global that you've just redefined. This is less of a problem in a module, but var OLMap = or something would be better practice IMO.

Thanks for the feedback @probins.  The noise from the Compiler is largely about our use of unrecognized annotations (@api mostly).  These are just warnings and can be suppressed.  It would be nice if the JS version of the compiler supported extraAnnotationNames (ticketed as google/closure-compiler-js#48).
Regarding the use of Map in the examples, please keep in mind that these are just examples.  The library doesn't use the identifier Map.

Nice to see work towards this modernization, and excellent information on the usefulness of codemod. I adapted a smallish project to use 3.21.0-beta.12 from npm and I haven't found any issues yet. Though I agree on the point that major restructuring will be needed to enable named imports etc. and that it might require a new major release.

Nice to see work towards this modernization, and excellent information on the usefulness of codemod. I adapted a smallish project to use 3.21.0-beta.12 from npm and I haven't found any issues yet. Though I agree on the point that major restructuring will be needed to enable named imports etc. and that it might require a new major release.

I added a basic Browserify example.
Still digging to find ways to make Uglify builds more compact.  I'm getting 30K gzipped builds with the Closure Compiler and 60K gzipped with Uglify.  There is still a lot of code that is unconditionally imported/executed (e.g. the map requires the WebGL renderer which has a transitive dependency on RBush) and Uglify cannot effectively determine that it is all side-effect free.
@ahocevar and I have talked this through and think that the way to get smaller builds for all is to have WebGL* versions of the map, tile layer, image layer, and vector layer.  I'll keep digging to see if there are other easy wins.

In case anybody is interested, here is an interactive bundle explorer that shows the relative contribution to the bundle size:

I'm using the same main.js from the examples above in this example.  In the graphic above, I'm hovering over ol.render.webgl, which contributes 20% to the bundle size.  We're still working to make this and other unused code easier to exclude.

I'll stop dropping big graphics here, but this is good news.

After #6314, simple dead code elimination brings the ol.render code down to 6% of the bundle size (where before ol.render.webgl made up 20%).

Here are the latest gzipped bundle sizes with<EMAIL_ADDRESS>and the main.js above:

Method
Size

Rollup + Closure Compiler
31K

Rollup + Uglify
55K

Webpack + Uglify
59K

Browserify + Uglify
64K

Results with Babili are similar to (slightly worse than) the Uglify results.  It would be nice to get Uglify builds down closer to the Closure Compiler builds - but the difference should be less pronounced for applications that use more of the library.

Is it not hard/impossible to reach closure levels of dead code elimination without named exports (vs. single object default export)?

Is it not hard/impossible to reach closure levels of dead code elimination without named exports (vs. single object default export)?

@bjornharrtell - this is not really about named exports vs. single default export.  It is more about functional vs. object oriented design.  For example, if you use ol.geom.Polygon, the Compiler figures out that you don't need all of ol.geom.flat.simplify because you don't call polygon.simplify().  If instead we organized the library so you needed to explicitly import a simplify function and call simplify(polygon), then you wouldn't be importing dead code in the first place.

@tschaub I think what you just said only is a good cure for things we expose through the API. Does it also help for things we use internally? E.g. since we use polygon.simplify internally (the vector renderer does), we would still explicitly import simplify, only somewhere else in the library.

I agree that it's about functional vs. object oriented design but there are already some degree of functional design in ol3. For example the extent.js module a collection of pure functions. But the current codemod exports applies them to single object which is then default exported, so you can only import the module as a whole. So, to be able to explicitly import a single function from the extent module would it not be a requirement to export each function as a named export?

@bjornharrtell - yes, good example.  I was picking on Polygon since it pulls in a bunch of code that is unused in the example above.  The ol/geom code accounts for 12% of the bundle and is entirely unused (only required because the drag-box interaction has a getGeometry() method).  In that same bundle, ol/extent accounts for 3% of the build.  So it might be possible to shave a bit off there - though I haven't confirmed how much of those functions are already used in the bundle.
So, when we convert the library to use ES6 modules, we definitely can restructure things to use more named imports and exports.  It may be possible to make the codemod use named exports and imports where we assign to (non-prototype) properties of the goog.provide() arg.  I'll try to dig into this - though I'm dubious it will result in significantly smaller bundles (with simple tree-shaking and dead code removal) given that the big offenders in the example bundle are all Constructors (plus prototype properties) and not straight functions

Agreed, with the current design perhaps there are only small savings to be had regarding named experts. But is not a functional redesign pretty far off? Makes me wonder what short term gains there is to be had with es6 modules and alternative bundling for ol, even though i very much prefer to write new code in that way.

The ol/geom code accounts for 12% of the bundle and is entirely unused

not quite true: view.fit() requires SimpleGeometry and Polygon

Makes me wonder what short term gains there are to be had with ES6 modules and alternative bundling for ol

@bjornharrtell - I agree that ES6 modules don't bring us a ton of immediate wins (over CommonJS modules).  As I've said elsewhere, my goal is to provide a more usable package for consumers of the library.  While it will continue to change quickly, my guess is that ES6 modules involve more work for people to consume than CommonJS modules.  But I know that either ES6 modules or CommonJS modules are accessible to more people than the current files with goog.* dependencies.

not quite true: view.fit() requires SimpleGeometry and Polygon

@probins - good point.  So "raster-only" apps are stuck with a lot of unused geom code (as long as we have things like polygon.simplify()).

"raster-only" apps are stuck with a lot of unused geom code

though both cases can quite easily be removed. dragzoom (and so dragbox) can be removed from default interactions (or, better, change map so defaults are no longer 'required' #6311). view.fit() only requires geom to convert an extent to a polygon; change that so that only a polygon is handled and the requires can be removed; developers that want to fit to an extent can easily use polygon.fromExtent() in their apps. However, as this is 'api stable', this would be an api-breaking change (even if only a minor one).

It worries me to see PRs like https://github.com/openlayers/openlayers/pull/6313 merged without much discussion, during the Christmas break.
@tschaub, I propose we organize a hangout this week where you can present your proposition and see what are the implications in OL3 and for the projects using OL3. We can also discuss my proposition about goog.module and explain the Camptocamp vision for the OpenLayers library and what we are doing in our projects related to using ES6 / goog.module and getting ride of closure-util.

Hi all interested in modernizing OpenLayers and projects depending on OpenLayers.
Are you available for a discussion this Thursday (usual time (17:30 Swiss zone, CET) and hangout URL (http://bit.ly/ol-hangout)) ?
Discussion topics:

using ES6 (code, modules) in OpenLayers;
future of closure-util, feedbacks of using other loaders for development/debug;
paths to ES6 (projects compiling with closure compiler, goog.module, ...)
...

Ping @openlayers.
Ping @tschaub, @probins, @ahocevar, @sbrunner, @fredj, @tsauerwein.

I'll likely have no time, sorry

Is eavesdropping ok? :)

@bjornharrtell, sure it is fine with me.

I'd be available for a call tomorrow.

I'll be there as well.

I won't be able to participate, so here's a few thoughts.
I think this PR makes it clear that there's more to 'modernizing OpenLayers' than just changing syntax. ES module loading in the browser has been a long time coming, but initial implementations are either released or on the way (landed in Safari TP last week, for example). The ES modules produced by this codemod can be used in these implementations, but are not very satisfactory because the amount of code that has to be loaded is way too big. The last time I looked, the most important class, map, had 154 dependencies. As another example, OL formats could in theory be used standalone for converting data from one format to another, but geojson has 70 dependencies and kml 80. Removing WebGL from map, separating out vector classes and making the default controls/interactions/projections optional have been mentioned. The main problem with the formats is that they require all geometry types whether they're needed by the data in question or not; they all require ol.Feature, which requires style, even though most formats don't support style; it would probably also be better to separate read and write. You can probably resolve these issues whilst keeping the same api by adding new classes or something similar, but is this worth it? Is it not better to stop/wind down development of v3, and start on a new, though similar, api based on small independently loadable ES modules?
There are programs like lebab which can help converting current code to ES6. Using standard ES instead of custom Closure code should make it easier for more people to contribute, either directly or by creating alternative/custom modules. But there are other simplifications I'd like to be considered:

OL implements its own version of Object; is this justified?
ES6 introduced Set/Map for collections; is ol.Collection still needed?
can ES6 class get/set replace current custom accessors?
are there simpler ways of implementing observables and the current custom event handling? (there is a TC39 Observable proposal, but it's not progressing very quickly)
I'd like to explore replacing the current DOM handling with custom elements (ol-map, ol-overlay, ol-scaleline ...), using shadow DOM for CSS scoping. Simple maps could then be created by people who know HTML but are not JS programmers.

@probins - please make these suggestions in a separate thread.  The purpose of this pull request is to publish a package that is more usable.  That is, we want to publish a package that is consumable by mainstream module bundlers.  And, it is my belief (shared with other core developers) that this can be achieved in the 3.x series.
Clearly there is a lot that can be done in a major revision of the library.  But I'd like to keep the discussion on this pull request more focussed.

Great but I don't understand why create a script to convert the sources files instance of having a way to support ES6 modules natively as it's purpose here: #5226

@sbrunner - #5226 is not about supporting ES6 modules natively.  That pull request is about introducing another dependency on the Closure Library (for the definition of goog.module()).  We have worked very hard to remove the dependency on the Closure Library, and have no interest in brining it back.  For anybody interested in talking about goog.module() (or any other conventions from the Closure Library), please do so on separate threads.
This pull request is about publishing a more usable package.

Thank you all for your participation. To summarize for the ones that could not come, we discussed many topics, notably:

there is a common vision that the OpenLayers code will be written in ES6 with modules, compilable with the Google Closure Compiler;
transforms will be applied to this source code to generate other module formats, eventually transpiling code to ES5;
before this happens, we will generate experimental betas, without any guarantee (see this PR #6302);
https://github.com/openlayers/closure-util/pull/267 is intended for projects wanting to use goog.module, merging would avoid un unnecessary fork;
Camptocamp is switching project code to goog.module. There is no requirement to OpenLayers to switch;
the transforms to convert goog.provide code to goog.module will be contributed in ./transforms/; (after #6302);
https://github.com/openlayers/closure-util/pull/274 and https://github.com/openlayers/openlayers/pull/6294 port the OpenLayers code base to latest GCC release and is ready for review;
GCC is implementing the nodejs and browser module resolution algorithms (see https://github.com/google/closure-compiler/pull/2130);
there are issues with some OpenLayers enums when compiling project code with GCC, OpenLayers code should take / or accept strings for enums.

Please leave a comment if I misunderstood or forget something.
Happy hacking!

ŧhanks for the resume @gberaudo :-)

@tschaub, what needs to be done to finish this pull request? Anything I can help with? I would like to create a release soon, and make updating the ol package part of the release process.

It would be nice if the JS version of the compiler supported extraAnnotationNames

this should be fixed with the newest compiler release, so if #6294 is merged before a new ol release, then this package should compile with latest JS compiler.

one other point: I think the readme should document that webgl is disabled

I've updated the readme and associated gists.  I think this is ready to go.  We can shuffle things around to accommodate additional transforms for other packages if there is reason.  That and additional changes to this transform can be handled after this is merged.


Update blog tutorials code sample relating to front-matter package

Descriptions

I am following the instructions in the blog tutorial in remix.run blog tutorial
I just encountered the error below

There was an error running the data loader for route routes/posts/index
Error: Function yaml.safeLoad is removed in js-yaml 4. Use yaml.load instead, which is now safe by default.
    at ./node_modules/js-yaml/index.js:10:11
    at parse (./node_modules/front-matter/index.js:64:20)
    at extractor (./node_modules/front-matter/index.js:26:12)
    at ./build/index.js:402:58
    at async Promise.all (index 1)
    at async Object.loadRouteData (./node_modules/@remix-run/server-runtime/data.js:32:14)
    at async Promise.all (index 1)
    at async handleDocumentRequest (./node_modules/@remix-run/server-runtime/server.js:296:28)
    at async ./node_modules/@remix-run/server-runtime/server.js:71:20
    at async ./node_modules/@remix-run/express/server.js:44:22
GET /posts 500 - - 25.677 ms

I think it is the problem related to front-matter package.
For now, to make tutorial work properly, allowUnsafe option should includes in parseFrontMatter from front-matter

Also encountered this error while going through the tutorial, this fix works for me! thanks.


Reference implementation of batch norm with tests

I think this was just a test PR, please feel free to reopen if I am mistaken.

Yup, this was a test.


x11docker: 7.1.4 -> 7.4.2

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/mviereck/x11docker/releases.
meta.description for x11docker is: "Run graphical applications with Docker"
meta.homepage for x11docker is: "https://github.com/mviereck/x11docker"
meta.changelog for x11docker is: ""
Updates performed

Version update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
found 7.4.0 with grep in /nix/store/51h6yb366mzbjh493afghb1hsz3v9cnx-x11docker-7.4.0
found 7.4.0 in filename of file in /nix/store/51h6yb366mzbjh493afghb1hsz3v9cnx-x11docker-7.4.0

Rebuild report (if merged into master) (click to expand)

1 total rebuild path(s)

1 package rebuild(s)

First fifty rebuilds by attrpath
python310Packages

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/51h6yb366mzbjh493afghb1hsz3v9cnx-x11docker-7.4.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A x11docker https://github.com/r-ryantm/nixpkgs/archive/50a076a2b9d0780af414e7b1ad5da9814c93f7d1.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/51h6yb366mzbjh493afghb1hsz3v9cnx-x11docker-7.4.0
ls -la /nix/store/51h6yb366mzbjh493afghb1hsz3v9cnx-x11docker-7.4.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

x11docker

Maintainer pings

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/mviereck/x11docker/releases.
meta.description for x11docker is: "Run graphical applications with Docker"
meta.homepage for x11docker is: "https://github.com/mviereck/x11docker"
meta.changelog for x11docker is: ""
Updates performed

Version update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
found 7.4.2 with grep in /nix/store/hyf54hngyd2a2ns2zal0rbgkyxi1n8cr-x11docker-7.4.2
found 7.4.2 in filename of file in /nix/store/hyf54hngyd2a2ns2zal0rbgkyxi1n8cr-x11docker-7.4.2

Rebuild report (if merged into master) (click to expand)

1 total rebuild path(s)

1 package rebuild(s)

First fifty rebuilds by attrpath
x11docker

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/hyf54hngyd2a2ns2zal0rbgkyxi1n8cr-x11docker-7.4.2 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A x11docker https://github.com/r-ryantm/nixpkgs/archive/93841d5a4996f7313885c215ebb51c179ee8c4d5.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/hyf54hngyd2a2ns2zal0rbgkyxi1n8cr-x11docker-7.4.2
ls -la /nix/store/hyf54hngyd2a2ns2zal0rbgkyxi1n8cr-x11docker-7.4.2/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.


Understanding of the output of the NAS

Here is my question: after training, what should be the output of the nas? Is that the best one ever during the training or the last architecture produced by the model?

In my opinion, you can select the best one. The author chose the model from last epoch because they wanted to have a fair comparison with NAS.

I see @nott0

Hey, I have a problem understanding the final discovered architecture. I can see there is a directory called output and some meta, data, and index files are saved in there. However, none of them is not the discovered architecture.  does anybody have any idea what this optimum architecture is saved?

I can answer for the macro search. In the output directory as you say, all what you want is in the stdout file. At each epoch, ENAS gives 10 architectures with the corresponding valid accuracy (computed on the valid data set). What I do, is just search for the best valid accuracyand take the corresponding architecture.
To remind you, an architecture looks like that :
[1]
[4 0]
[2 0 0]
[0 0 0 0]
[4 0 0 0 0]
[2 1 0 1 1 0]
[0 0 1 1 0 0 1]
[2 0 0 1 0 1 0 0]
[4 0 0 0 0 1 1 1 0]
[0 0 0 0 0 0 0 0 0 1]
[5 1 0 0 0 1 1 0 1 1 0]
[0 0 1 0 0 0 0 1 1 0 0 0]
val_acc=0.9062

If it can be of any help, I made a script for analyzing the result of ENAS: https://gitlab.com/ElieKadoche/enas_game_of_go/blob/master/outputs_saved/graph_maker_script.py. It will create a picture like that one.

I can answer for the macro search. In the output directory as you say, all what you want is in the stdout file. At each epoch, ENAS gives 10 architectures with the corresponding valid accuracy (computed on the valid data set). What I do, is just search for the best valid accuracyand take the corresponding architecture.
To remind you, an architecture looks like that :
[1]
[4 0]
[2 0 0]
[0 0 0 0]
[4 0 0 0 0]
[2 1 0 1 1 0]
[0 0 1 1 0 0 1]
[2 0 0 1 0 1 0 0]
[4 0 0 0 0 1 1 1 0]
[0 0 0 0 0 0 0 0 0 1]
[5 1 0 0 0 1 1 0 1 1 0]
[0 0 1 0 0 0 0 1 1 0 0 0]
val_acc=0.9062

If it can be of any help, I made a script for analyzing the result of ENAS: https://gitlab.com/ElieKadoche/enas_game_of_go/blob/master/outputs_saved/graph_maker_script.py. It will create a picture like that one.

Hey, your answer was quite helpful. I have one more question. The macro-search script is discovering the best architecture. The discovered architectures have 12 layers, but while they are running the macro-final script to train their model, their architecture has 24 layers. why they have changed the number of layers and how they find the best operation and connections for layers 13 to 24? Also, they are not using pooling( neither max nor average) when they are training.  This is contradicting to their discovered architecture.


Join a case - Changes

Hi @suresh12  @Taherabharmal @devarajd94
Can you please add the detailed requirement @nitish-beehyv will be working on it

Initial Join:

Simplify the process: Remove the question about joining as a party in person or through an advocate.
Default to Party in Person: Automatically set the user as a party in person by default.

Disable the following question and by default have “Yes” auto selected
“Will you be representing yourself (Party in Person)?
Yes
No, I have an Advocate”
Affidavit Requirement:

Make the affidavit optional: Allow users to proceed by entering a secret code to view the complaint copy.
Informative message: Display a subline message informing users that they can upload an affidavit later through a separate filing process.

“After joining with the secret code, you can upload an affidavit through a separate filing process. Alternatively, if you have an advocate, they can join the case using the filing number and a secret code obtained from their login.”

Hi @Ramu-kandimalla  @rajeshcherukumalli As discussed with @nitish-beehyv,  Tested this issue in dpg-dev, Working as expected.
Covered Cases,

Verify that Joining as a party in person or through an advocate question is removed -- Questions is removed working as expected
Verify that Affidavit screen is removed. -- Affidavit screen is removed Working as expected

Ref,
https://jam.dev/c/9e07d4db-a8ce-4905-b14a-f75e97c829c5

Hi @Ramu-kandimalla  @rajeshcherukumalli , Tested the issue in UAT environment, Working as expected.
Covered Cases,
Verify that Joining as a party in person or through an advocate question is removed -- Questions is removed working as expected
Verify that Affidavit screen is removed. -- Affidavit screen is removed Working as expected


Add program to recursively add 2 numbers without using addition operators

This pull request addresses issue #8.
Tests and comments explaining the code have been included.
Hacktoberfest contribution.
could the label "hacktoberfest" be added
Thanks.

Perfect @RandyKdev


replace 'problems' key with 'exercises' in config.json

see #22

I forgot to point out that as of exercism/x-api#137, it is safe to remove problems.

This is great, thanks @cgrayson!


docs(v2): add label option for docs

Motivation
Change docs accordingly.
Have you read the Contributing Guidelines on pull requests?
Yes.
Related PRs
https://github.com/facebook/docusaurus/pull/2374

Please add the docs for it along with the #2374. To be honest I'm not really in favor of this feature given that it will make the docs plugin more complicated.


hive: Fix start hook log output

Commit 07595dae12fc added module ID to the start/stop hook output by carrying the module ID in the hook. Unfortunately I forgot to fix the getHooKFuncName to actually handle this wrapped handle which caused the log output to always show "augmentedHook.Start" as the function name. Fix this by looking through the wrapper.
Fixes: 07595dae12fc ("hive: Add module ID to hive start/stop hook output")

/test


folder convention

are we gonna call the directory under 'archives' with the full url or with the name of the mailing list?
e.g.
http://www.ietf.org/mail-archive/text/ietf-privacy or ietf-privacy?
I remember we agreed on the first, but collect_mail still follow the second convention (while notebooks are totally inconsistent on that)

I believe this is because we tried to provide a convenience to the user.
If you supply a full URL in the notebook and the data has not yet been downloaded, BigBang will download it for you (it calls this load_data method). If the data has been downloaded, it will use the data you have locally:
https://github.com/datactive/bigbang/blob/master/bigbang/mailman.py#L49
If you give it the shorter name of the list (i.e. "ietf-privacy") then it can still find the files if they are in the right archive subdirectory.
I am +0 on using full URLs in the notebook as a convention.

The new notebooks have the names of the lists and relative paths to 'archives'.


APPS-534 poll analysis on reorg

The analysis is updated in the backend asynchronously, so we need to check the dependsOn field to determine if it is still waiting on the output of a previous stage before we can proceed with the reorg.
This PR also updates to the latest dxScala dependencies.

The analysis is updated in the backend asynchronously, so we need to check the dependsOn field to determine if it is still waiting on the output of a previous stage before we can proceed with the reorg.
This PR also updates to the latest dxScala dependencies.

The analysis is updated in the backend asynchronously, so we need to check the dependsOn field to determine if it is still waiting on the output of a previous stage before we can proceed with the reorg.
This PR also updates to the latest dxScala dependencies.


net class: split out attribute parser and export it

Can be used by other projects (eg node_exporter) to read a single sysfs file rather than all of them at once, which can cause lock contention in the kernel as many sysfs net files grab the RTNL lock.
Immediate consumer: https://github.com/prometheus/node_exporter/pull/2669

@discordianfish is this what you were thinking of?


Integration

All Pull Requests:

[ ] Have you considered Security (e.g. XSS)?
[ ] Have you considered desktop, mobile and tablet form-factors?
[ ] Have you considered Accessibility best practices?

Code

[ ] Are the commits squashed into one commit?
[ ] Is the branch name and commit message following team convention?
[ ] Is the code linted?
[ ] Are all open issues, questions and concerns by Reviewers answered, resolved and reviewed?

Quality Assurance

[ ] If applicable, are Unit Tests written and all tests are passing?
[ ] If applicable, are Integration Tests written and all tests are passing?
[ ] If applicable, are UI Automation Tests written and all tests are passing?

Documentation

[ ] Are there meaningful code comments included?
[ ] If applicable, are test cases written?
[ ] If applicable, is Change Log updated?
[ ] If applicable, are UI Texts reviewed by Documentation?

Security

[ ] If applicable, has Security reviewed this code?
[ ] If applicable, is a 3PP request submitted?

Some conflicts must be resolved?

Some conflicts must be resolved?

@coopaq yes, unsure why. We need to make sure that in the future if anyone merges something to master separate from integration, that change also gets into integration asap. Also, I discovered that it is impossible to use github to resolve a merge conflict between two protected branches. I had to disable integration's protection to resolve the conflict.


Mac OS X Mojave - unexpected delete key behaviour

Which operating system does the issue occur on?
OS X 10.14
The delete key moves the cursor a few spaces right instead of deleting as shown here:

This is with the default mapping which worked as expected when I tried it on a machine running High Sierra (exact same alacritty config).

Since this might be relevant here, which shell are you using? Is it just standard bash?

My guess would be that Mojave updated the terminfo for xterm-256color and that broke what appears to be zsh in Alacritty. @sganondorf, assuming that is indeed zsh, could you try adding this to your ~/.zshrc?
key[Home]="$terminfo[khome]"
key[End]="$terminfo[kend]"
key[Insert]="$terminfo[kich1]"
key[Backspace]="$terminfo[kbs]"
key[Delete]="$terminfo[kdch1]"
key[Up]="$terminfo[kcuu1]"
key[Down]="$terminfo[kcud1]"
key[Left]="$terminfo[kcub1]"
key[Right]="$terminfo[kcuf1]"
key[PageUp]="$terminfo[kpp]"
key[PageDown]="$terminfo[knp]"

# setup key accordingly
[[ -n "$key[Home]"      ]] && bindkey -- "$key[Home]"      beginning-of-line
[[ -n "$key[End]"       ]] && bindkey -- "$key[End]"       end-of-line
[[ -n "$key[Insert]"    ]] && bindkey -- "$key[Insert]"    overwrite-mode
[[ -n "$key[Backspace]" ]] && bindkey -- "$key[Backspace]" backward-delete-char
[[ -n "$key[Delete]"    ]] && bindkey -- "$key[Delete]"    delete-char
[[ -n "$key[Up]"        ]] && bindkey -- "$key[Up]"        up-line-or-history
[[ -n "$key[Down]"      ]] && bindkey -- "$key[Down]"      down-line-or-history
[[ -n "$key[Left]"      ]] && bindkey -- "$key[Left]"      backward-char
[[ -n "$key[Right]"     ]] && bindkey -- "$key[Right]"     forward-char

# Finally, make sure the terminal is in application mode, when zle is
# active. Only then are the values from $terminfo valid.
if (( ${+terminfo[smkx]} )) && (( ${+terminfo[rmkx]} )); then
    function zle-line-init () {
        echoti smkx
    }
    function zle-line-finish () {
        echoti rmkx
    }
    zle -N zle-line-init
    zle -N zle-line-finish
fi

Sorry, I should have specified: I am using zsh.
@jwilm I tried adding that to my .zshrc but it didn't change anything for me unfortunately.
I checked and this does not happen when using bash.
I did realize I had
TERM: xterm-256color-italic
in my config.
Commenting this out so that xterm-256color is used is what fixed the issue for me.
Thank you for taking a look at this!

@sganondorf If the xterm-256color-italic terminfo is the only one causing issues, I would assume that it's a problem with that terminfo file. Can you confirm that Alacritty's standard terminfo (alacritty) also works?
And if it really is the fault of xterm-256color-italic, could you test if it shows similar behavior in other terminal emulators?
Also just making sure that xterm-256color-italic exists is probably a good idea. Zsh will warn you if you do export TERM=xterm-256color-italic when it doesn't exist, so that's probably the best way to test this.

xterm-256color-italic did not exist which was the source of my issue, thank you @chrisduerr for that quick method of checking. The issue was solved when I added it myself. Likewise, once I installed alacritty it worked as well.
I may have misunderstood the problem here a bit, so I think this issue can probably be closed 🙃 thank you both!

No problem, glad we could sort this out! :)


Update ForC simplified

@ValentineHerr I'll be using the ForC simplified data in my analysis - I'm currently looking at sample sizes of different variables to plan the analysis, so it would be helpful to have the updated ForC simplified to work with. Is it okay for me to update using your script, or is this something you prefer to do yourself?

@beckybanbury,
I am still working on the duplicate records and a few other things. I feel that I am many steps away from getting it ready but maybe I should just go ahead and update the measurements with what I have so far. That will always be better than what the current version is.
There will be a few manual steps and you will probably be the best person to do them. Once you do those, you can update ForC simplified yourself.
I think I should be able to get that done today and I can post a specific issue for the steps you will have to do.
In the mean time, you can already update ForC simplified now (I think the code still works but let me know if it doesn't, there might be some issue with PLOTS column names), knowing it is not the best version yet, and start working on your scripts, both for looking at sample size (I don't think there will be major changes) and for your analysis. It will b easy to re-run them on the better version.

@ValentineHerr thanks! I've got the code to work, but currently the code is excluding the data Abby and I entered because the "conflicts" column isn't coded. I presume this will be sorted once the duplicates code is run on that data, or is it a different code that I need to run?

FYI, I just updated PLOTS and saw that you had just done it too. I don't think that it will cause problems because it should basically be the same thing but I wanted to point it out just in case.

All right... This is not happening...
On line of code that takes 2 minutes to run at work is taking more than 3 hours now and is still not completed on my machine.
I tried to run it on the Rstudio Cloud (pretty cool thing by the way) hoping that the server would run it quickly but it does not. Maybe it is because my connection is no amazing...
So, @beckybanbury, please:

[ ] run scripts\z_archive\Add_D.precedence.measurement.ID.R (you will have to install.packages("lubridate")) --> measurements should be updated --> commit the change in measurements with "add and assign D.precedence.measurement.ID"

[ ] open scripts\z_archive\ID_sets_of_duplicate_records.R and run until line 143. If you get this line to be done in a few minutes, that is great... if not, please let me know. If it works, continue running the rest and let me know if you find it confusing.  You can ignore the first one. The second message should tell you that "There is 76 groups where D precedence needs to be given manually." If you don't get 76 (or close) let me know.

[ ] If things go smoothly above, run the last line that is commented out (write.csv(MEASUREMENTS.final, "data/ForC_measurements.csv", row.names = F)) to save the measurements --> commit  the change in measurements with "update conflict related columns automatically"

[ ] Now you have to manually edit the D.precendence and D.precedence.measurement.ID of those D.groups that are unclear. you can see what the groups are by running again line 1484 to 1488. It will show them to you one by one, you have to press enter to go to the next one. You need to find those records by hand (in Excel) to decide what record gets precedence (probably the one you entered but it needs review). Once you picked what record is the best in one group, give it a "1" in D.precedence and give a "0" to the other records of the group. Then (very important) copy the measurement.ID of that record and paste it in D.precedence.measurement.ID for all the records of the group (including himself).

[ ] save and commit your changes with "manually edit the D.precendence and D.precedence.measurement.ID "

[ ] if all goes well, push and let me know yo are done. Next step for me is to update the script that reconcile the duplicates.

@ValentineHerr I have managed to get the code to work and get a pop-up that says I have 76 groups where D precedence has to be given manually. However, I think there may be a problem with the sections of codes that read like this (lines 1468 and 1484):
for(split.ID in missing.D.precedence.split.ID) {
X <- MEASUREMENTS.final[MEASUREMENTS.final$split.ID %in% split.ID & grepl("D", MEASUREMENTS.final$conflicts), ]
print(X)
readline()
}
From what you wrote above the whole script should run and then it is possible to go back and re-run these sections to see the groups. When I run these sections it's getting stuck + I have to manually press enter on the console to get it to run through. This means if I run the whole code, rather than line by line, it's getting stuck + not giving the right output beyond this point. I've managed to get it all to run by doing it line by line, but I'm guessing this is not ideal for the future?

@ValentineHerr I've saved the ForC_measurements file and committed it to my branch because I think that in a few cases the code may have overwritten previously assigned D precedence. It would be great if you could review the changes and let me know!

@beckybanbury thank you so much for running the code. I am glad it worked on your machine.
You are right, "readline()" in line 1484 should be commented out by default so the code runs smoothly for someone who won't be looking at those 76 groups of duplicates one by one.
I will review those overwritten D precendence. Hopefully it happens only when I agreed it should, meaning for the following measurement.ID:
"11525;11541", # Abby Ferson weird assignment of R group
"11542;11543;11552;11560", # Abby Ferson weird assignment of R group
"11576;19886;19887;19888", # Abby Ferson weird assignment of R group
"11647;11652", # Abby Ferson weird assignment of R group
"3229;3236;3239;12103",
"3230;3237;3240;12104",
"8816;17416;17480", # dat NA, start date 2012, 1992, en data 2012, 2012...
"3510;3511;3512;17378",
"1684;1685",
"15278;15279;15280;15281;15290;15291;15293;15294;15295"
I will keep you posted.

I pushed a code that will help comparing master version and your branch version (output of "ID_sets_of_duplicate_records.R") here
.

So there are a few D.precendence that have been changed because new entries (yours) came into play and are now part of previously existing D.groups. They are pretty complicated cases that my code does not handle well. You will have to review them by hand. Here are the 3 plots:

sites.sitename
plot.name
variable.name

Harvard Forest - Little Prospect Hill
EMS Tower (HFR1)
GPP_C

Le Bray
Managed 1970 pine plantation
GPP_C

Soroe
Managed
GPP_C

Keep what you find in a separate file because I'll be editing the code (see bellow) so you will have to do it again (you can also wait until I am done with task bellow). Make sure you enter conflict.notes adequatly (look at example of notes in the current measurement file)

There is some other issues that should be fixed in my code (D.groups "NAC" while there is a D.precedence). I think I know what is going on but I need to work on the code. For that I need your help @beckybanbury... Could you run  scripts\z_archive\ID_sets_of_duplicate_records.R again (on the master version of measurements, not the verion of your branch) until line 143 and then run:
save(MEASUREMENTS.split, file = "scripts/z_archive/MEASUREMENTS.split.Rdata")

commit and push that new file so that I can access it without having to run line 143.
Thanks !
Let me know if you have any questions!

@ValentineHerr I'm having trouble pushing the MEASUREMENTS.split.Rdata file because it's so large (0.2 GB). Is there a different way to get you the information? I'll keep trying but GitHub desktop isn't doing well with it!

hmm... did you try to drag the file directly into GitHub online?
you can try  saveRDS(MEASUREMENTS.split, file = "scripts/z_archive/MEASUREMENTS.split.rds") and see if that is smaller

For a file that large you probably want to set up Git LFS.


2022 07 01

Setting charset in itself does not fix problem, but leaving as it is the correct thing to do.


ERROR: Failed building wheel for miniaudio

Preconditions
Requirements fulfilled
The known issues did not answer my question
The question has not been asked in the past or the answer does not satisfy me
There is currently no other opened issue that matches my question
I cannot find any answer to my question in GitHub discussions
Current Situation
The Plugin is rebooting all the the time with the error code:
ERROR: Failed building wheel for miniaudio
Logs
Ã� Building wheel for miniaudio (pyproject.toml) did not run successfully.
â�� exit code: 1
â�°â��> [20 lines of output]
/tmp/pip-build-env-rww0rpmc/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py:261: UserWarning: Unknown distribution option: 'tests_require'
warnings.warn(msg)
/tmp/pip-build-env-rww0rpmc/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py:261: UserWarning: Unknown distribution option: 'test_suite'
warnings.warn(msg)
running bdist_wheel
running build
running build_py
creating build/lib.linux-aarch64-cpython-39
copying miniaudio.py -> build/lib.linux-aarch64-cpython-39
running build_ext
generating cffi module 'build/temp.linux-aarch64-cpython-39/_miniaudio.c'
creating build/temp.linux-aarch64-cpython-39
building '_miniaudio' extension
creating build/temp.linux-aarch64-cpython-39/build/temp.linux-aarch64-cpython-39
aarch64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -DMA_NO_GENERATION=1 -DMA_NO_ENCODING=1 -DMA_NO_RESOURCE_MANAGER=1 -DMA_NO_NODE_GRAPH=1 -DMA_NO_ENGINE=1 -I/tmp/pip-install-fj1d24or/miniaudio_01e5abf8e3934fa5b934a5fbf2fe631d -I/tmp/pip-install-fj1d24or/miniaudio_01e5abf8e3934fa5b934a5fbf2fe631d -I/var/lib/homebridge/appletv-enhanced/.venv/include -I/usr/include/python3.9 -c build/temp.linux-aarch64-cpython-39/_miniaudio.c -o build/temp.linux-aarch64-cpython-39/build/temp.linux-aarch64-cpython-39/_miniaudio.o -g1 -O3 -ffast-math
build/temp.linux-aarch64-cpython-39/_miniaudio.c:50:14: fatal error: pyconfig.h: No such file or directory
50 | # include <pyconfig.h>
| ^~~~~~~~~~~~
compilation terminated.
error: command '/usr/bin/aarch64-linux-gnu-gcc' failed with exit code 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
[10/29/2024, 11:20:04 AM] [Apple TV Enhanced] [E] Python check: ERROR: Failed building wheel for miniaudio
[10/29/2024, 11:20:04 AM] [Apple TV Enhanced] [I] Python check: Failed to build miniaudio
[10/29/2024, 11:20:04 AM] [Apple TV Enhanced] [E] Python check: ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (miniaudio)
[10/29/2024, 11:20:06 AM] [Apple TV Enhanced] [S] Python check: Python requirements installed.
[10/29/2024, 11:20:06 AM] [Apple TV Enhanced] [S] Python check: Finished
[10/29/2024, 11:20:06 AM] [Apple TV Enhanced] [I] Platform: Starting device discovery ...
Error: spawn /var/lib/homebridge/appletv-enhanced/.venv/bin/atvscript ENOENT
at Process.ChildProcess._handle.onexit (node:internal/child_process:286:19)
at onErrorNT (node:internal/child_process:484:16)
at processTicksAndRejections (node:internal/process/task_queues:82:21)
[10/29/2024, 11:20:06 AM] [Apple TV Enhanced] Child bridge process ended
Nothing change
Operating System
Linux Homebridge
Operating System: Bits
64-bit
Operating System: Distribution
Pi OS
Operating System: Distribution Version
12
Docker
yes
Docker Image
homebridge/homebridge
Docker Image Tag
No response
Homebridge Version
1.8.5
Homebridge Config UI Version
v4.62.0
Homebridge Storage Path
/var/lib/homebridge
Homebridge Apple TV Enhanced Version
1.9.1
Node Version
v20.18.0
NPM Version
10.8.2
Python Version
3.9.5
PIP Version
24.3.1
HDMI Hops
No response
Audio Output
no
Same Subnet
yes
Additional Context
No response

The amount of information given is astonishing.

Install python-dev apt package by executing sudo apt install python-dev and try again.

Thank you for the help
Than it says:
Reading package lists... Done Building dependency tree
Reading state information... Done
Package python-dev is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or is only available from another source However the following packages replace it:
python2 python2-dev
E: Package 'python-dev' has no installation candidate

Is python3-dev available?

Yes
after doing   apt-get install python3-de
It says :
Reading package lists... Done Building dependency tree
Reading state information... Done
python3-dev is already the newest version (3.8.2-0ubuntu2) •
The following packages were automatically installed and are no longer required: libpython2-dev libpython2.7 libpython2.7-dev python2-dev python2. 7-dev
Use 'apt autoremove'
to remove them.
0 upgraded, 0 newly installed,
0 to remove and 0 not upgraded.


fixed test-network-k8s/scripts/test_network.sh missing  ${NGINX_HTTPS…

fixed test-network-k8s/scripts/test_network.sh missing  ${NGINX_HTTPS}
fixed test-network-k8s/scripts/channel.sh missing  ${NGINX_HTTPS}

Thank you, @wisecoach . Please see the instructions here on how to address the DCO issue.


[GH-558] Fix issue: /jira assign should accept @username

Summary
Most of the changes in this PR are from PR #863. We just added the support to search for Jira users in case the @ symbol is not present as the prefix in the username (i.e. the approach which was being used earlier). Also, we fixed a small issue, in case the mentioned user account was not connected to Jira, the Jira issue was getting unassigned.
Demo video
Demo video for updated flow
Ticket
Fixes #558 ( /jira assign should accept @username)

@mickmister Fixed the review comments mentioned by you

This PR has been automatically labelled "stale" because it hasn't had recent activity.
A core team member will check in on the status of the PR to help with questions.
Thank you for your contribution!

@DHaussermann Gentle reminder for review

@DHaussermann Gentle reminder for review

@DHaussermann Gentle reminder for review

@DHaussermann Gentle reminder for review

@DHaussermann Gentle reminder for review

@AayushChaudhary0001 Would you be open to reviewing this PR?


Update plugin to 2.2.0 (2019-09-09)

Depends on #152 .
Changes to examine:

[ ] https://github.com/typescript-eslint/typescript-eslint/pull/810

Closed with #162 .


Why rubinius  is always cloned and failed?

The errors:
zhangysh1995@ubuntu-zhangyushao:~$ crawld -c crawld.conf
E0314 10:19:15.173304    6197 crawld.go:79] impossible to clone https://github.com/rubinius/rubinius.git in /var/crawld/c/rubinius/rubinius (network error) skipping
E0314 10:19:16.711802    6197 github.go:253] GET https://api.github.com/search/repositories?per_page=100&q=language%3Ac&sort=stars: 401 Bad credentials []
E0314 10:19:17.156407    6197 github.go:253] GET https://api.github.com/search/repositories?per_page=100&q=language%3Acpp&sort=stars: 401 Bad credentials []

After these two lines of information, the program is running but cannot see any outputs.
My configuration file:

{
    "database": {
        "hostname": "localhost",
        "port": 5432,
        "username": "crawler",
        "password": "crawler",
        "dbname": "crawler",
        "ssl_mode": "disable"
    },
    "clone_dir": "/var/crawld",
    "crawling_time_interval": "1h",
    "fetch_time_interval": "5m",
    "fetch_languages": [
        "c",
        "cpp"
    ],
    "tar_repositories": false,
    "tmp_dir": "/ramdisk",
    "tmp_dir_file_size_limit": 2.0,
    "max_fetcher_workers": 2,
    "throttler_wait_time": 900,
    "throttler_sliding_window_size": 60,
    "throttler_leak_interval": 1000,
    "crawlers": [
        {
            "type": "github",
            "languages": [
                "c",
                "cpp"
            ],
            "limit": 0,
            "since_id": 0,
            "fork": false,
            "oauth_access_token": "token here",
            "use_search_api": true
        }
    ]
}

Hi @zhangysh1995,
Not sure. I haven't used this project for about 2 years. I'll try to give it a shot one of these days and figure out the issue.


16-bit quantization issue

My question is whether Distiller support 16-bit quantization or not?
It works perfectly well with 8-bit quantization but things go wrong with 16-bit quantization scheme.
As I check the parameters of the wrapped modules from "quantized_checkpoint.pth.tar" file, only some of them are "truly" integer numbers.
My model is resnet20 and my command line is:
python3 compress_classifier.py --arch resnet20_cifar ../data.cifar --qebw 16 --qeba 16 --resume-from ./resnet20/resnet20_32fp_cifar.pth.tar --quantize-eval --evaluate --qe-mode sym
Thank you.

Hi @lehahoang,
You shouldn't be seeing any difference between 8 and 16-bits in terms of parameter values being integer numbers. But you need to make sure you're looking at the right parameters:

You should look only at parameters that end with ".weight" and ".base_b_q". Note you shouldn't look at ".bias". In short, since you're running dynamic quantization (not passing a stats file via --qe-stats-file), we can only save a "baseline" quantized bias, and we re-quantize that baseline with each new batch that comes in. We save that baseline in the ".base_b_q" parameter. The math behind this is detailed here.
Only convolution and linear layers are relevant. We don't quantize BatchNorm layers. So BN parameters stay in FP32. (Note that if you're running the latest code from master, which already has batch-norm folding, this shouldn't be an issue with ResNet-20. All BN modules should be eliminated prior to quantization)

I just tried and it works fine for me. Considering the points above, do you still see the problem?

Here's the code I used to check:
import torch
import distiller

cp_8b = torch.load('PATH-TO-8B-QUANTIZED-CHECKPOINT)
cp_16b = torch.load('PATH-TO-16B-QUANTIZED-CHECKPOINT)

for k in cp_8b['state_dict']:
    if not(k.endswith('weight') or k.endswith('base_b_q')):
        continue
    p_8b = cp_8b['state_dict'][k]
    p_16b = cp_16b['state_dict'][k]
    print(k)
    print(' 8b: ', (p_8b.round() == p_8b).all().item())
    print(' 16b:', (p_16b.round() == p_16b).all().item())

You should see an output with all 1's, something like:
module.conv1.base_b_q
 8b:  1
 16b: 1
module.conv1.wrapped_module.weight
 8b:  1
 16b: 1
module.layer1.0.conv1.base_b_q
 8b:  1
 16b: 1
module.layer1.0.conv1.wrapped_module.weight
 8b:  1
 16b: 1
...

Hi Jacob,
Thanks for your kind reply. I have considered all the points you enlisted but problem is unsolved at the moment. I have followed the instruction from this link:
To make it more clear, I would like to summarize steps I have done.

Step 1: I used command lines below to generate 16-bit quantized model:

$ python3 compress_classifier.py --arch resnet20_cifar ../data.cifar –resume-from resnet20/resnet20_32fp_cifar.pth.tar  --qe-calibration 0.05
$ python3 compress_classifier.py --arch resnet20_cifar ../data.cifar –resume-from resnet20/resnet20_32fp_cifar.pth.tar  --evaluate --quantize-eval –qe-stats-file acts_quantization_stats.yaml --qe-per-channel --qe-bits-acts 16 --qe-bits-wts 16
Please note that resnet20_32fp_cifar.pth.tar is the model I trained by myself.

Step 2: I took note the log file of quantization process

Log file:
Loading activation stats from: acts_quantization_stats.yaml
Preparing model for quantization using PostTrainLinearQuantizer
Parameter 'module.conv1.wrapped_module.weight' will be quantized to 16 bits
Parameter 'module.layer1.0.conv1.wrapped_module.weight' will be quantized to 16 bits
Parameter 'module.layer1.0.conv2.wrapped_module.weight' will be quantized to 16 bits
Parameter 'module.layer1.1.conv1.wrapped_module.weight' will be quantized to 16 bits
Parameter 'module.layer1.1.conv2.wrapped_module.weight' will be quantized to 16 bits
Parameter 'module.layer1.2.conv1.wrapped_module.weight' will be quantized to 16 bits
Parameter 'module.layer1.2.conv2.wrapped_module.weight' will be quantized to 16 bits
Parameter 'module.layer2.0.conv1.wrapped_module.weight' will be quantized to 16 bits
Parameter 'module.layer2.0.conv2.wrapped_module.weight' will be quantized to 16 bits
Parameter 'module.layer2.0.downsample.0.wrapped_module.weight' will be quantized to 16 bits
Parameter 'module.layer2.1.conv1.wrapped_module.weight' will be quantized to 16 bits
Parameter 'module.layer2.1.conv2.wrapped_module.weight' will be quantized to 16 bits
Parameter 'module.layer2.2.conv1.wrapped_module.weight' will be quantized to 16 bits
Parameter 'module.layer2.2.conv2.wrapped_module.weight' will be quantized to 16 bits
Parameter 'module.layer3.0.conv1.wrapped_module.weight' will be quantized to 16 bits
Parameter 'module.layer3.0.conv2.wrapped_module.weight' will be quantized to 16 bits
Parameter 'module.layer3.0.downsample.0.wrapped_module.weight' will be quantized to 16 bits
Parameter 'module.layer3.1.conv1.wrapped_module.weight' will be quantized to 16 bits
Parameter 'module.layer3.1.conv2.wrapped_module.weight' will be quantized to 16 bits
Parameter 'module.layer3.2.conv1.wrapped_module.weight' will be quantized to 16 bits
Parameter 'module.layer3.2.conv2.wrapped_module.weight' will be quantized to 16 bits
Parameter 'module.fc.wrapped_module.weight' will be quantized to 16 bits
Parameter 'module.fc.wrapped_module.bias' will be quantized to 32 bits

Step 3: I loaded the quantized model and checked the parameters in [‘state_dict’]

module.conv1.wrapped_module.weight
module.bn1.weight
module.layer1.0.conv1.wrapped_module.weight
module.layer1.0.bn1.weight
module.layer1.0.conv2.wrapped_module.weight
module.layer1.0.bn2.weight
……

Step 4: I print out the values of wrapped model and received the result as follow

$ print(cp_16b['state_dict']['module.conv1.wrapped_module.weight'])
tensor([[[[-1.1674e+04,  1.3735e+04,  1.7130e+04],
[-2.1960e+04, -2.3083e+04,  3.1799e+04],
[-2.3258e+04, -2.0422e+04,  1.6858e+04]],
…….
Step 5: I tried your test code and here is the result:
module.conv1.wrapped_module.weight
8b:  1
module.bn1.weight
8b:  0
module.layer1.0.conv1.wrapped_module.weight
8b:  1
module.layer1.0.bn1.weight
8b:  0
…..
It is a bit unusual that the parameter ends with ‘b_q’ is missing in 16-bit quantization scheme. However, I found it in case of 8-bit quantization scheme
Do you think I have missed any important step?
Thanks

You didn't miss any steps, and actually everything you're seeing is as expected:

tensor([[[[-1.1674e+04, 1.3735e+04, 1.7130e+04],
...
module.conv1.wrapped_module.weight
8b: 1
So actually we can see that the weights are indeed integer values. When I saw the tensor values that were printed I was initially confused, then realized these are actually integer values: -1.1674e+04 = 11674, etc. They're just printed in scientific notation.
module.bn1.weight
8b: 0
As I mentioned above, batch norm weights are not quantized. So this is also expected. Please update to the latest version of the code - we added batch-norm folding last week. When you run with the latest code these BN parameters will be gone from the quantized checkpoint.
Regarding base_b_q - The fact that you're not seeing it is not related to 16 vs. 8 bit. This is indeed confusing, I'll try to explain.
The original command line you posted was using "dynamic" quantization, and as I explained in my first response, one of the implications of that is that we save this base_b_q parameter, which we re-quantize for every batch.
This time you ran with "static" quantization - you used a stats file. In this case, we can do a one-time quantization of the bias upfront. We don't need to re-quantize for every new input. So in this case we don't save the base_b_q parameter. We directly quantize the bias parameter. So if you'd change the code I posted above to look for bias instead of base_b_q, you'll see those parameters have integer values as well. Again, I realize this is confusing - I hope I was able to clarify what's going on.

Hi Jacob,
Thanks for your clarification. I think this problem is solve now.


Images txt should not source blank image strings

If you don't pass in a imageArg (like the agent) and don't set REPO env param you will error out the script. But this could happen for other reasons, and if so we should ignore the image.

Not backporting, but will merge master.


Use defined constant in open garage

Description:
Use defined constant in open garage
Checklist:

[x] The code change is tested and works locally.
[x] Local tests pass with tox. Your PR cannot be merged unless tests pass
[x] There is no commented out code in this PR.
[x] I have followed the [development checklist][dev-checklist]

Yes, actually it does not seems to be needed.
I will do some more refactoring of the code


Nugetize!

Would love to have the compiled binary available as a NuGet package.
I'm happy to help with the creation of this if you want.

PR Raised for this: https://github.com/lobrien/TensorFlow.Xamarin.Android/pull/3