Correct typespecs

This corrects the typespecs, making it work with Dialyzer.
You can add Dialyzer to the project itself in dev and test environments (but not prod) by adding this to deps():
{:dialyxir, "~> 1.0", only: [:dev, :test], runtime: false}

then run
mix dialyzer

Updated branch


Remove duplicate if statement

Remove duplicate if statement.

I'll keep this because i like semantic chunks of code but ty


Error creating sidecar proxy for pods in remote cluster in Primary-Remote istio Multicluster Setup

Hi,
I am trying to setup Primary-Remote istio Multicluster Setup following the documentation of istio multiclter on AWS EKS:
https://istio.io/latest/docs/setup/install/multicluster/primary-remote/
However, when the pods in the remote cluster are starting up, the istio side car container is not coming up and I am getting the below error when I describe the replica set. Error:
"13m         Warning   FailedCreate        replicaset/sample-77d6f96495   Error creating: Internal error occurred: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://xyz.us-west-2.elb.amazonaws.com:15017/inject/cluster/cluster2/net/common-network?timeout=10s": x509: certificate is valid for istiod.istio-system.svc, istiod-remote.istio-system.svc, istio-pilot.istio-system.svc, not xyz-2122434418.us-west-2.elb.amazonaws.com"
xyz-2122434418.us-west-2.elb.amazonaws.com is the load balancer dns name of the Gateway which was provisioned in Primary Cluster (to expose the control plane)
When the sidecar is provisioned, the istiod (in primary cluster) is logging the below error msg:
"info    http: TLS handshake error from <IP_ADDRESS>:44892: remote error: tls: bad certificate"
<IP_ADDRESS> is the Pod IP of the eastwest gateway which is provisioned in primary cluster.
I checked the root certs from Mutatingwebhook both in Primary and remote clusters and those certs seems to be the same. So my guess is, the certs is not the issue here.
Any help in resolving this issue will be appreciated. Thanks a lot !
Thanks,
Hafeez

how do you create the mutatingwebhookconfiguration? For aws case, i think you should not depend on istio to self sign cert.

Almost forget there is an env ISTIOD_CUSTOM_HOST that allows setting custom host


Install dependencies

I have a working go installation. However when I run thyme dep, it gives the following message:
Install the following command-line utilities via your package manager (e.g., apt) of choice:
* xdpyinfo
* xwininfo
* xdotool
* wmctrl

However all these utilities are already installed. I am not a go programmer so maybe I am missing something.

Thyme dep simply lists dependencies, it's not an error.
See also #29

apt-get install xdpyinfo xwininfo xdotool wmctrl fails with
E: Unable to locate package xdpyinfo
E: Unable to locate package xwininfo

However it turns out (after googling, and then installing apt-file to discover which package provides these programs) that those programs were already installed (via x11-utils).
Suggested fix: check for which dependencies are actually installed already and indicate which ones need to be installed. Perhaps also mention the deb/rpm/etc package names to save users ~10 mins.
I think between this and the issue I posted the other day I've spent 30 mins trying to get a time tracking program working, which is kinda funny :P

Yep, absolutely agree, thanks for pointing this out. I'll unfortunately have limited time to work on this in the next couple of weeks, but I think this should be an easy fix. Thyme just needs to detect if the commands are installed (via LookPath()) before printing out the instructions. The only function you need to modify is LinuxTracker.Deps.

After seeing some of the other issues folks have reported, I'm a little hesitant to auto-install dependencies. I'd like to wait for things to stabilize a bit. For now, I'd prefer to keep manual installation instructions, as they're more flexible and less prone to break as the dependencies might change. See the last comment here: https://github.com/sourcegraph/thyme/pull/48.
If anyone has strong disagreements, feel free to re-open and comment.


feat: key management for Ed25519 and X25519

Description:
!!! DO NOT MERGE !!! - until tested with Node implementation
Support Ed25519 and X25519 key type of managed DID via curve parameter when managing DID using REST interface.

new key types are generated while secp256k1 keys are derived
add ManagedDIDService.getDIDKeyPair to get all available key pair types
TODO: make another PR with tests added

Alternatives Considered (optional):
Link to existing ADR (Architecture Decision Record), if any. If relevant, describe other approaches explored and the selected approach. Documenting why the methods were not selected will create a knowledge base for future reference, helping prevent others from revisiting less optimal ideas.
Checklist:

[ ] My PR follows the contribution guidelines of this project
[ ] My PR is free of third-party dependencies that don't comply with the Allowlist
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] I have checked the PR title to follow the conventional commit specification

@patlo-iog , could you revise the description as it shouts 'DO NOT MERGE' but the PR was still merged.


Disable Animation

Please allow to disable animation all together.
Update a copy process with lots of small files; the CircularProgressBar will lag a lot!

What happens if you set the animation duration to zero?

For now,
circularProgressBar.AnimationSpeed = 1;
Should solve your problem.
However, I am aware for some time that the underlying WinFormAnimation library needs an overhaul.


Fixes for dataviews using overviews

This fixes https://github.com/CartoDB/cartodb/issues/10783

@rochoa, can you review this?
There are two different fixes here (corresponding to the two commits):
The first one corrects a defect of the dataview classes for overviews which had an spurious parameters. I'm not exactly sure in which cases a problem was apparent to the user, but probably it caused incorrec results to filtered widgets applied to tables with overviews.
The second fixes this specific problem: CartoDB/cartodb#10783
/cc @rafatower


the loc of mplayer project drop from 1.0-rc2 to 1.3

links:

1.0-rc2: https://codeload.github.com/peeboo/mplayer/zip/refs/tags/MPlayer1.0-rc2
1.3: https://codeload.github.com/peeboo/mplayer/zip/refs/heads/branches/1.3


Fixes issue #10258

On CUDA versions previous to 8.0, only shared variables could be declared as static in the device code.

Can one of the admins verify this patch?

Jenkins, test this please.

Jenkins, test this please

Jenkins, test this please

https://ci.tensorflow.org/job/tensorflow-pull-requests-multijob/5266/
Current failure seems to be a flake.
merging.


Add thumbprint, or some other object containing the certificate's thumbprint to New-DbaComputerCertificate.

Feature Request
Add thumbprint, or some other object containing the certificate's thumbprint to New-DbaComputerCertificate's output
Problem to solve
New-DbaComputerCertificate doesn't emit the thumbprint, so we can't deterministicly find the certificate we just created to pass into Set-DbaNetworkCertificate.

Just ignore me, I'm going to sit here and drink more coffee. This works as it should. Excellent work y'all!


Issue #61 - overhauling the parser for better error messages.

The big change here is we move from Attoparsec to Megaparsec. The
latter is nearly as fast in general, but has much better error
handling support.
The code changes aren't actually very dramatic, as the two libraries
are fundamentally quite similar. Summarising what I've actually needed to alter:

The double-parse we were doing before - once to parse, once to
capture line breaks - is no longer necessary.
There are many simple function name changes. For example
skipSpaces becomes spaces, notChar becomes anySingleBut.
Instead of storing Position as an Int, it's now an alias for
megaparsec's SourcePos, which has richer position information.
The existing test data is almost exactly the same, except where the
line/column numbers are more accurate than before.
I've labelled most of the parsers we use, for better error messages.
I've added some syntax error test cases. Obviously you can't
predict every mistake a user can make, but what's there makes it clear
that the error messages are much more useful.

I've also tried to move towards this convention:

https://hackage.haskell.org/package/megaparsec-7.0.5/docs/Text-Megaparsec-Char-Lexer.html
We propose a convention where every lexeme parser assumes no spaces
before the lexeme and consumes all spaces after the lexeme;

I've found it generally makes it easier to resolve ambiguity, and know
who is responsible for consuming what whitespace. I haven't used the
lexeme code in that module though. I think it would probably be a good
idea, but wanted to keep this commit as uncontroversial as I could. :-)

We propose a convention where every lexeme parser assumes no spaces
before the lexeme and consumes all spaces after the lexeme
if we use this convention, how can we analyse fields with spaces in between?
{field1  fiedl2}

parser of field1 will consume all empty space and sepBy will continue reading at field2} and fail.


[Doc] Modules Update

This pull request has automatically created by PinpointAutoBot
Updated Item : [mariadb-java-client] has been updated
*Updated item list may differ if this PR is updated*

Codecov Report

Merging #7056 into master will decrease coverage by 0.00%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #7056      +/-   ##
==========================================
- Coverage   38.90%   38.89%   -0.01%
==========================================
  Files        3274     3274
  Lines       95093    95093
  Branches    11897    11897
==========================================
- Hits        36992    36985       -7
- Misses      55141    55149       +8
+ Partials     2960     2959       -1

Impacted Files
Coverage Δ

...p/pinpoint/rpc/client/WriteFailFutureListener.java
62.50% <0.00%> (-18.75%)
:arrow_down:

...vercorp/pinpoint/rpc/packet/ClientClosePacket.java
55.55% <0.00%> (-11.12%)
:arrow_down:

...nitor/DefaultDataSourceMonitorRegistryService.java
77.77% <0.00%> (-11.12%)
:arrow_down:

...pc/server/LoggingServerMessageListenerFactory.java
90.00% <0.00%> (-10.00%)
:arrow_down:

...rp/pinpoint/web/mapper/stat/AgentStatMapperV2.java
59.52% <0.00%> (-9.53%)
:arrow_down:

...orp/pinpoint/rpc/server/DefaultPinpointServer.java
79.22% <0.00%> (-1.06%)
:arrow_down:

...pinpoint/profiler/monitor/DeadlockMonitorTask.java
19.76% <0.00%> (+1.16%)
:arrow_up:

...ercorp/pinpoint/profiler/util/ThreadDumpUtils.java
62.12% <0.00%> (+1.51%)
:arrow_up:

...java/com/navercorp/pinpoint/rpc/DefaultFuture.java
74.16% <0.00%> (+1.66%)
:arrow_up:

...m/navercorp/pinpoint/web/util/ThreadDumpUtils.java
35.23% <0.00%> (+1.90%)
:arrow_up:

... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 357248a...a45f28b. Read the comment docs.


[RULE] - Allow placing of portal blocks via inventory

Proposed rule name
placeablePortalBlocks
Describe rule
Allows nether portal, end gateway portal, and end portal blocks to be placeable from a players inventory in creative.
Not entirely sure on the amount of work this will be but likely to be large in scope. The player should be able to see the blocks in the creative inventory, and from the /give command. This might require adding the blocks as items to the games registry, and exactly how that can be toggleable is uncertain.
Options for rule
true or false (default)
Requested by: @DH12043

Also should consider pick selecting the blocks into the inventory

Consider all illegal blocks

List  of some technical blocks for consideration.

Another list here.

Modified the name and description of this issue following the discussions on discord and the POC working.
Things to fix for POC:

[ ] proper feedback to command input
[ ] ensure blocks always face the player
[ ] command is displayed in commands list once rule is enabled

No longer active since I lost the branch with the POC work. Will pick it up again when time permits.


Blurred text on integrated terminal

VSCode Version: Code 1.17.1 (1e9d36539b0ae51ac09b9d4673ebea4e447e5353, 2017-10-10T14:21:18.489Z)
OS Version: Linux x64 4.10.0-37-generic
Distribution: Ubuntu 17.04 (Zesty)
Extensions: none

Steps to Reproduce:

Install VSCode from .deb package
Open Integrated terminal.

Images:
v1.16 (not-blurred):

v1.17(blurred):

Details:
On versions 1.16.x and 1.17.0 the integrated terminal works perfectly fine (except for some resize functionalities). After the 1.17.1 update, I noticed the blurred output. Everything else seems to work as expected. I tried reinstalling but the issue persisted. I tried downgrading to 1.16, and the difference persisted. Also, I used two new VMs, 16.04 and 17.04 respectively. In the 16.04 version, the output is not blurred but it is on 17.04. Thus, this issue only affects 17.04.

I have blurred text on Ubuntu after update from 1.17.0 to 1.17.1

I'm having the same issue on Windows 10.

Duplicate https://github.com/Microsoft/vscode/issues/35991


Update for encypting user password. - [closed]

In GitLab by @wknet123 on Sep 8, 2017, 07:36
Merges dev-encrypt-password -> dev

In GitLab by @wknet123 on Nov 13, 2019, 09:56
closed


Should we rename namespaces to lowercase?

In the interest of breaking things now rather than later it seems people don't like my style ;) basically types being UpperCamelCase and values and functions being lowerCamelCase is ok by most people but that fact that namespaces are also upper camel case bugs a lot of people. So we should probably change it.
The one breach in this style is the nested type in Metafunction::type. The problem is that TMP convention is to use lower case type but it is a type so according to this style guide it should be upper case. I think we can get away with this breach in our style guide and use lower case though because the user should never see this, its just library internals.

As I wrote a lot of ruby code, I could live with your style, but for consistency with existing C++ code, I would prefer the usual C++ style (all lower, underscores, template parameters in camle case etc.)

In addition, I would suggest to put all headers under one "kvasir" directory just to reduced the risk that a path or file name clashes with other parts of a project. Using lower case file and path names would be more consistent too.


Error

I am running below,
#include "MAX30100.h"
#include <Wire.h>
MAX30100* pulseOxymeter;
void setup() {
Wire.begin();
Serial.begin(115200);
Serial.println("Pulse oxymeter test!");
//pulseOxymeter = new MAX30100( DEFAULT_OPERATING_MODE, DEFAULT_SAMPLING_RATE, DEFAULT_LED_PULSE_WIDTH, DEFAULT_IR_LED_CURRENT, true, true );
pulseOxymeter = new MAX30100();
pinMode(2, OUTPUT);
}
void loop() {
//You have to call update with frequency at least 37Hz. But the closer you call it to 100Hz the better, the filter will work.
pulseoxymeter_t result = pulseOxymeter->update();
if( result.pulseDetected == true )
{
Serial.println("BEAT");
Serial.print( "BPM: " );
Serial.print( result.heartBPM );
Serial.print( " | " );

Serial.print( "SaO2: " );
Serial.print( result.SaO2 );
Serial.println( "%" );

}
}
getting error
C:\Users\sim_t\AppData\Local\Temp\arduino_modified_sketch_125959\Example8_SPO2.ino: In function 'void loop()':
Example8_SPO2:19:3: error: 'pulseoxymeter_t' was not declared in this scope
pulseoxymeter_t result = pulseOxymeter->update();
^
Example8_SPO2:19:19: error: expected ';' before 'result'
pulseoxymeter_t result = pulseOxymeter->update();
^
Example8_SPO2:22:7: error: 'result' was not declared in this scope
if( result.pulseDetected == true )
^
Multiple libraries were found for "MAX30100.h"
Used: C:\Users\sim_t\Documents\Arduino\libraries\MAX30100
Not used: C:\Users\sim_t\Documents\Arduino\libraries\MAX30100-master
Not used: C:\Users\sim_t\Documents\Arduino\libraries\MAX30100lib
Not used: C:\Users\sim_t\Documents\Arduino\libraries\MAX30100_milan
exit status 1
'pulseoxymeter_t' was not declared in this scope
any one can help to fix?

Hi could you fix it?

Hello.
Quickly glancing it seems you have 3 pulseoxymeter libraries installed and
they inconveniently have the same name. The linker is not able to figure
out which one to use.
So either rename the libraries or delete the ones you don't use and it
should work.
On Sun, 12 Sep 2021, 05:50 ProgrammerChii, @.***> wrote:

Hi could you fix it?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/xcoder123/MAX30100/issues/9#issuecomment-917551771,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABXIUZL2HXY2EUH3SZVROCLUBQWQZANCNFSM436L7NUA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.


Unprivileged LXD containers with large filesets takes hours to start

Description
LXD driver does not honor LXD idmap values which causes full disk remapping on each start. While it is not a problem on a small fileset, it causes really huge time to start a container with a large fileset.
The problem lies within two values in the container's config, which are
config.volatile.idmap.next
config.volatile.last_state.idmap
If one of these is empty, but the container has an idmap, it causes LXD to run a full remap of all underlying disks. On a large fileset (millions of files) it is a costly operation and might take hours to complete.
To Reproduce
Steps to reproduce the behavior.

Create a regular container
Add LOTS of files into it
Stop the container
Try to start it again

Expected behavior
If those values are maintained properly, container starts immediately.
Details

Affected Component: VMM
Hypervisor: LXD
Version: 5.8.4

Suggestion
I think that LXD driver should save the successful config somewhere and reuse it's values to rebuild newer deployments.

Hi @divanikus could you test https://github.com/OpenNebula/one/pull/3892 ?

Hello @dann1. Unfortunately I cannot test it right now, because I'm already using my own patch in our production. Regarding the issue, you should really store and restore only config.volatile.last_state.idmap, because config.volatile.idmap.next is calculated and maintaned by lxd daemon by itself.
My patch against the current master is looking like that:
diff --git a/src/vmm_mad/remotes/lib/lxd/container.rb b/src/vmm_mad/remotes/lib/lxd/container.rb
index 526ef4b7b..10117309b 100644
--- a/src/vmm_mad/remotes/lib/lxd/container.rb
+++ b/src/vmm_mad/remotes/lib/lxd/container.rb
@@ -486,6 +486,8 @@ class Container

         @lxc = @client.get("#{CONTAINERS}/#{name}")['metadata']

+        @one.from_lxc(@lxc)
+
         status
     end

diff --git a/src/vmm_mad/remotes/lib/lxd/opennebula_vm.rb b/src/vmm_mad/remotes/lib/lxd/opennebula_vm.rb
index 47df5c64b..d94a32857 100644
--- a/src/vmm_mad/remotes/lib/lxd/opennebula_vm.rb
+++ b/src/vmm_mad/remotes/lib/lxd/opennebula_vm.rb
@@ -65,6 +65,15 @@ class OpenNebulaVM
         sysds_id = @xml['//HISTORY_RECORDS/HISTORY/DS_ID']
         @sysds_path = "#{@lxdrc[:datastore_location]}/#{sysds_id}"

+        @prev = if File.exist?("#{@sysds_path}/#{@vm_id}/previous")
+            begin
+                YAML.load(File.read("#{@sysds_path}/#{@vm_id}/previous"))
+            rescue => e
+            end
+        else
+            nil
+        end
+
         return if wild?

         # Sets the DISK ID of the root filesystem
@@ -104,6 +113,14 @@ class OpenNebulaVM
         lxc
     end

+    def from_lxc(lxc)
+        file = "#{@sysds_path}/#{@vm_id}/previous"
+        begin
+            File.write(file, lxc.to_yaml)
+        rescue => e
+        end
+    end
+
     #---------------------------------------------------------------------------
     # Container Attribute Mapping
     #---------------------------------------------------------------------------
@@ -354,6 +371,10 @@ class OpenNebulaVM
             end
         end

+        if<EMAIL_ADDRESS>&& @prev.key?('config') && @prev['config'].key?('volatile.last_state.idmap')
+            raw_data['volatile.last_state.idmap'] = @prev['config']['volatile.last_state.idmap']
+        end
+
         hash.merge!(raw_data) unless raw_data.empty?
     end

--

I'm just saving the whole lxc definition to a previous file on each state change and reusing it on container start to get and set the config.volatile.last_state.idmap.


[RFC] Make Dwifft an enum, not a struct

It looks like Dwifft is only used as a namespace:
https://github.com/jflinter/Dwifft/blob/master/Dwifft/Dwifft.swift#L79
The problem with a struct is that it can be instantiated, Dwifft(). This isn't a problem now, since there are no instance members, but it is a bit awkward.
Changing this to enum Dwifft (with 0 cases) provides the same effect of namespacing, except you cannot instantiate an enum. 😄
See also:

https://www.natashatherobot.com/swift-enum-no-cases/
http://ericasadun.com/2016/07/18/dear-erica-no-case-enums/

That makes a lot of sense, thanks for the proposal! I'd happily accept a PR
making this change.
On Thu, Jul 27, 2017 at 12:59 PM Jesse Squires<EMAIL_ADDRESS>wrote:

It looks like Dwifft is only used as a namespace:
https://github.com/jflinter/Dwifft/blob/master/Dwifft/Dwifft.swift#L79
The problem with a struct is that it can be instantiated, Dwifft(). This
isn't a problem now, since there are no instance members, but it is a bit
awkward.
Changing this to enum Dwifft (with 0 cases) provides the same effect of
namespacing, except you cannot instantiate an enum. 😄
See also:

https://www.natashatherobot.com/swift-enum-no-cases/
http://ericasadun.com/2016/07/18/dear-erica-no-case-enums/

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/jflinter/Dwifft/issues/67, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAtM5wHhny9apyBjl1gEfyERmsbpZVGWks5sSMH5gaJpZM4OlkR2
.


Trying to learn how to code

defining strings in Python
all of the following are equivalent
my_string = 'Hello'
print(my_string)
my_string = "Hello"
print(my_string)
my_string = '''Hello'''
print(my_string)
triple quotes string can extend multiple lines
my_string = """Hello, welcome to
the world of Python"""
print(my_string)

#Accessing string characters in Python
str = 'programiz'
print('str = ', str)
#first character
print('str[0] = ', str[0])
#last character
print('str[-1] = ', str[-1])
#slicing 2nd to 5th character
print('str[1:5] = ', str[1:5])
#slicing 6th to 2nd last character
print('str[5:-2] = ', str[5:-2])


idp_metadata_parser not parsing correct idp_cert from metadata

The method certificate_base64 method here: https://github.com/onelogin/ruby-saml/blob/master/lib/onelogin/ruby-saml/idp_metadata_parser.rb#L186-L203, which populates the idp_cert of the settings is not parsing the correct x509 cert. This is causing the saml_response object to be invalid. So when I run saml_response.is_valid? I get ["Invalid Signature on SAML Response"]. There are many x509s (appx 20) in my customer's xml who is using ADFS. Has anyone else come across this issue? Are any updates being made to improve this parsing? Thanks in advance! :)

Right now the parser extracts the first x509certificate found.

Referencing new comment I made here: https://github.com/onelogin/ruby-saml/issues/329

Ah I see. Ok I guess I'll have to monkey patch a workaround. Thanks.


New last price source

https://runescape.wiki/w/Module:LastPrices/data
.mw-body-content .mw-parser-output .mw-code

Solution? 7a79b78

Funny enough they added it back to their data source, but I'm going to keep my method since I believe it should work and won't change unless I want it to.

Found a happy medium 23bb03b


various updates for CM tools for Windows and better debugging

@anandhu-eng - I updated Windows tools with the newest wget.exe as you suggested:

https://github.com/flexaihq/cm4mlops/blob/main/script/get-sys-utils-min/_cm.json#L11
https://zenodo.org/records/13868077
Can you please check if it works on your side for this ticket, please: https://github.com/mlcommons/cm4mlops/issues/272

I turned on test on Windows to make sure that the core CM functionality is not broken.

@gfursin Windows test won't work with MLPerf as the default workflow is doing llvm installation for loadgen build which works only with user interaction.


wip/ji/dbif-parsers

With parsers logging so much information in NCWrapper for each request absolutely kills it's performance, because it tries to write it everything to log widget. (with proper logging we could set it to some TRACE log level, but for now I think our best option is to simply remove it).

@altran01 I agree. detailed_debug_info_ is now set to false by default. In the future we might add a possibility to turn it on/off from options dialog.

Eh, even with above fix, performance drop for pyc parser is abysmal at best - .pyc file that before took < 10seconds to parse now it took around 40minutes. From brief analysis it looks like the main culprit is huge amount of small transactions in sqlite which can't handle it in timely manner (CPU usage for both Python and C++ processes was negligible, while Python constantly used disk I/O on sqlite db and journal files).


Added layout-aware check

Updated code so "application" layout isn't hard-coded

c0dcc1d454f497e281606f78664384e48603c031


Invalid Svelte Usage Docs

When using this lib with Svelte as described with the readme, results in Slide component is not defined error.
This is how It worked for me, similar to app/component.svelte
<Carousel current="3">
	<Slide>

1

	</Slide>
	<Slide>

2

	</Slide>
	<Slide>

3

	</Slide>
	<Slide>

4

	</Slide>
	<Slide>

5

	</Slide>
	<Slide>

6

	</Slide>
</Carousel>

I ran into this myself - I had misunderstood how the 'svelte' attribute works in package.json. I'll update the docs accordingly.


Add autocomplete for /giveachievement

self-explanatory (look at my other pull request)
also added a thing in the giveachievement function where previously, thanksforplaying could not be removed with commands, but now it can

why does this pr also has the other one in it :staring_cat:

why does this pr also has the other one in it :staring_cat:

uhh I forked it from the other one
I can do a new pull request without those if you want

can this be closed

can this be closed

why
I just updated

or im gonna hit you with a comically big spoon

you should do it anyway I think

should ach_id validity checking be removed or should it be kept as a failsafe

not only as failsafe, autocompletes are just suggestions

frinkifail stop sabotaging

im not :<

any issues?

okay but why is this straight up not working :staring_cat:

idk

what

like the autocomplete is just straight up empty

idk man sounds like a skill issue

idk man sounds like a skill issue

yes

idk man sounds like a skill issue

yes

that was so helpful of you

if you cat bot if you cat bot if you cat bot if you cat bot

after fixing the error logs i now finally know why

ig .title works

god dangit


The pattern question for real sense camera

Dear Sir/Madam,
Currently, I have a project regarding the simulating of the Real Sense D415 in Unreal Engine.
The pattern influences the accuracy of the stereo camera image output and reconstruction output.
Could Real Sense open&provide its pattern for usage?
Thanks for any suggestion&information:-)
Best regards,
Mingchuan

If you are asking for the camera's algorithms, these are not likely to be made available, as algorithms tend to be kept private by Intel as part of their intellectual property (IP) and not made open-source.
However, the RealSense SDK does provide a means via its software_device sample program of simulating camera hardware in software.
https://github.com/IntelRealSense/librealsense/tree/master/examples/software-device

Thanks for your quick answer, I will check the link. However, I am not asking for the algorithm. I would like to ask whether I can get the pattern of the projector:-) . I saw in the image from infrared camera that it is kind of repetitive pattern with dots.

Thank you very much for your clarification.  The dot pattern provided by the projector is a semi-random pattern of dots.
Intel have an excellent illustrated camera tuning guide which contains recommendations about the dot pattern.  This information is provided in an extract below.
https://realsense.intel.com/wp-content/uploads/sites/63/BKM-For-Tuning-D435-and-D415-Cameras-Webinar_Rev3.pdf?language=en_US

PROJECTION PATTERNS

I’m not sure the pattern is stored anywhere in the system, but you can get a sense of it by taking snapshot of blank wall in ir. Unlike classic structured light, the pattern in theory should not matter, in the sense that the camera does not need to know what is the pattern. The D43x for instance have a different type of pattern from the d41x.
Of course different patterns do cause slight differences in behavior but so do different scenes.
Sent with GitHawk


Develop

Description
[Describe what this change achieves]
Issues Resolved
[List any existing issues this PR resolves]
Contribution Check List

[ ] All tests pass.
[ ] New functionality includes testing.
[ ] New functionality has been documented in the README if applicable

          1 Error

:no_entry_sign:
Please include a CHANGELOG entry.

          1 Warning

:warning:
This Pull Request is probably missing tests.

  Generated by :no_entry_sign: Danger


Does showdown has Sanitizer like Pagedown has?

Pagedown fork of showdown has sanitizer to sanitize xss or malicious tags - https://github.com/StackExchange/pagedown/blob/master/Markdown.Sanitizer.js
Does showdown has any?
It is crucial for user generated markdown content.

Showdown tries to convert the input text as closely as possible, without any concerns for XSS attacks or malicious intent. So, the basic rules are:

removing HTML entities from markdown does not prevent XSS. Markdown syntax can generate XSS attacks.
XSS filtering should be done AFTER Showdown has processed any input, not before or during. If you filter before, it’ll break some of Markdown’s features and will leave security holes.
perform the necessary filtering server-side, not client side. XSS filtering libraries are useful but shouldn't be used blindly.

https://github.com/showdownjs/showdown/wiki/Markdown's-XSS-Vulnerability-(and-how-to-mitigate-it)

Showdown tries to convert the input text as closely as possible, without any concerns for XSS attacks or malicious intent. So, the basic rules are:

removing HTML entities from markdown does not prevent XSS. Markdown syntax can generate XSS attacks.
XSS filtering should be done AFTER Showdown has processed any input, not before or during. If you filter before, it’ll break some of Markdown’s features and will leave security holes.
perform the necessary filtering server-side, not client side. XSS filtering libraries are useful but shouldn't be used blindly.

https://github.com/showdownjs/showdown/wiki/Markdown's-XSS-Vulnerability-(and-how-to-mitigate-it)

But it doesn't say "how to mitigate it". It just tells you what are some of the problems. When you finish reading it, you are still with the same problem with no solution but "you have to mitigate it somehow, but not this way, neither this way".


Update slpjs version.

Update examples. Add NFT example. Update balances calls.

:tada: This PR is included in version 1.1.13 :tada:
The release is available on npm package (@latest dist-tag)
Your semantic-release bot :package::rocket:


Chinese error reporting bug in string

The test found that as long as Chinese or Japanese appeared in the string, there would be parsing problems.
const amauiZip = new AmauiZip(JSON.stringify({"_id":"page_2qewpv","name":"这是一句中文"}))
// Encoded
const response = amauiZip.encoded
console.log(response)
console.log(JSON.parse(AmauiZip.decode(response.value).value))

Will have a look, it's a fresh library, haven't tested with characters outside ASCII (utf-8) I think. Thanks

@bbfpl fixed, new version published now.
https://github.com/amaui-org/amaui-zip/commit/0f023e1e7d3315a393eb9d737340cee24f64dca9

Fixed in lz77 decoding text back to UTF-16
Fixed in huffman-code last character having a bug with padded 0s for the } bracket in JSON

FYI, you don't have to stringify, serialization happens automatically.
You can simply:
    new AmauiZip({"_id":"page_2qewpv","name":"这是一句中文"})

And it on decode, it will parse it back to the JSON object.


zlib now required in all system definitions?

I believe zlib was added to https://github.com/LLNL/benchpark/tree/develop/systems/genericx86/externals/base/00-packages.yaml to make Laghos work.

[ ] What was the reason?  Is this because we need the handle?
[ ] Is the correct solution to put it in externals for each system?
[ ] If yes, does every system definition have to have it?
[ ] If yes, is there a better place to put it?

I believe zlib was added to https://github.com/LLNL/benchpark/tree/develop/systems/genericx86/externals/base/00-packages.yaml to make Laghos work.
* [ ]  What was the reason?  Is this because we need the handle?

This was to avoid spack from defaulting to building zlib-ng, which does not work for non-intel compilers

* [ ]  Is the correct solution to put it in externals for each system?

Possibly. But that will assume every system has to have zlib installed

* [ ]  If yes, does every system definition have to have it?
* [ ]  If yes, is there a better place to put it?

The Spack team may be able to clarify this better

After discussion 11/26, we should remove zlib from packages on all systems except for lassen. @rfhaque.

@rfhaque @scheibelp is this now resolved?


⚠ Bofrost.es has degraded performance

In 86872c2, Bofrost.es (https://www.bofrost.es) experienced degraded performance:

HTTP code: 200
Response time: 1591 ms

Resolved: Bofrost.es performance has improved in 88c17e0 after 14 minutes.


Remove a literature reference #19

When a pathway element is deleted, also remove the literature reference link to the pathway element.

Hi Jonathan. I had a quick look at the commit. I am missing the step, where you check if the literature reference is referenced from any other object. It is possible that the same paper might be linked from different pathway elements. If you then simply delete the reference, then we will have the issue that something references to a pubref that has been deleted.

Hi Tina, yes you're right. I missed that case.

Looks like the function to check for existing references was already there?

https://github.com/PathVisio/pathvisio/pull/20/commits/af5866b7049998ae392cb810ad0329562cee1b29
getBiopax().hasReferences(node) will check the temporary list of the pathway's dataObjects.
The datanode that is being removed doesn't appear in the temporary list.
So if hasReferences return false, no other datanode use this reference, then the literature reference can be delete.


Upgrading from 1.61 to 1.6.3 causes error: refuse to serve WS client for illegal ws path: ▒▒I

Hi,
I've been successfully using ttyd version 1.6.1-079aa68 as follows:
$ ttyd --port=8004 --base-path=/nbhatia1/project2/ttyd /bin/bash
[2021/01/18 02:13:21:4000] N: ttyd 1.6.1-079aa68 (libwebsockets 4.0.19)
[2021/01/18 02:13:21:4000] N: tty configuration:
[2021/01/18 02:13:21:4000] N:   start command: /bin/bash
[2021/01/18 02:13:21:4000] N:   close signal: SIGHUP (1)
[2021/01/18 02:13:21:4000] N:   terminal type: xterm-256color
[2021/01/18 02:13:21:4000] N: endpoints:
[2021/01/18 02:13:21:4000] N:   base-path: /nbhatia1/project2/ttyd
[2021/01/18 02:13:21:4000] N:   index    : /nbhatia1/project2/ttyd/
[2021/01/18 02:13:21:4000] N:   token    : /nbhatia1/project2/ttyd/token
[2021/01/18 02:13:21:4000] N:   websocket: /nbhatia1/project2/ttyd/ws
[2021/01/18 02:13:21:4044] N:  Using foreign event loop...
[2021/01/18 02:13:21:4044] N:  Listening on port: 8004
[2021/01/18 02:13:23:2686] N: HTTP /nbhatia1/project2/ttyd/token - <IP_ADDRESS>
[2021/01/18 02:13:23:3432] N: WS   /nbhatia1/project2/ttyd/ws - <IP_ADDRESS>, clients: 1
[2021/01/18 02:13:23:3672] N: started process, pid: 156

After upgrading to ttyd version 1.6.3-3b174da the same command fails to accept connections:
$ ttyd --port=8004 --base-path=/nbhatia1/project2/ttyd /bin/bash
[2021/01/18 02:17:33:1304] N: ttyd 1.6.3-3b174da (libwebsockets 4.1.6-)
[2021/01/18 02:17:33:1305] N: tty configuration:
[2021/01/18 02:17:33:1305] N:   start command: /bin/bash
[2021/01/18 02:17:33:1305] N:   close signal: SIGHUP (1)
[2021/01/18 02:17:33:1305] N:   terminal type: xterm-256color
[2021/01/18 02:17:33:1305] N: endpoints:
[2021/01/18 02:17:33:1305] N:   base-path: /nbhatia1/project2/ttyd
[2021/01/18 02:17:33:1305] N:   index    : /nbhatia1/project2/ttyd/
[2021/01/18 02:17:33:1305] N:   token    : /nbhatia1/project2/ttyd/token
[2021/01/18 02:17:33:1305] N:   websocket: /nbhatia1/project2/ttyd/ws
[2021/01/18 02:17:33:1305] N: LWS: 4.1.6-, loglevel 7
[2021/01/18 02:17:33:1305] N: NET SRV H1 WS IPV6-off
[2021/01/18 02:17:33:1349] N:  Using foreign event loop...
[2021/01/18 02:17:33:1350] N:  Listening on port: 8004
[2021/01/18 02:17:38:0511] N: HTTP /nbhatia1/project2/ttyd/ - <IP_ADDRESS>
[2021/01/18 02:17:38:1701] N: HTTP /nbhatia1/project2/ttyd/token - <IP_ADDRESS>
[2021/01/18 02:17:38:2520] W: refuse to serve WS client for illegal ws path: ▒▒I
[2021/01/18 02:17:38:2520] W: User code denied connection
[2021/01/18 02:17:38:2521] W: refuse to serve WS client for illegal ws path: ▒▒I
[2021/01/18 02:17:38:2521] W: User code denied connection

I'm running in a docker container using a CentOS 7 image. Nothing else is changing other than the ttyd version I run (and I can switch between the two versions and replicate the exact same error every time). Could anyone help me debug what's wrong with the websocket client path?

What if you start ttyd without the --base-path option?

What if you start ttyd without the --base-path option?

Fixed on master (7bd16386d28cf35a9ead3ca48c31c019ebbdafd5)

Fixed on master (7bd16386d28cf35a9ead3ca48c31c019ebbdafd5)


fix: Generate CRUD page - Datasource selected item padding issue

Description
Alignment fix for Generate CRUD page datasource dropdown
Fixes # (issue)
#12025
Type of change

Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

Tested locally

Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes

Test coverage results :test_tube:
:green_circle: Total coverage has increased
// Code coverage diff between base branch:release and head branch: bug/12025-generate-page-dropdown-alignment

Status
File
% Stmts
% Branch
% Funcs
% Lines

:green_circle:
total
55.63 (0.01)
36.86 (0.01)
34.94 (0)
55.93 (0)

:green_circle:
app/client/src/pages/Editor/GeneratePage/components/DataSourceOption.tsx
34.78 (0)
23.53 (1.31)
0 (0)
36.36 (0)

:green_circle:
app/client/src/utils/WorkerUtil.ts
89.76 (0.78)
72.55 (3.92)
100 (0)
93.33 (0.95)

/ok-to-test sha=186eb5c

/ok-to-test sha=186eb5c

/ok-to-test sha=186eb5c

/ok-to-test sha=2651dd3

/ok-to-test sha=e239fd9


extend PageableTask interface and provide a generic PageabaleTaskResult

NestedLoopTask/Operation still use the other Taskresults and need to be
adjusted...

ack

now ack

ack ack


Update httpclient.version to 4.5.13 to address CVE-2020-13956.

As per https://bugzilla.redhat.com/show_bug.cgi?id=1886587, http.client libraries below version 4.5.13 have the vulnerability CVE-2020-13956.
As Karaf 4.2.x rebundles http.client (4.5.6) classes as seen at https://github.com/apache/karaf/blob/karaf-4.2.10/jaas/modules/pom.xml#L180 . This makes it vulnerable and hence our security scans are detecting it as a vulnerable library. Hence updating the httpclient.version to 4.5.13.

Merged and backported to 4.2.x, thanks!

Thanks for the quick merge @skitt :)


Feature: use last matching mapping for linking

Hi!
Thanks for dfm, I've been using to set-up a few GNU/Linux machines and I'm liking it so far.
Rationale
So I've been wanting to include some third-party repos into my profile
(mainly plug-in managers such as vim-plug) and I've been adding them as modules.
Since installing them amounts to simply copying a few files,
I found it useful to only link some files in, and skip the rest.
Using this change, one can do the above like so:
# .dfm.yml for my profile
modules:
  - repo: my_repo
  mappings:
    # my catch-all pattern
    - match: '.*'
      skip: true
   # only my_file will be linked
    - match: my_file
       target_dit: ~/.my_config_dir

I'm not convinced by the picking of the last one instead of the first,
but it matches previous behaviour more closely when there was no skip or link_as_dir option in the module.
Before

generate_link would to skip all mappingsif a singleskip`option was found for the module,
it would return immediately if link_as_dir was found,
otherwise, it would pick the last matching mapping.

Now

generate_link always picks the last of matching mappings.

Hi @piptouque I'd like to see some new tests for this behaviour.
The code is a bit obtuse and I have some potential style improvements but first I'd like to see tests so we can make sure my suggestions don't break your intended new behaviour.

@piptouque this PR is quickly spiraling into a rewrite. I have no intention of merging the use of GitPython. I intentionally didn't use it because keeping a low dependency count is important to me as a Python application distributor it means less things break and mess with user's system python.

Hi there, sorry for the late reply, I hadn't realised my more recent changes would be reflected in this PR. I understand that you want to keep it simple. Thanks for the feedback, have a nice day!


Add deb_loader ruleset

Split from #94, this PR only contains the (slightly updated) deb_loader ruleset.
This will not work out of the box as-is, due to the fact that the binary for downloading files has to be served via HTTP (see deb_loader/deb_loader.bzl). It'll currently look on localhost instead of a publicly available location.
It is very likely that 2 additional parameters (ignored by the deb_loader tool but necessary to auto-update deb_packages rules in a safe manner) will be added later: pgp_keyid and pgp_keyservers (or maybe pgp_keyurls?) - these will contain the keyid and a list of keyservers that might host the defined key which signed the Releases file on the mirror(s). These can't really be hardcoded to support internal mirrors and people that might not want to publish their internal signing keys.
I hope this makes it easier to reason about the changes, the update_workspace helper tool will be the next thing to review/add in case you are willing to move forward with this. I can also add a container that will use the rules outlined here and/or convert currently existing rules to this new style in case you want to see how it looks like, though that can be deferred for later.
Changes between this and #94:

Renamed architecture to arch to be more in line with existing ruleset
smaller scope of PR (only rule, no other changes)
documented adding packages manually and with the update_workspace tool
Mirror URLs now have to point to the root folder of the repository, /debian or /ubuntu will not be automatically assumed or added (this solves the same issue as #93 and makes it easier to use the snapshots mirror in deb_packages rules for example)

I didn't split the attribute (packages and packages_sha256) since the only mention of Fileset I could find is on the bottom of https://www.bazel.build/support.html with the comment we do not intend to support them in Bazel, ever
I also didn't build upon package_manager since the whole code for deb_loader is just a few dozen lines and I don't need most of the functionality in package_manager. This way it is also possible to have both rule sets side by side, in case you want to make both available.

Thanks a lot for sending this PR! I'll take a close look soon.
I'm a big fan of the flow you proposed for automatic updating of deb versions via a WORKSPACE updater tool.
One thing that would help is if you could outline the behavioral differences between this and the existing package_manager solution. It looks mostly the same now, with a few minor tweaks. Would it be possible to iterate on the existing ones instead of starting a new set of rules?

The major differences are:

Package hashes and paths are declared in the WORKSPACE file, not fetched and parsed from a remote server at build(!) time
Package sources (mirror servers) and package names are combined in the same, single rule (the package_manager solution splits them, because it first needs to fetch the Packages file, parse it to the Packages.json file and only then it knows where to get the actual deb files from)
The key in the packages/packages_sha256 dict doesn't actually have to be a valid package name (unlike the package_manager ruleset), but it makes it much easier to automatically update them using the update_workspace tool if they contain the package name (and optionally the version).
Any Debian style mirror is supported, not only the snapshots.debian.org one. This includes internal/local mirrors as well as other distributions like Ubuntu.
If you don't use the updater tool, you can even just use this as an abstraction for http_file() rules (though a rather poor one - I'd much prefer fetching files using Bazel than urllib.urlretrieve()).

Some minor differences:

downloaded deb files are renamed internally after their sha256 hash, not their base64 encoded package name
You need to enter the full URL to the root of the Debian style mirror, similar to a sources.list file - not http://deb.debian.org but http://deb.debian.org/debian/ (the former used to be possible before the current rule set was limited to the snapshots mirror)
It can properly handle mirrors that serve multiple versions of the same package with the same name (only the tuple (package_name, version, architecture) is unique on a mirror)

Similarities:

This ruleset is intentionally similar to the existing one in how rules look like and are used in BUILD files to make it easy for anyone to switch to these rules. Also I really liked the design of having a dictionary of deb files with their exact package names (and maybe even versions).

Initially I would just have written the updater tool to update http_file() rules, but these would have a HUGE amount of duplicated lines over time compared to this approach and they would have been much harder to reason about.
I consider the "download and parse a Packages.gz file at build time" a design flaw of the existing ruleset, so iterating on it would just lead to the removal of the dpkg_source() rules and adding mirrors, file paths and hashes to dpkg_list() instead. At this point you arrive at the implementation in this PR. The additional  information (arch, distro, distro_type) is not used by the package downloader, but is useful metadata that helps the updater tool figure out which exact package is referenced and (in the future) how to verify the root signature of the whole hosted package repository.

Thanks for clarifying! Some comments inline:

The major differences are:
Package hashes and paths are declared in the WORKSPACE file, not fetched and parsed from a remote server at build(!) time
By paths, are you referring to URLs? Wouldn't using something like http_archive also achieve that goal?

Package sources (mirror servers) and package names are combined in the same, single rule (the package_manager solution splits them, because it first needs to fetch the Packages file, parse it to the Packages.json file and only then it knows where to get the actual deb files from)

I think we could combine these into a single rule with the existing design if we wanted, we just did it this way to be more similar to how debian packages are typically fetched (admins add a set of package lists, apt-get fetches the package from one of the installed lists).
I do like the idea of having fewer rules though :)

The key in the packages/packages_sha256 dict doesn't actually have to be a valid package name (unlike the package_manager ruleset), but it makes it much easier to automatically update them using the update_workspace tool if they contain the package name (and optionally the version).

This definitely seems like a win. Bazel's naming rules don't match up well with debian packages.

Any Debian style mirror is supported, not only the snapshots.debian.org one. This includes internal/local mirrors as well as other distributions like Ubuntu.

I'm having a hard time seeing how this will work in practice in combination with bazel/WORKSPACE's assumption about things being mostly hermetic. Making it easy to use any debian-style mirror makes it likely that packages and their SHAs will change out from under you, which makes bazel unhappy.
How do you see this working in practice?
Standard debian mirrors change all the time, and packages appear/disappear. Internally at Google we solve these problems by checking everything we depend on into a single source control system. If you don't want to rely on the snapshot-style mirrors, I think the only other way to satisfy Bazel/WORKSPACE's constraints around reproducibility would be to commit the .debs you need (or at least package list files that reference debs in locations that won't change).

If you don't use the updater tool, you can even just use this as an abstraction for http_file() rules (though a rather poor one - I'd much prefer fetching files using Bazel than urllib.urlretrieve()).

By paths, are you referring to URLs? Wouldn't using something like http_archive also achieve that goal?

By paths I refer to the part of the URL after the mirror portion ( +  = <deb_file URL>). http_archive is about downloading something with a BUILD file inside, I think you mean http_file.
Yes, you could use these rules too, but then you'd have a block of mirrors below every single deb file with only the file location changing. There's also the issue of the rule name, which can't map 1:1 with the package name due to name restrictions in Bazel, so it is not really possible to automate this. I guess because of this, the package_manager ruleset was implemented in the first place.

Also, could you elaborate on the issues with fetching and parsing the packages list file at build time?

Aside from using up ressources on public infrastructure, it:

externalizes tracking desired state, hashes and versions to a third party (with no SLA by the way)
does the same work at every build (which - if pinning the hash of the Packages.gz file - has the same result every time)
makes it very hard to reason about which actual package version is used inside your containers (imagine openssl x.y.z being vulnerable - do you actually use it in one of your containers? If snapshot is down, you'll have to dig around in your Bazel cache and hope the deb file is still there)
would make the situation worse if you implement signature checking instead of pinning the Packages.gz hash (since then it could change the referred deb file without you noticing with a valid signature)
is only useful to use Packages.gz pinning on snapshot.debian.org, since on normal mirrors, this file changes even if all deb files you're interested in are still at the same location with the same hash

I think we could combine these into a single rule with the existing design if we wanted

I wonder how, since there are 3 actions (ensuring the downloader exists, fetch and parse the Packages.gz file into Packages.json, fetch packages using information in the Packages.json file) in the existing design. In my proposed design, the fetching and parsing of Packages.gz file(s) is done out of the loop (similar to the design in Debian, where apt-get update and apt-get upgrade are distinct actions and there is no actual single command for update && upgrade

This definitely seems like a win. Bazel's naming rules don't match up well with debian packages.

To be fair, the current ruleset also supports naming the referred deb files exactly like the package name, they just have to enforce it (since the name is the key in the Packages.json file).

How do you see this working in practice?

As I wrote in #94, there are three approaches you can take to make sure that builds of historic commits work as long as possible:

(the only supported approach in package_manager): Use snapshot.debian.org at a certain time stamp as one of your mirrors. As long as it is available, your deb file actually is/was public and the mirror works as advertised, this works.
Run a local/internal mirror that stores multiple versions per package. Update packages from a public source (e.g. deb.debian.org), drop every package you end up using also on your own mirror and refer to it first, before any other mirror. This is also more friendly towards public infrastructure, since you are caching your files locally. It also is probably faster.
Combine both approaches: first enter your local mirror, then (if it fails) the public, up-to-date one (deb.debian.org is load balanced and refers to a MUCH larger infrastructure than snapshot) and if that one also already moved forward, refer to snapshot as the public archive. Once this fails, you still know exactly which file you'd need (full package name and version as well as a sha256 hash) and can start combing backups and caches.

You might also not care that much about being able to build a container from 2 years ago, if you can exactly know what packages were in there to begin with, which currently is a bit hard to do (Which exact openssl version is included in your base container? How long does it take you to find out? More difficult: Without an internet connection and just this repository here?). Currently you just know that some package named e.g. "openjdk-8-jre-headless" was in your container at one point in time. If it turns out to have been malicious because someone at Debian got compromised, it might even get pulled from snapshot.debian.org too.
I don't really understand the comment about hermeticity, as far as I see it, there are no side effects other than a file potentially not being available at multiple locations. Filesystems can fail, servers/mirrors can go down (or no longer serve the same content, which is the same thing in practice) and I'm not sure that Bazel considers this a failure of hermeticity, more of availablity. There are no side effects to downloading a deb file from a mirror. If I had to adapt the existing ruleset to support any mirror (again), this would lead exactly to problems like you described, this is why this PR exists.
This makes it also easier to reason about changes like https://github.com/GoogleCloudPlatform/debian-docker/pull/85 - why is it possible to change the referenced date by nearly three weeks, yet keep the sha256 hash of the Packages file the same? If the file is still the same, the update is not an update at all, if not, there could be an error somewhere and you're depending on unverified deb packages downloaded via http(!). With deb_loader, you'd see exactly if/which packages changed.

Thanks again for taking the time to walk me through this. I think I'm getting close to understanding the fundamental differences :)

Aside from using up ressources on public infrastructure, it:

Everything should be cached by bazel for most builds, right? Unless you do a clean --expunge or change the mirror url, we won't be downloading anything afaik.

externalizes tracking desired state, hashes and versions to a third party (with no SLA by the way)

It's true that the source of truth lives in the third party with this design, but I think that's true for any rules that use a mirror you don't control, right? There's nothing stopping someone that needs a stricter SLA from running their own SNAPSHOT-style mirror, or vendoring the Packages.gz/.debs directly into their source control, though.

does the same work at every build (which - if pinning the hash of the Packages.gz file - has the same result every time)

Maybe I'm missing something - this is by design. Bazel can very effectively cache build results if nothing changes. If we're repeating work every build, we should fix that :)

makes it very hard to reason about which actual package version is used inside your containers (imagine openssl x.y.z being vulnerable - do you actually use it in one of your containers? If snapshot is down, you'll have to dig around in your Bazel cache and hope the deb file is still there)

This is definitely an unfortunate problem with package_manager now, I think we're planning to tackle it by including extra metadata inside the image itself though, similar to /var/lib/dpkg/status. @r2d4 has some ideas here. Bazel has some limitations that prevent us from populating a single file like that, so the metadata will likely need to be sharded across several files - one per layer or one per package.

would make the situation worse if you implement signature checking instead of pinning the Packages.gz hash (since then it could change the referred deb file without you noticing with a valid signature)

Is there any reason we couldn't do both signature and hash pinning?

is only useful to use Packages.gz pinning on snapshot.debian.org, since on normal mirrors, this file changes even if all deb files you're interested in are still at the same location with the same hash

I think I agree here - these rules are currently only useful with mirrors that preserve packages.gz files and .debs.

Run a local/internal mirror that stores multiple versions per package. Update packages from a public source (e.g. deb.debian.org), drop every package you end up using also on your own mirror and refer to it first, before any other mirror. This is also more friendly towards public infrastructure, since you are caching your files locally. It also is probably faster.

I think this is the main sticking point I have with my understanding of your design - I don't see this as an actual solution that fits into bazel's constraints around reproducibility. I think this solution works by outsourcing reproducibility to your local mirror.
I think there's a fourth option from your list that also satisfies all of the contraints without depending on snapshot-style mirrors. You would fetch the packages.gz file, vendor it into your source repository and reference it as an ordinary file. When you want to update it, fetch it again and update the file in source control. You would still be depending on the .debs being hosted by another system, but the SHAs and locations are vendored. This is conceptually the same as checking in a packages.json or maven configuration. If you want to take it further, you can vendor all the .debs as well.
We could definitely add some tooling to make this update process easier, similar to the gazelle and update_workspace ideas you proposed earlier.

I don't really understand the comment about hermeticity, as far as I see it, there are no side effects other than a file potentially not being available at multiple locations. Filesystems can fail, servers/mirrors can go down (or no longer serve the same content, which is the same thing in practice) and I'm not sure that Bazel considers this a failure of hermeticity, more of availablity. There are no side effects to downloading a deb file from a mirror. If I had to adapt the existing ruleset to support any mirror (again), this would lead exactly to problems like you described, this is why this PR exists.

Yeah, hermeticity was the wrong word here. The main point is that these rules need to either result in the same exact output every time they are run or fail.
I think we have some more options on making the existing ruleset a bit more user friendly without scrapping them and rewriting. I'm in favor of making vendoring Packages.gz files easier, and thinking of ways to relax the constraint we have now on pinning the entire packages.gz file.

This makes it also easier to reason about changes like GoogleCloudPlatform/debian-docker#85 - why is it possible to change the referenced date by nearly three weeks, yet keep the sha256 hash of the Packages file the same? If the file is still the same, the update is not an update at all, if not, there could be an error somewhere and you're depending on unverified deb packages downloaded via http(!). With deb_loader, you'd see exactly if/which packages changed.

One other question - I'm not sure what you mean here. That repo doesn't actually use package_manager, it uses normal apt-get today. What Packages file are you referring to as not having changed? The sha of the resulting image, and the versions of the debian packages inside changed after updating the SNAPSHOT variable.

There's nothing stopping someone that needs a stricter SLA from running their own SNAPSHOT-style mirror, or vendoring the Packages.gz/.debs directly into their source control, though.

The "snapshot style mirror" is not a generally supported use case and I am not so sure if there is software out there that supports a layout like this. Vendoring binaries is frowned upon, but would be a generally valid solution for this problem, yes.

Is there any reason we couldn't do both signature and hash pinning?

The signature pinning would be useless in that case (as you already know which file you will receive). Of course you can check them, but in general you should be checking signatures if you get files you haven't encountered yet and hashes of files you have already verified to be ok. Hashing a file is a stronger guarantee that it will be safe (as you have already had it 100% locally before hashing it), verifying a signature of a file that contains hashes of other files is only as good as the trust you put into the person that signed this.

I think this solution works by outsourcing reproducibility to your local mirror.

You mean availability, right? Since files are hashed, the resulting container will always be reproducibly built - as long as the input files are available.

I think there's a fourth option from your list that also satisfies all of the contraints without depending on snapshot-style mirrors. [...]

Yes this would be a valid option with the package_manager ruleset.
BUT: The only information you actually need these files for, is to obtain a trusted record of the sha256 hash and the path of a (package, version) tuple. Why drag these files along (including several megabytes of data that you first parse and then will just discard) and version them, just to get these 2 values per deb file? You can just enter them in the WORKSPACE file and save an extra build step, several MB of data per update in your repository and get more flexibility too (the proposed ruleset can already handle versions, the existing one can't and needs quite some work to support this). It also makes the data sent to evaluate the dpkg_list rule (or its equivalent, deb_packages) shrink from a few MB of JSON to a few lines containing only the information that is actually needed to fetch the files.

One other question - I'm not sure what you mean here. That repo doesn't actually use package_manager, it uses normal apt-get today. What Packages file are you referring to as not having changed? The sha of the resulting image, and the versions of the debian packages inside changed after updating the SNAPSHOT variable.

https://github.com/GoogleCloudPlatform/debian-docker/pull/85/commits/570df208b79d11e674ecc10bccff3fe17ba49813 changed the date of the snapshot mirror. This means you are no longer referencing http://snapshot.debian.org/archive/debian/20170816T214423Z/dists/jessie/main/binary-amd64/Packages.gz (SHA256: 142cceae78a1343e66a0d27f1b142c406243d7940f626972c2c39ef71499ce61) but http://snapshot.debian.org/archive/debian/20170906T160441Z/dists/jessie/main/binary-amd64/Packages.gz (same SHA256 hash). This file hasn't actually changed since July 22nd 2017. This also means not a single package in Debian Jessie has changed over the summer (so much for fast changing files on official mirros... :D).
Since the hashes are the same, the whole PR was good for nothing (other than removing a line in the README in an unrelated change). It did not "switch to the newest snapshot", it did just change the URL where your deb files are fetched from and I'm not so sure if Bazel actually could/did properly detect that it already had these files in its cache - after all the inputs to the downloader tool changed and it has no idea what's going on in that downloader tool.
I really hope that your "builder" containers did NOT change a single bit when you changed that variable. Also at least your Jessie containers also should be bit-identical, otherwise you have at least a reproducibility problem.

Since the hashes are the same, the whole PR was good for nothing (other than removing a line in the README in an unrelated change). It did not "switch to the newest snapshot", it did just change the URL where your deb files are fetched from and I'm not so sure if Bazel actually could/did properly detect that it already had these files in its cache - after all the inputs to the downloader tool changed and it has no idea what's going on in that downloader tool.

Ah, I see what you mean now. Yeah, had anything changed the build would have failed unless we updated the SHA as well. This code isn't actually live yet though, we're still switching over to it from the old, non-reproducible deboostrap system. That PR was a test of some build hooks we're working on internally :)
No images have actually been built or released using those debian rules yet. Hopefully soon though.

Alright, it's been about 3 weeks for this PR now, over 2 weeks without additional comments or questions and I hope you had enough time to discuss this internally. I would really like to move forward either way with this.
@dlorenc, @mattmoor, @r2d4:

Do you want to remove the "create a Package.json file from the Packages.gz" stage from each build or not? If yes, I guess you'll reach a state close to this PR sooner or later and I'd love to bring it up to your standards, if not, I'd like to know why and which advantages you see with that approach that wouldn't be possible with my proposal.

Would you be OK with adding this rule set as an alternative to the existing one into this repository, in case you want to take the "wait and see"/"let the user decide" approach or do you only want to support a single mechanism for defining and retrieving deb files in distroless? I can think of advantages and disadvantages for that approach of course, so I'd love to know your opinion.

Thanks a lot for the great work already, especially on rules_docker and the existing helper tool for distroless. :-)

This ruleset is a superset of the one you are currently using (it can also handle your current use case with the snapshots mirror). It would be easy to test this in the future with the same caveats/guarantees as the current ruleset (if snapshots is down, you have to provide alternative deb sources with my rules and you have to provide alternative Packages.gz and deb sources with the current rules).
If you are interested, I can convert the existing containers and rules to the ones in here, so you can see what I mean (in the end the mirror URL would be "http://snapshot.debian.org/archive/debian/20170821T035341Z/" instead of a dynamic one like "http://deb.debian.org/debian/").
The reason that I want to merge instead of fork is mainly that I would strongly prefer not to maintain a full fork of this including CI, hosting binaries of the helper tool, converting fixes you are adding (e.g the "/etc/os-release" file), new example base containers etc. just because the method of defining and fetching the exact same deb files is improved.
There will be a small additional commit in here (pgp_keyid and pgp_keyurls fields which already work in the rewritten version of the update helper and probably porting the /etc/os-release code), but all in all the rule set in here is mostly complete. Of course I'll go ahead with it anyways if you really don't like this ruleset in this PR, but I'd really like to share the work and do my best to convince you that this approach has its merits while removing quite some risks and downsides yours currently has.

Alright, since there doesn't seem to be much interest in merging this upstream, I'll close this PR at the end of next week.
I prepared a branch at https://github.com/mgit-at/distroless/tree/deb_packages_and_updater that I'll try to regularly either rebase onto master or at least make sure it can be merged cleanly onto master.
It contains the rewritten updater tool, this ruleset from the PR here and as examples all containers from this repository also as a build and test target using this ruleset. I didn't replace the existing rules/BUILD files on purpose, as I don't really want to manually deal with merge conflicts all the time.
I also uploaded the deb_file_loader tool to a public location, so travis should feel more comfortable with testing this.
If there is interest in still merging this here, I'm all up for it, the only changes necessary would be to replace "deb.debian.org" as a mirror with the snapshot one (since building historic versions from public sources seems to be a stronger focus for you) and deciding which ruleset that you want to use (I'm not sure if it makes sense to build everything with both rulesets).
I chose to replace pgp_keyid and pgp_keyurls with a single pgp_key field as well as a mandatory http_file rule that points to the actual key. The reason is that golang (the language of the helper tool) only support 64bit pgp keyids, which is a bit short, while it is possible to define a full sha256 hash of the key that way.

After thinking about it a bit more, I rewrote the functionality of the Python tool in here in skylark to use Bazel's https://docs.bazel.build/versions/master/skylark/lib/repository_ctx.html#download function. This also removes the deb_repositories rule.
After all, all it does is download a few files with known paths from one or more mirrors, check hashes and create a few files with contents that are also known when looking at the inputs to this rule. All in all it is mostly a nicer abstraction for http_file anyways, since there is no need to deal with Packages.gz files (handling this is externalized to the updater tool).

I can also offer to include this in https://github.com/bazelbuild/rules_docker, if you think it might fit better there. After all it is closely related to the docker_build/container_image rules. @mattmoor

@MarkusTeufelberger @dlorenc @r2d4 Ultimately there may be a better home for these than this repo.
I had a good conversation with @damienmg @dslomov @davidstanke on Friday, which may be relevant to the core of the discussion here.  Essentially what it boils down to is WORKSPACE vs. WORKSPACE.lock.
I think that we want the capacity for users to express dependencies in similar terms to what they use with apt-get install where they simply have a set of mirrors and a package name (maybe expressing a particular version).  However, this is inadequate for reproducible builds because it drifts.  Pinning to a particular package resolution is adequate for reproducible builds, but will often break because packages disappear as the resolution starts pointing to something new.
Here's a strawman:

Users express their desire at a very high-level in WORKSPACE
Bazel can resolve WORKSPACE to WORKSPACE.lock, which fetches exact deps with checksums.
Bazel reads WORKSPACE.lock if present, otherwise it resolves it from WORKSPACE

There are two workflow variations on top of this that I would recommend:

Only WORKSPACE is committed to master, but WORKSPACE.lock is checked into release branches.  This keeps master from breaking often, but gives deterministic release builds.
Both WORKSPACE and WORKSPACE.lock are committed to master.  This keeps all builds deterministic, but makes HEAD more susceptible to breaks.

In both cases, producing WORKSPACE.lock should involve a single Bazel command.
I think it's relevant here because IIUC we've adopted snapshot mirrors to get something closer to the first workflow (stable HEAD) without the repo churn.  However, I think @MarkusTeufelberger wants something closer to the second workflow.
Am I understanding the nuances of the conversation correctly, or am I horribly mischaracterizing something?  Do you think that given this kind of facility in Bazel that we could potentially agree on a common definition and just have varied workflows for how we achieve our respective workflows?

I personally don't see the need for a WORKSPACE.lock file, as I already wrote elsewhere:
If you want to lock down exact versions and only state general locations in WORKSPACE (e.g. WORKSPACE would contain only the git repository location, WORKSPACE.lock would contain commit IDs or tags), in the end the only thing that you could accomplish is to strip out everything that would usually change when updating a WORKSPACE file into a WORKSPACE.lock file. To understand what is going on, one still would need to look at both files.
A way to implement your design and experiment with its merits and drawbacks right now would be to write a rules file for buildozer that reads some keys (package version, git commits...) from a lock file and the basic structure from a WORKSPACE template and generates a WORKSPACE file from that on the fly which is not checked into version control. This rules file also could be generated via a script to get updated.
In your second scenario "HEAD" only breaks if links become invalid. There are ways to make sure this doesn't happen - using snapshot mirrors, internal artifact storages/mirrors... This is an orthogonal issue, not related to this rule set or approach. As I already wrote, if you care about having build inputs available, it is easily possible to just specify the snapshot mirror at a certain time as a mirror in the rules.
My argument is that currently WORKSPACE files typically contain all information that is necessary to assess which external software in which version etc. is being used in a build. Extending Bazel to optionally allow "latest" versions (e.g. all git rules fetch from HEAD, specialized rules like the deb packages rules in distroless fetch from an up-to-date mirror...) for some or all rules might sound interesting for some use cases, but is not useful enough to be implemented as a main pattern.
Here is my desired use case for Bazel in general:

WORKSPACE files contain the most exact possible description of every input to a build process for the software described in the repository.
Every build of a freshly checked out repository must create the exact same outputs every time, as long as all inputs are available. If they are not, the build must fail, not try to do some magic trying to find a newer/compatible version or whatever else.
If one (optionally) wants to build with different inputs, the WORKSPACE file must be changed before executing bazel build ... - in the case that more recent versions or other, more dynamic things are required, this is a trivial task (since the more exact, locked down version is already in the WORKSPACE file, so one just needs to transform that one to a different one)

The current approach in this repository violates rule 1 a bit (it is not clear which packages come from backports for example).
Rule 2 currently is fulfilled because you trust the snapshot mirror, not because your rules are written in a way that supports this use case.
Rule 3 is kinda fulfilled, it is only possible to change everything at once by selecting a different snapshot, not to change different packages. This is because of your implicit dependency on the Packages.gz file (which then guides every further decision down the line)
Instead of splitting WORKSPACE, it is or could be possible to tell bazel that some or all rules in WORKSPACE also support an "unlocked" mode, which would be the opposite of your proposal (locked down by default, unlocked on demand, for canary builds for example). I don't see the value in having default unlocked builds, as you already stated that master would break with the unlocked approach from time to time and would require locking for releases at least.
I value bazel for its ability to exactly specify build inputs as exactly as possible and while Google might not have this problem with having to lock down external dependencies (since you don't have any in a monorepo), this is a big problem in the general IT landscape.
Currently it is extremely hard to know which actual deb files are put into which distroless container. A WORKSPACE.lock file won't help with your current approach (incorrectly parsing Debian archive files, generating a json file and reading everything from there while depending on an external service to provide the input file which you mostly throw away anyways - and no automated way to update to a trusted current mirror state) and imho breaks some things and assumptions that make bazel great. It would be nice to know what @damienmg @dslomov and @davidstanke had to say on this matter or if you have a more concrete design proposal.
As an aside: apt-get install ... only will resolve the local state and not fetch any updates from a Package file on a mirror - that's the job of apt-get update. This makes apt-get install ... deterministic and is the reason why it was implemented that way in Debian (the WORKSPACE.lock file equivalent there is the apt cache). apt-get install ... NEVER drifts unless users willingly update the apt cache with a different command. My design reflects this by splitting the process (update_workspace tool = apt-get update && apt-get dist-upgrade, bazel build = apt-get install), the current one is a mixture of fetching and parsing Package files (apt-get update) and fetching the actual deb file (apt-get install), all in one single step, currently only secured by the person that checked in the SHA256 hash of the Packages file, added after complaints about the procedure in https://github.com/GoogleCloudPlatform/distroless/issues/72.
This is an issue for me and one reason I still continue to push for changes in this process is that images produced using this process are used upstream in the https://github.com/bazelbuild/rules_docker repository as default base images. all while not even clearly documenting which of your packages are from which repository (jessie vs. jessie-backports, no versions documented even in comments...).
Lastly I'd still like to know answers to the questions I asked above: Do you want to continue with the Package.json approach and why? Would you be OK with merging this one, would you prefer having this one replace yours or would you prefer a fork? Your answer at least sounds like the latter for the second question. You are also often committing in https://github.com/bazelbuild/rules_docker though, which would be the other logical place for this rule upstream but I'm not so sure if I should submit it there.
Tl;dr: It is trivial to change a git_repository rule in a  WORKSPACE file to use the current HEAD before building, it does not help anyone to only record a repository location in a WORKSPACE file if the commit to be used is not specified in a WORKSPACE.lock file, as this will break builds sooner or later and not be reproducible (similar for deb package fetching rules). This means that the lock files are required anyways and could be merged into the WORKSPACE file, creating exactly the situation we already have.
I'm all for (optional) builds with "unlocked" WORKSPACE rules, I am strongly against having these as default.

I have a design that summarize our discussion, I'll share it wisely
tomorrow when everybody involved had time to take a quick look.
On Mon, Oct 9, 2017, 11:28 AM MarkusTeufelberger<EMAIL_ADDRESS>wrote:

I personally don't see the need for a WORKSPACE.lock file, as I already
wrote elsewhere:
If you want to lock down exact versions and only state general locations
in WORKSPACE (e.g. WORKSPACE would contain only the git repository
location, WORKSPACE.lock would contain commit IDs or tags), in the end
the only thing that you could accomplish is to strip out everything that
would usually change when updating a WORKSPACE file into a WORKSPACE.lock
file. To understand what is going on, one still would need to look at both
files.
A way to implement your design and experiment with its merits and
drawbacks right now would be to write a rules file for buildozer that
reads some keys (package version, git commits...) from a lock file and
the basic structure from a WORKSPACE template and generates a WORKSPACE
file from that on the fly which is not checked into version control. This
rules file also could be generated via a script to get updated.
In your second scenario "HEAD" only breaks if links become invalid. There
are ways to make sure this doesn't happen - using snapshot mirrors,
internal artifact storages/mirrors... This is an orthogonal issue, not
related to this rule set or approach. As I already wrote, if you care about
having build inputs available, it is easily possible to just specify the
snapshot mirror at a certain time as a mirror in the rules.
My argument is that currently WORKSPACE files typically contain all
information that is necessary to assess which external software in which
version etc. is being used in a build. Extending Bazel to optionally allow
"latest" versions (e.g. all git rules fetch from HEAD, specialized rules
like the deb packages rules in distroless fetch from an up-to-date
mirror...) for some or all rules might sound interesting for some use
cases, but is not useful enough to be implemented as a main pattern.
Here is my desired use case for Bazel in general:

WORKSPACE files contain the most exact possible description of
every input to a build process for the software described in the repository.
Every build of a freshly checked out repository must create the
exact same outputs every time, as long as all inputs are available. If they
are not, the build must fail, not try to do some magic trying to find a
newer/compatible version or whatever else.
If one (optionally) wants to build with different inputs, the
WORKSPACE file must be changed before executing bazel build ... - in
the case that more recent versions or other, more dynamic things are
required, this is a trivial task (since the more exact, locked down version
is already in the WORKSPACE file, so one just needs to transform that
one to a different one)

The current approach in this repository violates rule 1 a bit (it is not
clear which packages come from backports for example).
Rule 2 currently is fulfilled because you trust the snapshot mirror, not
because your rules are written in a way that supports this use case.
Rule 3 is kinda fulfilled, it is only possible to change everything at
once by selecting a different snapshot, not to change different packages.
This is because of your implicit dependency on the Packages.gz file
(which then guides every further decision down the line)
Instead of splitting WORKSPACE, it is or could be possible to tell bazel
that some or all rules in WORKSPACE also support an "unlocked" mode,
which would be the opposite of your proposal (locked down by default,
unlocked on demand, for canary builds for example). I don't see the value
in having default unlocked builds, as you already stated that master
would break with the unlocked approach from time to time and would require
locking for releases at least.
I value bazel for its ability to exactly specify build inputs as exactly
as possible and while Google might not have this problem with having to
lock down external dependencies (since you don't have any in a monorepo),
this is a big problem in the general IT landscape.
Currently it is extremely hard to know which actual deb files are put into
which distroless container. A WORKSPACE.lock file won't help with your
current approach (incorrectly parsing Debian archive files, generating a
json file and reading everything from there while depending on an external
service to provide the input file which you mostly throw away anyways - and
no automated way to update to a trusted current mirror state) and imho
breaks some things and assumptions that make bazel great. It would be nice
to know what @damienmg https://github.com/damienmg @dslomov
https://github.com/dslomov and @davidstanke
https://github.com/davidstanke had to say on this matter or if you have
a more concrete design proposal.
As an aside: apt-get install ... only will resolve the local state and
not fetch any updates from a Package file on a mirror - that's the job of apt-get
update. This makes apt-get install ... deterministic and is the reason
why it was implemented that way in Debian (the WORKSPACE.lock file
equivalent there is the apt cache). apt-get install ... NEVER drifts
unless users willingly update the apt cache with a different command. My
design reflects this by splitting the process (update_workspace tool = apt-get
update && apt-get dist-upgrade, bazel build = apt-get install), the
current one is a mixture of fetching and parsing Package files (apt-get
update) and fetching the actual deb file (apt-get install), all in one
single step, currently only secured by the person that checked in the
SHA256 hash of the Packages file, added after complaints about the
procedure in #72
https://github.com/GoogleCloudPlatform/distroless/issues/72.
This is an issue for me and one reason I still continue to push for
changes in this process is that images produced using this process are used
upstream in the https://github.com/bazelbuild/rules_docker repository as
default base images. all while not even clearly documenting which of your
packages are from which repository (jessie vs. jessie-backports, no
versions documented even in comments...).
Lastly I'd still like to know answers to the questions I asked above: Do
you want to continue with the Package.json approach and why? Would you be
OK with merging this one, would you prefer having this one replace yours or
would you prefer a fork? Your answer at least sounds like the latter for
the second question. You are also often committing in
https://github.com/bazelbuild/rules_docker though, which would be the
other logical place for this rule upstream but I'm not so sure if I should
submit it there.
Tl;dr: It is trivial to change a git_repository rule in a WORKSPACE file
to use the current HEAD before building, it does not help anyone to only
record a repository location in a WORKSPACE file if the commit to be used
is not specified in a WORKSPACE.lock file, as this will break builds
sooner or later and not be reproducible (similar for deb package fetching
rules). This means that the lock files are required anyways and could be
merged into the WORKSPACE file, creating exactly the situation we already
have.
I'm all for (optional) builds with "unlocked" WORKSPACE rules, I am
strongly against having these as default.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/distroless/pull/96#issuecomment-335106813,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADjHf5hzZYxbQAk4GcRk4cdanEYgEdVxks5sqedXgaJpZM4PNTtB
.

Thanks! :+1:

We've talked a bit about putting these rules (or different versions of them) into a repo in the bazel org. Maybe rules_debian or rules_dpkg? Or maybe include rules for other package managers there?
Like I said earlier, I'd prefer to keep the design we have for now in distroless. We're not really planning to use these rules too widely for now, and what we have meets our use-case. The bazel org is the right place for rules that will be publicly shared, we're mainly using this repo to host the images and samples.
I'm definitely interested in seeing the new design, @damienmg!

Great thanks, trying to set up a repository like that would have been my next way of action anyways. :-)
Maybe it would even be useful to add the one(s) from here to something like rules_pkg_files as well and only maintain BUILD files in distroless so you really just host images and samples? Anyways, thanks for responding and working on solutions.

@MarkusTeufelberger Are you by chance coming to the Bazel conference?  If so, that might be a good place to discuss some of this face-to-face?

I'd love too, really, but it is a bit far (and expensive) from here unfortunately. I'll try to come to the European version though.

@damienmg Can you create an issue to track this on the main repo, so that we can capture the discussion in a more appropriate forum?  This has become a huge tangent here, for which I apologize.
@MarkusTeufelberger all of your responses further convince me we're after the same thing, so clearly I'm not articulating my goal well. :)

it does not help anyone to only record a repository location in a WORKSPACE file if the commit to be used is not specified in a WORKSPACE.lock file

e.g. here I would fully expect WORKSPACE => WORKSPACE.lock to add a commit attribute.  @damienmg's example has an example of how we might achieve this IIRC.
Let's create the umbrella issue and continue the discussion there.  It may make sense to set up a video call to discuss this further.

Agreed, I think we are after very similar things and only disagreeing about some semantics (e.g. which file should be the one where some specific info like a file hash or commit ID should be stored).
I'm on the same time zone as the Munich office with regards to the video call.
Back on topic, I'll set up a repository (and come up with a good name for it) asap to contain at least the rule and tool from here, maybe (if you want to) also the dpkg_* ones from distroless itself.

To keep the tangent going, I've spent some time looking into the Ubuntu mirrors, and am a bit worried that neither of these designs will work well there.
I'll open an issue with some thoughts on how we might support Ubuntu.

@dlorenc I don't think that's the tangent :)
@MarkusTeufelberger @damienmg We might also consider splitting pkg_tar off of Bazel Core while we're at it (this contains the logic to expand debs into a tar.

@dlorenc, Ubuntu mirrors work out of the box with my rules, though there is no "snapshot" mirror I am aware of. What excatly didn't work, design wise? deb_packages rules are more or less aliased http_file rules, so I don't see an immediate issue.

Yeah, they have no snapshot and aggressively delete old versions of packages. Any WORKSPACE that pins to full shas of Ubuntu packages will be broken very frequently.
This might just be "working as intended", though...

I would say that's by design, the snapshot mirror is the exceptional case...

Yeah, the downside is that you won't be able to reproduce any old builds without running your own mirror.

...or hosting the deb files elsewhere, yes. At least you'd know exactly which deb file with which name and which SHA256 hash you'd need, with the dpkg_* rules you'd need to archive the Packages file too.

So I propose we move the discussion to the design doc: https://docs.google.com/document/d/1HfRGRW4MwnVUG24rw3HJIBkdEYfXlESnlKOboO97A3A/
For moving the rules I am up for that, want to create rules_pkg to contains RPM, DEB and so on for packaging rules.

I created https://github.com/mgit-at/rules_pkg (with the boilerplate files like LICENSE, README.md etc. missing) and added a smallish CI script for Travis-ci. Is this kinda what you have in mind, @damienmg ? If yes, as soon as bazelbuild/rules_pkg is available, I would open a pull request.

Sounds good, I'll try to bootstrap that soonish, I'll move docker / pkg
rules outside of bazel_tools definitely with that but I cannot guarantee
any ETA.
On Wed, Oct 11, 2017 at 5:27 PM MarkusTeufelberger<EMAIL_ADDRESS>wrote:

I created https://github.com/mgit-at/rules_pkg (with the boilerplate
files like LICENSE, README.md etc. missing) and added a smallish CI script
for Travis-ci. Is this kinda what you have in mind, @damienmg
https://github.com/damienmg ? If yes, as soon as bazelbuild/rules_pkg
is available, I would open a pull request.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/distroless/pull/96#issuecomment-335849356,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADjHf7WHcOVAWyzRHdtWhFREqpHzrPpgks5srN5ngaJpZM4PNTtB
.

Closing in favor of rules_pkg. Would be great to have the repo in the bazelbuild org soonish, it is not that urgent to immediately consolidate all rules in there...

ping @damienmg  - it would be nice to get started on bazelbuild/rules_pkg. :-)

Sorry I have a hundred things on my plate, I cannot guarantee any ETA for
this.
On Mon, Oct 30, 2017 at 1:13 PM MarkusTeufelberger<EMAIL_ADDRESS>wrote:

ping @damienmg https://github.com/damienmg - it would be nice to get
started on bazelbuild/rules_pkg. :-)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/distroless/pull/96#issuecomment-340426396,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADjHfysDKIdh35_Vt2okd3u86GwBuvCpks5sxb1PgaJpZM4PNTtB
.

Maybe there's someone else that you could delegate the repo creation to? It is not extremely urgent of course, I just didn't want to let it slip completely out of sight. I'll ping again in a while, if that's ok.

Another sneaky ping @damienmg - I hope the conference was great! :-)

The conference was great :) Unfortunately I have still no visibility on
doing that.
On Mon, Nov 20, 2017 at 11:41 PM MarkusTeufelberger <
<EMAIL_ADDRESS>wrote:

Another sneaky ping @damienmg https://github.com/damienmg - I hope the
conference was great! :-)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/distroless/pull/96#issuecomment-345856004,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADjHf4LwMWh_eoj9GfQvz1xyCwDtNW8wks5s4gAIgaJpZM4PNTtB
.

And according to the mailing list you're gone from bazel! :-(
Hope your new job is great, whom should I contact about getting this repo created?

/cc @davidstanke is the other Github owner but I believe they need to figure out a proper owner for pkg_tar.

Hey hi! Apologies that this got lost in the shuffle. I'll pick this up now. My only nit is this: AFAICT, these rules are specific to debian packages, yes? Given that, I believe the name rules_pkg is too generic.  Can we do rules_dpkg?
Or something else that specifically references Debian, to preclude any possible confusion with other packaging mechanisms? Thanks!

The idea was to move several packaging rules (including dpkg, deb, tar... maybe yum or rpm in the future?) into a common repo as far as I remember.

Cool, in that case, I'm good with it. Will proceed with the repo... when
you push to it, can you include a README that speaks to those future plans?
On Fri, Dec 15, 2017 at 3:25 AM MarkusTeufelberger<EMAIL_ADDRESS>wrote:

The idea was to move several packaging rules (including dpkg, deb, tar...
maybe yum or rpm in the future?) into a common repo as far as I remember.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/distroless/pull/96#issuecomment-351943565,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACENj-y5w3TsoCRMuiqFhOK94WLQUwnBks5tAiz0gaJpZM4PNTtB
.

Of course. I would also love to see the current rules + helper tool of distroless go in there, since right now this repository is a mix of production BUILD files, bzl rules and some more general examples.

Update: I've kicked off the process to create the repo (github.com/bazelbuild/rules_pkg). It takes a few days for Google to review the request, so bear with us. :)

Okay, the repo is live: https://github.com/bazelbuild/rules_pkg
Can you make a PR there with the initial content?

Awesome, I'm currently working on it and hope to be done tomorrow. :+1: