Vercast add menubar command

Description
Added menubar command to vercast extension and updated packages to latest.
Screencast

Checklist

[x] I read the extension guidelines
[x] I read the documentation about publishing
[x] I ran npm run build and tested this distribution build in Raycast
[x] I checked that files in the assets folder are used by the extension itself
[x] I checked that assets used by the README are placed outside of the metadata folder

I can see you're still working on this @bgiovand - right?


Error: unsupported broker: can't import module zipline.gens.brokers.ib_broker

Dear Zipline-Trader Maintainers,
Key Question: what do I need to install on top of python, C++ 14.0, IB TWS and zipline-trader for live trading? Backtesting is working well.

Before I tell you about my issue, let me describe my environment:

Operating System: Windows 10
Python Version: Python 3.6.8rc1
Python Bitness: 64
How did you install Zipline: pip
Python packages: pure zipline-trader

(2) Now that you know a little about me, let me tell you about the issue I am having:
I am trying to connect to IB using TWS for trading, e.g. with

zipline run -f my_zl_trader_test.py --state-file my_zl_trader_test.state --broker ib --broker-url localhost:7496:1232 --bundle quantopian-quandl --data-frequency daily --realtime-bar-target ./realtime-bars
The error message is:

Error: unsupported broker: can't import module zipline.gens.brokers.ib_broker
If I check the file ib_broker.py I find:
from ib.ext.EClientSocket import EClientSocket
from ib.ext.EWrapper import EWrapper
from ib.ext.Contract import Contract
from ib.ext.Order import Order
from ib.ext.ExecutionFilter import ExecutionFilter
from ib.ext.EClientErrors import EClientErrors

-> Do I need to install anything else?
(3) Here is how you can reproduce this issue on your machine:
Create a new env
Install zipline-trader from pip
Have TWS running
Use the CLI

(4) What steps have you taken to resolve this already?
Followed the instructions from the zipline-trader site and followed the video on installing zipline
Followed the instructions from IB on how to set up the API

Did a lot of Research and found e.g.
    https://groups.google.com/forum/#!topic/zipline-live/jLc1jU4-nWQ

Did some bugfixes:

C:\Users\Frank\AppData\Local\Programs\Python\Python36\myzipenv\Lib\site-packages\zipline\__main__.py
in line 19, add:
    import pkgutil (-> fixed a bug)
    from importlib import import_module (-> fixed a bug)

pip install Polling (-> fixed a bug)

Installed the TWS API from IB (-> same error remains)

The error results from these lines in __main__:
    try:
        bmod = import_module(mod_name)
    except ImportError:
        ctx.fail("unsupported broker: can't import module %s" % mod_name)

Thanks a lot for taking the time,
Frank

Hi there,
it seems that zipline live requires IBpy to connect to TWS. Since IBpy has not been updated for several years, I try to connect to the official TWS API. Therefore, I installed the TWS API and created a new file ib_broker.zip which needs to be placed in the directory /zipline/gens/brokers. Any thoughts on this file are highly appreciated - so far no errors in my configuration, but not used in live trading so far as it is Sunday.
Thanks,
Frank

Summary of changes bade (see above):
Imports
#from ib.ext.EClientSocket import EClientSocket
from ibapi.client import *
#from ib.ext.EWrapper import EWrapper
from ibapi.wrapper import *
#from ib.ext.Contract import Contract
from ibapi.contract import *
#from ib.ext.Order import Order
from ibapi.order import *
#from ib.ext.ExecutionFilter import ExecutionFilter
#from ib.ext.EClientErrors import EClientErrors
from ibapi.common import *
from ibapi import *
Classes
#class TWSConnection(EClientSocket, EWrapper):
class TWSConnection(EClient, EWrapper):
In the def init of class TWSConnection(EClient, EWrapper)
# EClientSocket.__init__(self, anyWrapper=self)
EClient.__init__(self, wrapper=self)
self.eConnect = self.connect
At the end of class TWSConnection(EClient, EWrapper)
#self.connect()
self.myConnect()
Replacing def connect(self)
 #def connect(self):
def myConnect(self):

Frank,
Check out:
https://pypi.org/project/zipline-live2/
https://github.com/shlomikushchi/zipline-live2
Will save you work!
alan

Hi Alan,
thanks a lot for the links!
On the 2nd link, it says that zipline-live2 is now continued under https://github.com/shlomikushchi/zipline-trader, which is what I installed. Both versions import ib.ext in the file ib_broker.py and therefore seem to require an IBpy installation.
On https://github.com/blampe/IbPy (last updated 2017), it says Beginning with release 9.73, InteractiveBrokers is now officially supporting a new Python API client (Python 3 only). This should make this repo superfluous except for Python 2.
Is this what the community is using? Very good, should it still be up to date! With with Python version does it run stable, and should Python then be installed from Conda or from Python.Org?
Thanks again,
Frank

Frank,
I don't know exactly what is wrong with what your are doing.

We are using anaconda, and then installing our slight modification(zipline-broker) of zipline-live2 (Note: we haven't tried the new version zipline-broker yet) on top of it.
That kind of install I can help you with...some version of Linux...for us, at this time, Windows 10 is a step too far that does nothing for our current goals.
alan

Hi Alan,
thanks a lot for your hepl! In the meantime, I was able to move a bit further, one step at a time... Still way to go but on other topics now, so I am closing this issue. It feels a bit strange to work with such old versions but it only means more thorough testing ;)
Frank


Update modules tutorial to use new DSL2 native syntax

https://github.com/nf-core/nf-co.re/pull/1090#pullrequestreview-914448313

PR https://github.com/nf-core/nf-co.re/pull/1090 fixes this issue and has been completed.


Added badges to the readme

To be honest, I am not really excited about badge shenanigans. It provides no useful information:

License badge is pointless since GitHub already shows the license.
Code size is irrelevant -what's the point?- and inaccurate -what's being measured?-.
Twitter is not really relevant enough to have a badge of its own. Plus, it's just mentioned in the readme.
Discord is not really relevant enough to have a badge of its own.

The only reason I see to add badges would be for providing dynamic information about the current state of the repository, e.g. build status, tests, coverage, and other metrics.

Thanks. To be honest, I am not really excited about badge shenanigans. It provides no useful information:

License badge is pointless since GitHub already shows the license.
Code size is irrelevant -what's the point?- and inaccurate -what's being measured?-.
Twitter is not really relevant enough to have a badge of its own. Plus, it's just mentioned in the readme.
Discord is not really relevant enough to have a badge of its own.

The only reason I see to add badges would be for providing dynamic information about the current state of the repository, e.g. build status, tests, coverage, and other metrics.

hmm.
Perhaps a revisit of the idea would work when it gets a GitHub release and CI.

Sure thing.


[CARBONDATA-3371] Fix ArrayIndexOutOfBoundsException of compaction after sort_columns modification

Modification:

SegmentPropertiesWrapper should check the column order for different segments
Because sort_columns modification can change the column order.
dictionaryColumnChunkIndex of blockExecutionInfo should keep projection order for compaction
if column drift happened, it should convert measure to dimension in RawResultIterator

[x] Any interfaces changed?
no

[x] Any backward compatibility impacted?
no

[x] Document update required?
no

[x] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
added unit test cases
- How it is tested? Please attach test report.
unit test cases passed
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.

[x] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
small chagnes

Build Failed  with Spark 2.3.2, Please check CI http://<IP_ADDRESS>:8080/job/carbondataprbuilder2.3/11348/

Build Failed with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/3283/

Build Failed  with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder2.1/3084/

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder2.1/3085/

Build Failed  with Spark 2.3.2, Please check CI http://<IP_ADDRESS>:8080/job/carbondataprbuilder2.3/11349/

Build Failed with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/3284/

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder2.1/3089/

Build Success with Spark 2.3.2, Please check CI http://<IP_ADDRESS>:8080/job/carbondataprbuilder2.3/11353/

Build Success with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/3288/

@ravipesala please review again

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder2.1/3113/

Build Success with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/3312/

Build Success with Spark 2.3.2, Please check CI http://<IP_ADDRESS>:8080/job/carbondataprbuilder2.3/11377/

@QiangCai Please resolve conflicts

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder2.1/3134/

Build Success with Spark 2.3.2, Please check CI http://<IP_ADDRESS>:8080/job/carbondataprbuilder2.3/11399/

Build Success with Spark 2.3.2, Please check CI http://<IP_ADDRESS>:8080/job/carbondataprbuilder2.3/11399/

LGTM


Rename gem to opentelemetry-exporter-datadog

Per https://github.com/open-telemetry/opentelemetry-specification/pull/629 and https://github.com/open-telemetry/opentelemetry-ruby/issues/307, this gem should probably be called opentelemetry-exporter-datadog. Note the singular exporter.
https://github.com/DataDog/dd-opentelemetry-exporter-ruby/blob/9ff3c2b2d2ebce67c2276c15083c87c4a62a8de9/opentelemetry-exporters-datadog.gemspec#L14

@fbogsany Thanks for the heads up here. I'll work to get this updated.


Adds edit link to blog posts

Direct link to edit a post from a blog post. Only visible when logged in with proper privileges

Thanks! Should make it in the next release.


Task1/Week1 Completed by SauravKhanal

Learning of SQLITE/Week1

It would have been okay to have all the sql into one file but overall it's good work. merging it.


PP-4492: Expose gatway transaction ID for refunds

Links RefundEntity to new gatway_transaction_id column
Doesn't expose details column, this should be updated given a use case
Adds gateway_transaction_id to refunds_history as this directly tracks the columns in the refunds table
Depends on https://github.com/alphagov/pay-connector/pull/1158

[x] Update any relevant tests
[x] Update any other entities that needs this information

Note Codacy rule flagged here is going to be ignored in the future, it doesn't seem to actually line up with repository uses:

Use explicit scoping instead of the default package private level

Having looked at the code again, looks like HistoryCustomiser is taking care of adding gateway_account_id into RefundsHistory. We still need to decide if we need to add gateway_account_id for select on RefundsHistory (Don't need it if impacts any API) https://github.com/alphagov/pay-connector/blob/master/src/main/java/uk/gov/pay/connector/refund/model/domain/RefundEntity.java#L37


Release v1.1.0

User Story

As a [user role below]:
Release manager

I need to:
create a release branch, one or multiple release candidates, and the final v1.1.0 release of the IBM.com library

so that I can:
release the next version of the IBM.com library to the public

Additional information

Release Management Playlist: https://ibm.ent.box.com/notes/497596874716
Release Plan: https://github.com/carbon-design-system/ibm-dotcom-library/issues/99

Acceptance criteria

Release branch created per timeline (should be Friday before the sprint begins)
v1.1.0 of the IBM.com Library released per timeline

@annawen1 let me know if you want me to walk through this with you when you're doing this.

Release was made on Nov 4th:
https://github.com/carbon-design-system/ibm-dotcom-library/releases/tag/v1.1.0


Make google maps link directly to the HQ location

If we only use the coordinates (@52.3314517,4.9166934,17z) it shows the right location, but with no red dot.
If we use the location (place/Van+Marwijk+Kooystraat+15,+1114+AG+Amsterdam,+Netherlands) it shows the dot, but the dot is not exactly Picnic HQ.
To make it refer to the exact location the link has to have the coordinates, the location and a data block whatever that means. The link becomes very long, so I've opted for a shortened one.
Full link: https://www.google.com/maps/place/Picnic+HQ/@52.3313828,4.9192147,18z/data=!4m5!3m4!1s0x47c60b5f55cddecd:0x664c31f150321239!8m2!3d52.3314484!4d4.9188821
WDYT?
Fixes #12.

@Stephan202 I've played with the concept a bit. Do you think this would make sense?


Elf: invalid relocations to not mapped memory

This is https://github.com/adimetrius/Herschel/commit/cc4bead0534a6824bca446a8d906e7952b0d8ffd.
ELF segments of Albert sample:
Kind	File Offset	File Size	Virt. Adr	Phys. Adr	Mem. Size	Align	Flags
load	00001000	00000080	00405000	00405000	00000080	00001000	write, read
load	00002000	00000058	00406000	00406000	00000058	00001000	exec, read
load	00003000	00000090	00407000	00407000	000000D0	00001000	write, read
load	00004000	000002D8	00408000	00408000	000002D8	00001000	write, read

Relocations:
Address		Old		Kind	Value	Addend
00405080	00000000	abs64	exit	0
00405078	00000000	abs64	puts	0

Note that address 00405080 do not point to any existing segment.

I attached Albert.odc that contains report of my ElfDecoder.
Albert.zip

Disassembly utilities such as IDA Pro Free fails to recognize exit function because it is located outside of segment.

Well, this is by design, actually, but I'm open for criticism and suggestions.
The idea is that for every external function (imported from a library) the compiler expects a proxy in the special proxy table. So, when an external function is called, the actual address is loaded from the proxy table into a register, and a CALL [reg] is executed. The indexes into the proxy table are hard-coded by the compiler. It is the loader's responsibility to fill out the proxy table to provide for proper linking to external funcitons (be it the host OS or the BlackBox linker).
And it only seemed reasonable to me that the proxy table does not need to be present in the ELF file (and neither in the OCF file). Memory space is allocated for it (at the end of meta, segment 0 in your awesome decoder output), and then the OS's dynamic loader fills that out based on information provided in the DYNAMIC segment.

The problem is there are extra zero first entry in proxy table that is not included in segment size. Increasing segment that contains proxy table by 8 bytes will fix the issue. I am not sure what exact code should be adjusted in HrElf to properly fix issue. I added 1 to m.imps.

I see what you're saying. I think the line is in HrElf.DoAddOcf, and it's this one:
	AllocSeg(m.meta, AdAligned(m.ms) + (m.imps + m.exts) * addrSize[version], pageAligned);
shoud be
AllocSeg(m.meta, AdAligned(m.ms) + (1 + m.imps + m.exts) * addrSize[version], pageAligned);
If you confirm this fixes the issue, I'll put it into my modules.

AllocSeg(m.meta, AdAligned(m.ms) + (1 + m.imps + m.exts) * addrSize[version], pageAligned);

This will cause trap in size mismatch.
Place that I changed:
	PROCEDURE DoAddOcf (ocf: Files.File; VAR res: INTEGER);
		VAR s: OcfScanner; _, i: INTEGER; n: Name; meta, desc, code, imps: POINTER TO ARRAY OF BYTE;
			prev, m: MModule;
			c: ARRAY 3 OF LONGINT;
	BEGIN
		s.ConnectTo(ocf, 0);	(* header and descriptor are in, module can be scanned *)
		IF ~s.ok THEN HALT(20)
		ELSE
			NEW(m);
			m.ms := s.ms; m.ds := s.ds; m.cs:= s.cs; m.vs := s.vs; m.imps := s.imps + 1; (* zero entry in proxy table *) m.exts := s.exts; m.name := s.name$;

Hm... I tried my location of +1, and can't get a trap. Can you reproduce and send the trap you receive?

Hm... I tried my location of +1, and can't get a trap.

Sorry, trap was caused when inserting "+1" in one line above. Your change don't cause trap but it don't solve issue:
Reloc (Addend)
Address	Old	Kind	Value	Addend
004050B0	00000000	abs64	Do6	0
004050A8	00000000	abs64	Do5	0
004050A0	00000000	abs64	Do4	0
00405098	00000000	abs64	Do3	0
00405090	00000000	abs64	Do2	0
00405088	00000000	abs64	Do1	0
00405080	00000000	abs64	exit	0
00405078	00000000	abs64	puts	0

Segment 0 (00405000, 000000B0, data)
00405000  17 FD 07 FC 99 01 24 24 00 FD 10 80 70 40 00 00<EMAIL_ADDRESS>00405010  73 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  s...............
00405020  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
00405030  00 62 75 74 20 6E 6F 74 20 73 69 6D 70 6C 65 72  .but not simpler
00405040  00 41 6C 62 65 72 74 20 61 73 6B 65 64 20 74 6F  .Albert asked to
00405050  20 6D 61 6B 65 20 69 74 20 61 73 20 73 69 6D 70   make it as simp
00405060  6C 65 20 61 73 20 70 6F 73 73 69 62 6C 65 2C 00  le as possible,.
00405070  00 00 00 00 00 00 00 00                          ........
  __imp_puts
00405070                          00 00 00 00 00 00 00 00          ........
  __imp_exit
00405080  00 00 00 00 00 00 00 00                          ........
  __imp_Do1
00405080                          00 00 00 00 00 00 00 00          ........
  __imp_Do2
00405090  00 00 00 00 00 00 00 00                          ........
  __imp_Do3
00405090                          00 00 00 00 00 00 00 00          ........
  __imp_Do4
004050A0  00 00 00 00 00 00 00 00                          ........
  __imp_Do5
004050A0                          00 00 00 00 00 00 00 00          ........

Note that "__imp_Do6" is missing. My position fixes issue:
Reloc (Addend)
Address	Old	Kind	Value	Addend
004050B0	00000000	abs64	Do6	0
004050A8	00000000	abs64	Do5	0
004050A0	00000000	abs64	Do4	0
00405098	00000000	abs64	Do3	0
00405090	00000000	abs64	Do2	0
00405088	00000000	abs64	Do1	0
00405080	00000000	abs64	exit	0
00405078	00000000	abs64	puts	0

Segment 0 (00405000, 000000B8, data)
00405000  17 FD 07 FC 99 01 24 24 00 FD 10 80 70 40 00 00<EMAIL_ADDRESS>00405010  73 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  s...............
00405020  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
00405030  00 62 75 74 20 6E 6F 74 20 73 69 6D 70 6C 65 72  .but not simpler
00405040  00 41 6C 62 65 72 74 20 61 73 6B 65 64 20 74 6F  .Albert asked to
00405050  20 6D 61 6B 65 20 69 74 20 61 73 20 73 69 6D 70   make it as simp
00405060  6C 65 20 61 73 20 70 6F 73 73 69 62 6C 65 2C 00  le as possible,.
00405070  00 00 00 00 00 00 00 00                          ........
  __imp_puts
00405070                          00 00 00 00 00 00 00 00          ........
  __imp_exit
00405080  00 00 00 00 00 00 00 00                          ........
  __imp_Do1
00405080                          00 00 00 00 00 00 00 00          ........
  __imp_Do2
00405090  00 00 00 00 00 00 00 00                          ........
  __imp_Do3
00405090                          00 00 00 00 00 00 00 00          ........
  __imp_Do4
004050A0  00 00 00 00 00 00 00 00                          ........
  __imp_Do5
004050A0                          00 00 00 00 00 00 00 00          ........
  __imp_Do6
004050B0  00 00 00 00 00 00 00 00                          ........

In this code segment size is defined by LEN(meta) + LEN(imps):
				IF m.imps > 0 THEN NEW(imps, m.imps * addrSize[version]) END;
...
					OutSeg(m.meta, meta, imps);

Fixed In HrBinder.

Fixed In HrBinder.


Use document.documentElement.cloneNode() when parsing to fix resolution independence

This follows on from issue https://github.com/rails/turbolinks/issues/413 (which has a fair bit more background info).
In order to fix the resolution independence issues we've been seeing (where the @1x image is always chosen even when higher resolutions would be appropriate) we've been using the createDocument function in this pull request, using document.documentElement.cloneNode() in place of the other document creation techniques. We've been using this on a staging environment and so far have seen no issues.
For this initial pull request I've ripped out all of the other document creation functions as it seems cleaner to me to use a single function across all browsers with a fallback only where necessary. Unfortunately I'm not really clear how to check where a fallback would be necessary. I've done sanity checks on Chrome, Safari, Opera, IE10, iOS Safari, Android Chrome, and Firefox and it all looks good, but what else should I look at to add confidence?
I have written a performance check and it seems that using cloneNode() is generally faster (though performance is so high already there's no appreciable benefit): http://jsperf.com/compare-turbolinks-document-creation
Any comments, questions, and help getting this pull request either improved or rejected will be gratefully received.

Because of a customer using a very old version of Firefox (11.0) which required a parameter to cloneNode. The problem caused the links not working at all.
  doc = document.documentElement.cloneNode(true)


SSL: CERTIFICATE_VERIFY_FAILED error on "az bicep install" Resurfaced

Describe the bug
This is a reopening of #19420 the az bicep install command is once again throwing an SSL: CERTIFICATE_VERIFY_FAILED error
To Reproduce
$ az bicep install
Error while attempting to retrieve the latest Bicep version: HTTPSConnectionPool(host='aka.ms', port=443): Max retries exceeded with url: /BicepLatestRelease (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)).

With debug:
$ az bicep install --debug
cli.knack.cli: Command arguments: ['bicep', 'install', '--debug']
cli.knack.cli: __init__ debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7f603acf9b70>, <function OutputProducer.on_global_arguments at 0x7f603ac2b730>, <function CLIQuery.on_global_arguments at 0x7f603ac4b840>]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: Modules found from index for 'bicep': ['azure.cli.command_modules.resource']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name                  Load Time    Groups  Commands
cli.azure.cli.core: resource                  0.010        40       187
cli.azure.cli.core: Total (1)                 0.010        40       187
cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next']
cli.azure.cli.core: Loading extensions:
cli.azure.cli.core: Name                  Load Time    Groups  Commands  Directory
cli.azure.cli.core: Total (0)                 0.000         0         0
cli.azure.cli.core: Loaded 40 groups, 187 commands.
cli.azure.cli.core: Found a match in the command table.
cli.azure.cli.core: Raw command  : bicep install
cli.azure.cli.core: Command table: bicep install
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x7f603a76c598>]
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.<locals>.add_subscription_parameter at 0x7f603a725620>, <function register_global_query_examples_argument.<locals>.register_query_examples at 0x7f603a6af840>]
cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument.<locals>.add_ids_arguments at 0x7f603a6af8c8>, <function register_cache_arguments.<locals>.add_cache_arguments at 0x7f603a6af9d8>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x7f603ac2b7b8>, <function CLIQuery.handle_query_parameter at 0x7f603ac4b8c8>, <function register_global_query_examples_argument.<locals>.handle_example_parameter at 0x7f603a6af7b8>, <function register_ids_argument.<locals>.parse_ids_arguments at 0x7f603a6af950>]
urllib3.connectionpool: Starting new HTTPS connection (1): aka.ms:443
cli.azure.cli.core.util: azure.cli.core.util.handle_exception is called with an exception:
cli.azure.cli.core.util: Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/opt/az/lib/python3.6/site-packages/urllib3/connection.py", line 426, in connect
    tls_in_tls=tls_in_tls,
  File "/opt/az/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 450, in ssl_wrap_socket
    sock, context, tls_in_tls, server_hostname=server_hostname
  File "/opt/az/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/opt/az/lib/python3.6/ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
  File "/opt/az/lib/python3.6/ssl.py", line 817, in __init__
    self.do_handshake()
  File "/opt/az/lib/python3.6/ssl.py", line 1077, in do_handshake
    self._sslobj.do_handshake()
  File "/opt/az/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py", line 756, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/opt/az/lib/python3.6/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='aka.ms', port=443): Max retries exceeded with url: /BicepLatestRelease (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/resource/_bicep.py", line 156, in get_bicep_latest_release_tag
    response = requests.get("https://aka.ms/BicepLatestRelease", verify=ca_file)
  File "/opt/az/lib/python3.6/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/az/lib/python3.6/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='aka.ms', port=443): Max retries exceeded with url: /BicepLatestRelease (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/knack/cli.py", line 231, in invoke
    cmd_result = self.invocation.execute(args)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 658, in execute
    raise ex
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 721, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 692, in _run_job
    result = cmd_copy(params)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 328, in __call__
    return self.handler(*args, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/resource/custom.py", line 3588, in install_bicep_cli
    ensure_bicep_installation(release_tag=version, target_platform=target_platform)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/resource/_bicep.py", line 106, in ensure_bicep_installation
    release_tag = release_tag if release_tag else get_bicep_latest_release_tag()
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/resource/_bicep.py", line 160, in get_bicep_latest_release_tag
    raise ClientRequestError(f"Error while attempting to retrieve the latest Bicep version: {err}.")
azure.cli.core.azclierror.ClientRequestError: Error while attempting to retrieve the latest Bicep version: HTTPSConnectionPool(host='aka.ms', port=443): Max retries exceeded with url: /BicepLatestRelease (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)).

cli.azure.cli.core.azclierror: Error while attempting to retrieve the latest Bicep version: HTTPSConnectionPool(host='aka.ms', port=443): Max retries exceeded with url: /BicepLatestRelease (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)).
az_command_data_logger: Error while attempting to retrieve the latest Bicep version: HTTPSConnectionPool(host='aka.ms', port=443): Max retries exceeded with url: /BicepLatestRelease (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)).
cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7f603a76c7b8>]
cli.__main__: Command ran in 0.447 seconds (init: 0.089, invoke: 0.358)

Expected behavior
Bicep successfully installs.
Environment summary
Linux-<IP_ADDRESS>-microsoft-standard-WSL2-x86_64-with-debian-bullseye-sid, Ubuntu 20.04.3 LTS
Python 3.6.10
Installer: DEB

azure-cli 2.31.0

Extensions:
ml 2.0.3

Additional context
Like the pervious ticket I am also behind a corporate proxy, other commands that require the proxy certificate like az login are working as expected.  The REQUESTS_CA_BUNDLE environment variable has been set to point to a file including the proxy certificate.  In addition, it doesn't not appear that bicep is obeying the AZURE_CLI_DISABLE_CONNECTION_VERIFICATION environment variable as running the following command export AZURE_CLI_DISABLE_CONNECTION_VERIFICATION=1 before attempting to do the install is having no effect.

route to service team

I have just came into this problem, not with bicep but with another extension (az artifacts).
It seems the az internal pip version has been upgraded to 21.0.1:
/opt/az/bin/python3 -m pip --version
pip 21.0.1 from /opt/az/lib/python3.6/site-packages/pip
and this is causing problems:
https://github.com/pypa/pip/issues/9568
The workaround I did is to install manually the extension in the proper location with a previous pip version.

As a temporary workaround, you can install bicep manually outside the az cli following the instructions here: https://docs.microsoft.com/en-us/azure/azure-resource-manager/bicep/install#install-manually

Also experiencing this in version 2.32.0 behind a corporate proxy.

I have also confirmed that this issue exists in version 2.32.0. I posted additional details in my comment on the previously-closed issue, #19420.
Unfortunately, this is a showstopper that prevents me from adopting bicep in a corporate setting at the moment, which is a shame - the DSL and tooling offered appear to be a great evolution & streamlining of the ARM template approach!

Upon inspecting the previous commit for #19420, the issue is that the bicep module is hardcoded to use the default cacert.pem certificate bundle (which is provided by the certifi python library dependency) when invoking HTTP requests via the requests python library. This approach ignores the REQUESTS_CA_BUNDLE environment variable that many of us are setting to utilize the az cli behind a corporate proxy. I believe these lines need to be adjusted to match how other az cli modules are setting the verify parameter for their calls to requests.get().
@Zerthick and @Jaffacakes82 - in the meantime, I found another workaround that allowed az bicep install to complete successfully for me on a Windows machine:
Append your corporate proxy certificate(s) to the end of the certify library's cacert.pem file. You'll likely find the file at one of the following locations:

On Windows: C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\Lib\site-packages\certifi\cacert.pem
On Linux: /opt/az/lib/python3.6/site-packages/certifi/cacert.pem

Upon inspecting the previous commit for #19420, the issue is that the bicep module is hardcoded to use the default cacert.pem certificate bundle (which is provided by the certifi python library dependency) when invoking HTTP requests via the requests python library. This approach ignores the REQUESTS_CA_BUNDLE environment variable that many of us are setting to utilize the az cli behind a corporate proxy. I believe these lines need to be adjusted to match how other az cli modules are setting the verify parameter for their calls to requests.get().
@Zerthick and @Jaffacakes82 - in the meantime, I found another workaround that allowed az bicep install to complete successfully for me on a Windows machine:
Append your corporate proxy certificate(s) to the end of the certify library's cacert.pem file. You'll likely find the file at one of the following locations:

On Windows: C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\Lib\site-packages\certifi\cacert.pem
On Linux: /opt/az/lib/python3.6/site-packages/certifi/cacert.pem

Thanks for this - worked for me. In case it helps others: I'm on a mac and found the correct cacert.pem location by running az --version which spit out
Python location '/Users/<username>/.brew/Cellar/azure-cli/2.32.0/libexec/bin/python'

and subsequently found the right cacert.pem file here: ~/.brew/Cellar/azure-cli/2.32.0/libexec/lib/python3.10/site-packages/certifi/cacert.pem. Appending my corporate certs onto it allowed me to successfully run az bicep install :+1:

These suggestions work for us and provide a temporary workaround, for anyone looking to run Bicep from a Docker container (such as VS devcontainers) we use the following solution (with the cert fix that @jtrivett mentions)
FROM ubuntu
RUN apt-get update && \
    apt-get install -y --no-install-recommends \
        gnupg && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# Get Azure CLI
# https://docs.microsoft.com/en-us/cli/azure/install-azure-cli-apt?view=azure-cli-latest
RUN wget -q -O - https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor | tee /etc/apt/trusted.gpg.d/microsoft.asc.gpg > /dev/null && \
    echo "deb [arch=amd64] https://packages.microsoft.com/repos/azure-cli/ bionic main" | tee /etc/apt/sources.list.d/azure-cli.list && \
    apt-get update && \
    apt-get install -y --no-install-recommends azure-cli && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# Azure Bicep
RUN curl -Lo bicep https://github.com/Azure/bicep/releases/latest/download/bicep-linux-x64 && \
    chmod +x ./bicep && \
    mv ./bicep /usr/local/bin/bicep && \
    bicep --help

# Hack to fix Bicep cert issue: https://github.com/Azure/azure-cli/issues/20842#issuecomment-1018139759
RUN cp /etc/ssl/certs/<CORPORATE CERT CHAIN>.pem /opt/az/lib/python3.6/site-packages/certifi/cacert.pem

@Zerthick Apologies for the late reply. Could you please let us know if you have been able to test using the above mentioned suggestions ? Please let us know if you are still running into the same issue. Awaiting your reply.

@navba-MSFT The suggestions above do indeed appear to work, though as others on this ticket have commented they are a workaround, bicep should be ideally obeying the REQUESTS_CA_BUNDLE environment variable to handle including corporate certificates and the AZURE_CLI_DISABLE_CONNECTION_VERIFICATION environment variable to disable the ssl check completely.

Thanks @ssvalkyrie, worked a treat. @navba-MSFT, agree with @Zerthick, this needs addressing by Microsoft.

@navba-MSFT Yeah, let us know what is the plan. As mentioned before there is a lot of people behind a corporate firewall.

Adding the Service team to look into this issue and share the update on this.
@josephkwchan, @jennyhunter-msft could you please provide an update on this once you get a chance ? Awaiting your reply.

By the way I'm facing this issue when running: az bicep build

By the way I'm facing this issue when running: az bicep build

I also had the same issue but I noticed there should be a fix (https://github.com/Azure/azure-cli/pull/20183) and my az cli was not up to date
Have you tried az upgrade before doing az bicep build again?

By the way I'm facing this issue when running: az bicep build

I also had the same issue but I noticed there should be a fix (#20183) and my az cli was not up to date
Have you tried az upgrade before doing az bicep build again?

Having the same issue with the latest version of azure cli. So No upgrading does not solve the issue.
"azure-cli": "2.40.0",
  "azure-cli-core": "2.40.0",
  "azure-cli-telemetry": "1.0.8",

Experiencing exactly the same issue, @navba-MSFT  What is the plan for solving this issue?

I do not use the proxy, but have the same issue when trying to run:
az bicep decompile --file main.json
OS: MacOS Ventura 13.5
azure-cli                         2.51.0
core                              2.51.0
telemetry                          1.1.0
Extensions:
azure-devops                      0.26.0
Dependencies:
msal                            1.24.0b1
azure-mgmt-resource             23.1.0b2
Bicep CLI version 0.19.5 (87ca110fc0)

Not sure why this issue was closed, because it's still happening for me in December 2023.


601 - superfluous URL suffices?

I just stumbled upon this error code in the docs, but have no idea what it means.
It doesn't seem to be raised anywhere in the code, but still, how would such an URL look like?

It is actually used as an error code, e.g. when executing this command:
curl -X GET http://<IP_ADDRESS>:8529/_api/document/foo/bar/baz --dump -
HTTP/1.1 400 Bad Request
Server: ArangoDB
Connection: Keep-Alive
Content-Type: application/json; charset=utf-8
Content-Length: 160

{"error":true,"errorMessage":"expecting GET \/_api\/document\/<document-handle> or GET \/_api\/document?collection=<collection-name>","code":400,"errorNum":601}

There are other places that use this error code, too:

arangod/RestHandler/RestDocumentHandler.cpp: retrieving/saving documents
arangod/RestHandler/RestEdgeHandler.cpp: retrieving/saving edges
arangod/RestHandler/RestImportHandler.cpp: importing data
arangod/RestHandler/RestReplicationHandler.cpp: replication control

So it can be interpreted as "extra arguments in request". The word "suffices" confused me, I believe it should be "suffixes" ("satisfies" vs. "additions at the end")

You are right, it should be "suffixes". The use of "suffices" throughout the docs is probably a case of hypercorrection (via "index" -> "indices").


[DX] auto-reload for local webserver

Should we add auto-reloading for webserver by using something like https://github.com/cosmtrek/air ?
Helps speed up development, can share PR if we are aligned on the approach.
@dm161

@sareensumanau yes please - are you able to update the run-local-webserver.sh? and setup the auto-reload without having to install dependencies? I think that tool supports docker. If can be done with docker good to me, feel free to send a PR!

/bounty $10


Add a way to clear localStorage in useLocalStorage

I use useLocalStorage a lot, and it could use a cleanup story in some cases.
Maybe as a third tuple member, have a function to call to clear the item, that respects the side-effects lifetime, as I found that simply calling localStorage.removeItem(key) can create a race condition with the hook recreating the value from memory.
Example:
const Foo = () => {
  const [value, setValue, clearValue] = useLocalStorage('key')
  return (
    <>
      <input type="text" value={value} onChange={e => setValue(e.target.value)} />
      <button onClick={clearValue}>Clear</button>
    </>
  )
}

Here is a demo of the issue I'm having with clearing the local storage: it comes back after a page refresh.
https://codesandbox.io/s/j3xr9k25lv

Shipped in v15.0.0, thanks @suyingtao  🎉


Docker build for Prejournal fails

mkdir test
cd test
git clone  https://github.com/ylebre/prejournal.git
cd prejournal
docker build .

Step 9/10 : RUN composer install
 ---> Running in 290ac668a930
Installing dependencies from lock file (including require-dev)
Verifying lock file contents can be installed on current platform.
Your lock file does not contain a compatible set of packages. Please run composer update.

  Problem 1
    - phpspec/prophecy is locked to version v1.15.0 and an update of this package was not requested.
    - phpspec/prophecy v1.15.0 requires php ^7.2 || ~8.0, <8.2 -> your php version (8.2.2) does not satisfy that requirement.
  Problem 2
    - phpspec/prophecy v1.15.0 requires php ^7.2 || ~8.0, <8.2 -> your php version (8.2.2) does not satisfy that requirement.
    - phpunit/phpunit 9.5.21 requires phpspec/prophecy ^1.12.1 -> satisfiable by phpspec/prophecy[v1.15.0].
    - phpunit/phpunit is locked to version 9.5.21 and an update of this package was not requested.

The command '/bin/sh -c composer install' returned a non-zero code: 2

It seems the locked version of phpspec/prophecy v1.15.0 is not compatible with PHP 8.2

Trying:
rm composer.lock

Next problem:  php-http/discovery contains a Composer plugin which is blocked by your allo w-plugins config.
Trying:
composer config --no-plugins allow-plugins.php-http/discovery

That command didn't work, but adding this does:
itpod /workspace/federated-timesheets-tests/test/prejournal (main) $ git diff
diff --git a/composer.json b/composer.json
index 1ea6d3e5..f50e3137 100644
--- a/composer.json
+++ b/composer.json
@@ -30,6 +30,11 @@
     "require-dev": {
         "phpunit/phpunit": "^9"
     },
+    "config": {
+       "allow-plugins": {
+                  "php-http/discovery": true
+       }
+    },
     "scripts": {
         "lint": "./vendor/bin/php-cs-fixer fix .",
         "test": "PRE[...]

I'll create a PR for this


Start new chat messages at the top of the screen?

Thank you for this awesome library!
I wanted to ask you if there is a way to start the first few messages of a chat, where the whole screen isn't yet filled, from the top of the screen and not from the bottom as in the current chat widget implementation.
If it's not yet possible I'd be happy to look into it and create a PR when I have some time.

Hi! this is not the issue with this package, but with the flutter chat UI. Unfortunately, due to architecture of the current implementation it will be impossible to do it, however I am very slowly working on greatly improved v2 where this will be a default behaviour. Have practically no time with my current workload since start of the summer though, waiting for some time I could get to finish this.
https://github.com/flyerhq/flutter_firebase_chat_core/assets/14123304/454f3cb2-e7f5-477c-a4a0-a2b9c86f8558

You are right about the UI, thank you for clarifying. Looking forward the v2 implementation, it looks great! Thank you for all your hard work @demchenkoalex!


Adds python3.6 to travis test matrix

That got done as part of the 0.2 release I just made, just by accident :-) Thanks for the PR, sorry for not maintaining this package for years...


Image builder create doesn't support identity configuration

The section of "identity" in the image template is ignored.
Wonder is this a bug or will be supported later?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: f8e24b37-83e6-abcb-65f3-8e10aca64fa5
Version Independent ID: 803fa652-41f2-dcdd-ed5b-2e60b11186af
Content: az image builder
Content Source: latest/docs-ref-autogen/image/builder.yml
Service: virtual-machines
GitHub Login: @rloutlaw
Microsoft Alias: routlaw

@qwordy please help take a look.

It is not supported in CLI yet. I am working on it.
One question, is the value of dictionary always "{}"?
"identity": {
        "type": "UserAssigned",
        "userAssignedIdentities": {
          "/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/rg1/providers/Microsoft.ManagedIdentity/userAssignedIdentities/identity_1": {}
        }
      },

When configuring user assigned identity, the value of the key-value pair is empty. Only the key is filled with identity resource id. But when service returns the provisioned resource, the value should not be empty (readonly).


There are 3 sentences in the language file that cannot be translated.

What is your feature request?

What type of change is this?

[ ] Visual
[ ] Features/Mechanics
[ ] Performance
[ ] Compat
[X] Others

Hi!
This is because these are modded attributes, and I attempt to discover existing translations.
AdditionalEntityAttributes lacks any translations for MAGIC_PROTECTION in this language.
https://github.com/DaFuqs/AdditionalEntityAttributes/blob/1.20.1/src/main/resources/assets/additionalentityattributes/lang/en_us.json
Ranged Weapon API adds "Draw Speed" (HASTE) and "Ranged Damage", but lacks translations in this language.
https://github.com/FabricExtras/RangedWeaponAPI/tree/1.20.1/src/main/resources/assets/ranged_weapon/lang
I will provide you the specific translation keys to target so you can open respective PR's.
ADDITIONAL ENTITY ATTRIBUTES
`attribute.name.additionalentityattributes.generic.magic_protection`

RANGED WEAPON API
  `attribute.name.ranged_weapon.damage`
  `attribute.name.ranged_weapon.haste`


Remove 'priv' and 'const' statements

This PR is more minimalist than the previous PR. It:

Merges 'priv' and 'const' statements into 'let' statements.
Removes the 'priv' modifier (it remains the default modifier - no need to explicitly state priv)

It is worth noting there should be no loss of functionality with the removal of const statements since users can already specify const types. The following is valid:
let x: const Field = 2;

This PR addresses issue #143


Sidecar Middleware Mode

The middleware mode of the sidecar will involve connecting to multiple nodes. The sidecar will be responsible for establishing connections to the nodes and handle the complexity of interacting with the network.

Closed as completed.


[Bug Report]: wrong parameter name in the readme.md of the Microsoft.Compute/virtualMachines

Describe the bug
wrong parameter name in the readme.md of the Microsoft.Compute/virtualMachines
To reproduce
subnetID: should be changed to subnetResourceId
Code snippet
"nicConfigurations": {
  "value": [
    {
      "nicSuffix": "-nic-01",
      "deleteOption": "Delete", // Optional. Can be 'Delete' or 'Detach'
      "ipConfigurations": [
        {
          "name": "ipconfig1",
          "subnetId": "/subscriptions/<subscriptionId>/resourceGroups/<resourceGroupName>/providers/Microsoft.Network/virtualNetworks/<vNetName>/subnets/<subnetName>",
          "pipConfiguration": {
            "publicIpNameSuffix": "-pip-01",
            "roleAssignments": [
              {
                "roleDefinitionIdOrName": "Reader",
                "principalIds": [
                  "<principalId>"
                ]
              }
            ]
          }

Relevant log output
az deployment sub create -c --location westeurope --template-file deploy.bicep --parameters @(".test/parameters.json")          in pwsh at 16:02:16
C:\Users\olivergulich\GitHub\CARML-ChrisS\ResourceModules\modules\Microsoft.Compute\virtualMachines\deploy.bicep(446,9) : Warning use-resource-id-functions: If property "id" represents a resource ID, it must use a symbolic resource reference, be a parameter or start with one of these functions: extensionResourceId, guid, if, reference, resourceId, subscription, subscriptionResourceId, tenantResourceId. [https://aka.ms/bicep/linter/use-resource-id-functions]

Could you please assign this issue to me and I will fix it?

Hey @oliverlabs , just did. Thank you very much.

Submitted a pull request #2120 @MrMCake


Does not pull through styling

I'm converting a super light 1 page WP site with Elementor but none of the styling is pulling through and some images missing see https://konigdigital.com/dev/ tried Zip and S3 same results, please help?

#209 is a related issue.
Have you had any luck figuring it out yet?

This should be a non-issue under version 7 (latest master branch from this repo). Within version 6.x, I'd try installing the Autoptimize plugin, which can help consolidate assets, allowing for easier export. Tends to help in these kind of cases.


Violation of changeset design principals?

Isn't the concept of a changeset that it's in preparation for a change?  IE, you create the changeset then later you apply it.. it seems by suggesting users add prepare_version to their models changeset method that you're now making calling changeset have side-effects (writing to the DB), where-as before (intentionally) it had none.
Could be I'm misunderstanding the concept, but I don't think so.  I feel like we need yet another function to encapsulate the concept of versioning without piggy-backing on changeset itself.

I feel like the right injection point might be a new Repo module or wrapper function around the repo itself vs the changeset...
changeset = Model.changeset(data)
VersioningRepo.insert!(changeset)

or
changeset = Model.changeset(data)
TrackChanges.insert!(changeset, MyApp.Repo)

Just some thoughts.

My original attempt was to override the insert and delete actions on the repo. José’s feedback was to use the changeset, so that is what I did.
I don’t think there is anything stopping someone one doing at the repo level. Just don’t use the prepare_version api. The basic APIs should be available to do the inserts at the repo level.
Steve

On Jul 23, 2016, at 3:04 PM, Josh Goebel<EMAIL_ADDRESS>wrote:
I feel like the right injection point might be a new Repo module or wrapper function around the repo itself vs the changeset...
changeset = Model.changeset(data)
VersioningRepo.insert!(changeset)
or
changeset = Model.changeset(data)
TrackChanges.insert!(changeset, MyApp.Repo)
Just some thoughts.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub https://github.com/smpallen99/whatwasit/issues/4#issuecomment-234734569, or mute the thread https://github.com/notifications/unsubscribe-auth/AA2YpeWXMHGTptWMF8qUG-3sCUvhAgHIks5qYmXJgaJpZM4JTa52.

I pinged him on Twitter with my question.  It just seems like you're making (suggesting) a change to changeset() that entirely alters the behavior of it.
I'm very curious to see the "right" way to do this in Elixir (it's the exact reason i came to read this code in the first place), but this doesn't feel like it.  And imagine we start to have a bunch of other plugins that do things with side effects (send an email, text, etc)... is the recommendation really to put them ALL inside changeset?

Also for most systems shouldn't we really include a transaction in the examples?  What about something like:
Repo.transaction fn ->
  changeset = Model.changeset(model, new_data)
  |> Model.save_prior_version!
  |> Repo.insert!
end

@yyyc514 Actually, there are no side effects in the changeset. prepare_versions uses the Changeset.prepare_changes api. It is passed an ann fun that gets called during the Repo action. If the Repo action fails, the prepare_changes function does not get called. So, there is no need for a transaction.
If you read the docs for prepare_changes you'll see that it is designed for this type of post processing.

So to clarify: changesets are broken in two stages. The casting, validation, preparing as aspect which is "side-effect" free. Then, once you send it to the repository, all side-effects happen. This includes inserting to the database, running prepare_changes, and setting up transactions (if prepare changes or associations are included). So I wouldn't say changesets are side-effects free but rather they split the stages with side-effects and without side-effects in two very clear steps. The transaction is automatically taken care of if you have associations or set prepare_changes.

Changeset.prepare_changes  Ah that is what I was somehow missing.  Thank you!  Now it's all so very clear. :-)  Closing issue.

@smpallen99 Sorry for the confusion. :)

Josh,
No problem. It was a good discussion. I appreciate José’s description of the two stages of a changeset.
Steve

On Jul 24, 2016, at 12:32 PM, Josh Goebel<EMAIL_ADDRESS>wrote:
@smpallen99 https://github.com/smpallen99 Sorry for the confusion. :)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/smpallen99/whatwasit/issues/4#issuecomment-234786933, or mute the thread https://github.com/notifications/unsubscribe-auth/AA2YpTEX044AvKcdi-dBxIaAZtmYi8ckks5qY5OIgaJpZM4JTa52.


Indexer runs twice on container start

Version affected
6.2.12
Description
If environment variable CREATE_PLAIN_TABLES=1, then indexer runs twice on container start, because ENTRYPOINT script docker-entrypoint.sh calls itself at line 168:
      exec gosu manticore "$0" "$@"

Please check and fix.

First download the configuration file from the archive - archive.conf.zip
Steps to reproduce
docker run --rm -e CREATE_PLAIN_TABLES=1 --name manticore -v $(pwd)/manticore.conf:/etc/manticoresearch/manticore.conf manticoresearch/manticore

Logs
Manticore 6.2.12 dc5144d35@230822
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-2023, Manticore Software LTD (https://manticoresearch.com)

using config file '/etc/manticoresearch/manticore.conf'...
indexing table 'idx'...
collected 1 docs, 0.0 MB
creating lookup: 0.0 Kdocs, 100.0% done
sorted 0.0 Mhits, 100.0% done
total 1 docs, 6 bytes
total 0.068 sec, 87 bytes/sec, 14.62 docs/sec
total 3 reads, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg
total 15 writes, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg
Manticore 6.2.12 dc5144d35@230822
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-2023, Manticore Software LTD (https://manticoresearch.com)

using config file '/etc/manticoresearch/manticore.conf'...
indexing table 'idx'...
collected 1 docs, 0.0 MB
creating lookup: 0.0 Kdocs, 100.0% done
sorted 0.0 Mhits, 100.0% done
total 1 docs, 6 bytes
total 0.053 sec, 112 bytes/sec, 18.68 docs/sec
total 3 reads, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg
total 15 writes, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg
[Tue Sep 19 07:57:32.293 2023] [1] using config file '/etc/manticoresearch/manticore.conf' (280 chars)...
[Tue Sep 19 07:57:32.294 2023] [1] WARNING: secondary_indexes set but failed to initialize secondary library: (null)
starting daemon version '6.2.12 dc5144d35@230822' ...
listening on all interfaces for mysql, port=9306
Manticore 6.2.12 dc5144d35@230822
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-2023, Manticore Software LTD (https://manticoresearch.com)

precaching table 'idx'
WARNING: secondary library not loaded; secondary index(es) disabled
prereading 1 tables
preread 1 tables in 0.000 sec
WARNING: [BUDDY] no SPHINX or HTTP listeners found, disabled
accepting connections
pavelshilin@Pavels-MacBook-Air ~ %```

@pwoszczyk Hi, if you want to fix this, please let me know.

I will try.


Accuracy counter

Hi,
I'm using a mix of HoxHUD, PocoHUD and GoonMod. Since the last update, the accuracy counter (tab view) doesn't display correctly and is always at 0%. It occurs only with people who have poco/goon. HoxHUD only users doesn't have this, and yet Goon is the only mod which got updated recently.
(And there's a bug, maybe from incompatibility, that makes the enemy damage counter from HoxHUD renders always 0.0 HP, but I guess that's none of your buisness)

This should be fixed in the next update.

Thanks for the reply. Will the HoxHUD enemy counter get repaired too, or it's not goonmod related?

I haven't tried with HoxHud, but the change will probably fix the issue. If it doesn't, let me know and I'll take a more indepth look tomorrow.

Okay, I'll wait for the update and leave a reply here!

Everything seems fine now, good job! ^^


Add omit and pick Handlebars helpers and revert some changes in weapon sheet

Yeah I think this solves a large number of selectOptions incompatibilities!


SAK-49028 Fixing classes on Private Messages

Fixing classes on Private Messages
https://sakaiproject.atlassian.net/browse/SAK-49028

please fill out the JIRA properly

Just filled! :)


Tags in the notifications and events

Hello everyone, this is an amazing project, I build my own version twice but it does not compare to this! Thank you so much!
I really liked the tags feature and i can hope for a tree like grouping in the future, but for now
it will help if tags are included in the notification message and in the event table (on the dashboard)
Something like this

The notification is from the Rocket.chat application

@panos-stavrianos Is this one of the issues which will be resolved by #2693?

That PR would be absolutely perfect!


taking too much time in cleaning

how we can reduce the time?

If you are currently running on a cpu, you can use the gpu
Try to use crop strategy,


Add support for devices with "service create"

Initially reported: https://github.com/docker/docker/issues/24865, but I realized it actually belongs here. Feel free to close the other one if you want. Content of the original issue copied below.
Related: #1030
Currently, it's not possible to add devices with docker service create, there is no equivalent for docker run --device=/dev/foo.
I'm an author of nvidia-docker with @3XX0 and we need to add devices files (the GPUs) and volumes to the starting containers in order to enable GPU apps as services.
See the discussion here: https://github.com/docker/docker/issues/23917#issuecomment-233670078 (summarized below).
We figured out how to add a volume provided by a volume plugin:
$ docker service create --mount type=volume,source=nvidia_driver_367.35,target=/usr/local/nvidia,volume-driver=nvidia-docker [...]

But there is no solution for devices, @cpuguy83 and @justincormack suggested using --mount type=bind. But it doesn't seem to work, it's probably like doing a mknod but without the proper device cgroup whitelisting.
$ docker service create --mount type=bind,source=/dev/nvidiactl,target=/dev/nvidiactl ubuntu:14.04 sh -c 'echo foo > /dev/nvidiactl'
$ docker logs stupefied_kilby.1.2445ld28x6ooo0rjns26ezsfg
sh: 1: cannot create /dev/nvidiactl: Operation not permitted

It's probably equivalent to this:
$ docker run -ti ubuntu:14.04
root@76d4bb08b07c:/# mknod -m 666 /dev/nvidiactl c 195 255
root@76d4bb08b07c:/# echo foo > /dev/nvidiactl
bash: /dev/nvidiactl: Operation not permitted

Whereas the following works (invalid arg is normal, but no permission error):
$ docker run -ti --device /dev/nvidiactl ubuntu:14.04
root@ea53a1b96226:/# echo foo > /dev/nvidiactl
bash: echo: write error: Invalid argument

@flx42 For the container runtime, devices require special handling (a mknod syscall), so mounts won't work. We'll probably have to add some sort of support for this. (cc @crosbymichael)
Ideally, we'd like to be able to schedule over devices, as well.

@stevvooe Already have device support in the runtime, just not exposed in swarm.

Ideally, we'd like to be able to schedule over devices, as well.

This question was raised here: https://github.com/docker/docker/issues/24750
But the discussion was redirected here: https://github.com/docker/docker/issues/23917, in order to have a single discussion thread.

@stevvooe I quickly hacked a solution, it's not too difficult:
https://github.com/flx42/swarmkit/commit/a82b9fb2b1f3387baa1e4d4447ba9af4f3e05f16
This is not a PR yet, would you be interested if I do one? Or are the swarmkit features frozen right now before 1.12?
The next step would be to also modify the engine API.

Forgot to mention that I can now run GPU containers by mimicking what nvidia-docker does:
./bin/swarmctl service create --device /dev/nvidia-uvm --device /dev/nvidiactl --device /dev/nvidia0 --bind /var/lib/nvidia-docker/volumes/nvidia_driver/367.35:/usr/local/nvidia --image nvidia/digits:4.0 --name digits

@flx42 I took a quick peak and the PR looks like a decent start. I am not sure about representing these as cluster-level resources for container startup. From an orchestration perspective, we have to match these up with announced resources at the node level, which might be okay. It might be better on ContainerSpec, but I'm not sure yet.
Go ahead and file as a [WIP] PR.

@stevvooe Yeah, that's the biggest discussion point for sure.
In engine-api, devices are resources:
https://github.com/docker/engine-api/blob/master/types/container/host_config.go#L249
But in swarmkit, resources are so far "fungible" objects like CPU shares and memory, with a base value and a limit. A device doesn't really fit that definition. For GPU apps we have devices that must be shared (/dev/nvidiactl) and devices that could be exclusively acquired (like /dev/nvidia0).
I decided to initially put devices into resources because there is already a function in swarmkit that creates a engine-api Resource object from a swarm Resource object:
https://github.com/docker/swarmkit/blob/master/agent/exec/container/container.go#L301-L324
This method would also need to access the container spec.
I will file a PR soon to continue the discussion.

@flx42 Great!
We really aren't planning on following the same resource model from HostConfig for SwarmKit. In this case, we are instructing the container to mount these devices, which is specific to a container runtime. Other runtimes may not have a container or devices. Thus, I would err on ContainerSpec.
Now, I would like to see scheduling of fungible GPUs but that might a wholly separate flow, keeping the initial support narrow. Such services would require manual constraint and device assignment, but you still achieve the goal.
Let's discuss this in the context of the PR.

Thanks @flx42 - I think GPU is definitly something we want to support medium term.
/cc @mgoelzer

Thanks @aluzzardi, PR created, it's quite basic.

The --device option is really import for my use case too. I am trying to use swarm to manage 50 Raspberry Pi's to do computer vision, but I need to be able to access /dev/video0 to capture images. Without this option, I'm stuck, and have to manage them without swarm, which is painful.

@mlhales We need someone who is willing to workout the issues with --device in a clustered environment and support that solution, rather than just a drive by PR. If you or a colleague want to take this on, that would be great, but this isn't as simple as adding --device.

Using --device=/dev/gpiomem would be great on a RPi swarm to access GPIO on each node without privileged mode.

Using --device=/dev/fuse would be great for mounting FUSE, which isn't currently possible.

We found an easier way for Blinkt! LED strip to use sysfs. Now we can run Blinkt! in docker swarm mode without privileges.

@StefanScherer is it a proper alternative for using e.g. --device=/dev/mem to access GPIO on a RPi ? Would love to see an example if you would care to share :)

@mathiasimmer For the use-case with Blinkt! LED strip there are only eight RGB LED's. So using sysfs it not time critical for these few LED's. If you want to drive hundreds of them you still need faster GPIO access to have a higher clock rate. But for Blinkt! we have forked the Node.js module and adjusted in in this branch https://github.com/sealsystems/node-blinkt/tree/sysfs.
A sample application can be found as well and how to use this forked module as dependency in an own package.json.

/cc @cyli

@aluzzardi I think we should resurrect the --device patch. I don't think there is anything in the pipeline that is sophisticated enough to handle proper, cluster-level dynamic resource allocation. Looking back at this issue, there isn't necessarily a model that will work well in all cases (mostly because no one here can seem to enumerate them).
We can always add logic in the scheduler to prevent device contention in the future.

Attempt to add devices to the container spec and plugin spec here:  https://github.com/docker/swarmkit/pull/1964
I've no objection to the --device flag - cc @diogomonica ?

--device allows any service to escalate privileges. Why would we add this w/out profiles on services?

@diogomonica I thought profiles mainly covered capabilities, etc?

@cyli well, if we believe "devices" are easy enough to understand for easy user acceptance then we might not need them, but we should look critically at adding anything that allows escalation of privileges of a container to the cmd-line before we have agood way of informing everything the service will need from a security perspective to the user.

Also following this. Very interested in access to character devices (/dev/bus/usb/...) in a docker swarm.

will be great to add --devices in swarm service

@flx42 , Can you let us know, if your patch is available for latest docker swarm, i.e. if someone has ported your patch to the latest, docker swarm API 1.24+, where swarmkit is integrated within docker daemon.

@sudharkrish No, it isn't ported AFAIK.

Wondering about the current state as well.. I was trying to set up a simple at home swarm environment (so I could manage with a simple yaml file and a docker stack deploy) and was dissapointed to find --device was missing from swarm mode, keeping me from being able to mount my raspberry pi camera via swarm.

Adding my use case, my company is deploying IoT sensors, and without support for --device equivalent swarm mode can't be used

The year was 2018 and people had been waiting to use Swarm Mode in the IoT space for two years. The GitHub issue was dusty and neglected. Confidence and hope faded into frustration and despair. Hundreds wept as they logged into their Azure Device Management dashboard. Big tech had won. The cloud vendor handcuffs clicked - laughing - as they closed around our product wrists.
In the distance a fumbling mob of sales people chanted hyperbolic mantras. A focused and wary engineer can pick-out only one common word, though no two would spell it the same way: Kubernetes. The marketing hellscape is leaking out of the large-scale service software space and laid siege to our precious little corner previously too small for slick enterprise sales people.
A weak but proud voice cries out:

No. We won't go there. Not with the insecure platform composition model, configuration nightmare, dependencies on cloud-focused configuration management tooling, and ridiculous disk and memory footprint. We will not build a dependency on a centralized architecture in our distributed environment. We will not bring our own secret vault backend. We will not build our own internal certificate authorities. If we wanted to build our own platform we'd have just done that. We'd rather just hack something and abandon the whole product space. Glue it together with shell scripts and docker run -dev /dev/video0 my-video-forwarder. Maybe sprinkle in a few SSH tunnels. It would get the job done and be both more simple and maintainable.

What they really wanted was a to use Swarm Mode. To take advantage of its secure foundation, distributed nature, and simplified encrypted networking model. They'd love to take only this single dependency. Maybe there was hope yet for Swarm Mode in the IoT, but they were not going to wait for it. Not forever.

Contributions are welcome.
On Mon, 4 Jun 2018, 07:17 Jeff Nickoloff<EMAIL_ADDRESS>wrote:

The year was 2018 and people had been waiting to use Swarm Mode in the IoT
space for two years. The GitHub issue was dusty and neglected. Confidence
and hope faded into frustration and despair. Hundreds wept as they logged
into their Azure Device Management dashboard. Big tech had won. The cloud
vendor handcuffs clicked - laughing - as they closed around our product
wrists.
In the distance a fumbling mob of sales people chanted hyperbolic mantras.
A focused and wary engineer can pick-out only one common word, though no
two would spell it the same way: Kubernetes. The marketing hellscape is
leaking out of the large-scale service software space and laid siege to our
precious little corner previously too small for slick enterprise sales
people.
A weak but proud voice cries out:
No. We won't go there. Not with the insecure platform composition model,
configuration nightmare, dependencies on cloud-focused configuration
management tooling, and ridiculous disk and memory footprint. We will not
build a dependency on a centralized architecture in our distributed
environment. We will not bring our own secret vault backend. We will not
build our own internal certificate authorities. If we wanted to build our
own platform we'd have just done that. We'd rather just hack something and
abandon the whole product space. Glue it together with shell scripts and docker
run -dev /dev/video0 my-video-forwarder. Maybe sprinkle in a few SSH
tunnels. It would get the job done and be both more simple and maintainable.
What they really wanted was a to use Swarm Mode. To take advantage of its
secure foundation, distributed nature, and simplified encrypted networking
model. They'd love to take only this single dependency. Maybe there was
hope yet for Swarm Mode in the IoT, but they were not going to wait for it.
Not forever.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/docker/swarmkit/issues/1244#issuecomment-394245542,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAdcPFqP3FbCWwpYKvVsl5JQGhLtL9gdks5t5ND0gaJpZM4JVo5I
.

I know it's a poor substitute but could something like doing docker run things inside a service task work?:
services:
  iot-ftw:
    image: docker
    command: "docker run --device xxxx <image>"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    deploy:
      mode: global

or:
https://blog.viktoradam.net/2018/05/14/podlike/

@BretFisher Sure, that's how I'm addressing this "missing feature" in Swarm. It's a workaround or someone would call it a hack, but it works great. Can even run easily as a Global Service. 😇

Running a container that has full access to Docker instance is at very least dangerous.

Normally I would agree for app services. The only thing in that container with daemon access is the docker image (in my 1st example). The iot app wouldn't have anything elevated. I don't see an increased risk here. You're trusting docker to be safe with docker.

Just to be clear, I think a workaround like @BretFisher mentioned is a perfectly acceptable temporary solution. Especially so for global service use-cases where all worker nodes have the same requisite device attached. @nazar-pc I'm less concerned about the privilege here as the service itself is not processing user-provided input. But maybe that is naive.
@justincormack I think maybe some of the frustration is the lack of clear direction. There was at least some discussion happening in this thread right up until the end of February 2017. But that left the direction ambiguous. Pairing that will the time and research investment it takes to contribute to any Docker project at this depth and whatever effort people put in on this would almost certainly have to be funded. Perhaps I can convince one of my clients to make the investment. I'm just surprised that Docker hasn't funded this issue yet (especially considering the potential security impact).
@diogomonica Can you elaborate on what you think should be put in place re: your last comment? We're not actually preventing escalation risk by not offering devices in services. People aren't skipping the feature, they're just not using services or their using a work around. I'm not advocating for blind addition of the feature without consideration for security. But I think we need some informed vision to get started.

There are a lot of desired features from Swarmkit. We don't have the peoplepower right now to chase down very many of them, and it's difficult to say from here what is important. I think the least contribution we'd need to work on it would be a design proposal.
Someone has suggested out-of-band that perhaps we provide a power-user field allowing the specification of any flags that Docker supports (including --device), with the understanding that using these flags means your tasks might do weird things or fail in weird ways unless you know what you're doing. This is an ugly and dirty solution which I am not a fan of, but it may be What We Have To Do in order to just make thing work.
A more elegant solution would be building out a system for noting what resources (devices) a node has available, and making scheduling decisions in swarmkit in order to put tasks where resources are available for them. This would be more complicated to build, but would end up likely being easier to use. In fact, some old proto messages that never got implemented (GenericResource) hint at that this design was being pursued as some point, but was abandoned, likely due to available human time constraints.
We've been working on swarmkit a lot, but the velocity has definitely slowed down since its release. Work has been focused on stabilizing the software, and feature development is much slower because there are fewer people working on it right now.
WDYT?

I would prefer to have something that allow me to use swarm kit and manage my devices than not, anyway I am doing that atm with or without swarm. Currently I can't use swarm for my company use case because the lack of support for --device.
I think that at the current state, swarm is missing a perfect use case for IoT where I/O from devices is a must, leaving this use case open for other alternatives which has much less fit for that in any other aspect, solely by not allowing to connecting to devices.
Bottom line, your suggestion of having something working for power users is much better than having nothing.

The biggest drawback of power-user flag-pass-through is that those features might interact with swarm in unpredictable ways, and cause really esoteric issues. I'm worried that doing it too haphazardly may increase the support burden.

@dperny maybe enable this with some kind of user marking that says i-know-what-i-am-doing like unsafe block or similar?

It really isn’t Swarm’s job to check that the root user telling it what to
do is a “power user.” And I do think modeling devices, advertising those
devices and modeling dependencies on devices would be great. You could
really do powerful things with this.

To me, this is a vital issue for IoT. We can't build the things we need to build without having access to devices. It's further frustrated by the fact that there doesn't exist a way (that I know of) to manage --data-path-addr flags outside of docker swarm. Otherwise, docker-compose could be a simple solution to the issue, at least to manage services deployed on a single device.
Right now, it's a catch 22. I can either manage my data path (so that I can send data through my connected cellular device) but I can't access my devices (like the camera), or vice-versa.
If anyone has a workaround I would greatly appreciate it.
Being able to do all of that through docker swarm would be much better.

@cnrmck, the workaround above from @bretfisher is one option - https://github.com/docker/swarmkit/issues/1244#issuecomment-394343097
And another option from @brubbel is another... https://github.com/docker/swarmkit/issues/1244#issuecomment-285935430

@Cinderhaze Thank you so much for summarizing that. Perhaps your comment should be pinned so that other people can find it. I'll try @BretFisher's solution.

Hey, it's been a while, but y'all should take a look at #2682 that i just opened, which is a proposal for device support in swarm. Tell me what you think.

Any changes yet?
And I really need --device for map Intel VPU to docker container.

 Access to /dev/net/tun is easy enough to schedule across a cluster.

 Access to /dev/net/tun is easy enough to schedule across a cluster.

quote this

This would be probably also useful for exposing Bluetooth to a smart home manager running in swarm

In the vein of not-too-ugly workarounds, see also https://docs.nuvla.io/nuvla/advanced-usage/compose-options.html

no offence @thandal but that's a really ugly workaround 😂 . It unnecessarily exposes the docker socket to a container. I'm not comfortable doing that 😬

I'm a bit biased here 😛 but exposing the docker socket is actually safe in many cases, and in fact many mainstream de facto tools do use it (cadvisor, traefik, etc.). the thing is that it can be dangerous...so it has acquired a bad reputation over time, even if people don't really understand how it can be dangerous. In the example posted by @thandal , I'd agree that is not the preferred solution, but when it comes to security, it all boils down to the nature of the container you are deploying. The one I wrote in the Nuvla docs complies with the following:

it comes from a trusted entity
it is not exposing the docker socket outside the container
it is not making the said container (which is using using the docker socket) visible nor reachable from outside the host

So in this regard, it is safe.
Now, obviously, if you're building a web application, in a multi-tenant infrastructure, then yes, I'd agree with @allfro and you should avoid exposing it.

Maybe stop begging someone else to write features you need?
That is why there is exactly one person working on this repo... in their spare time.

@cpuguy83 isn't swarmkit developed by Docker corp which is also commercially sold as part of Docker EE?

@allfro No. Docker sold off the EE stuff to Mirantis... but even before then Swarmkit had very little support.

The docs say to use:
deploy:
  resources:
    reservations:
      devices:
        - driver: nvidia
          count: 1
          capabilities: [gpu]

But unsurprisingly this is broken in swarm.
Time to move to k8s.

I know this is a 6 year old issue, but is there actually an open PR for this that just needs a bit of attention? Maybe I could help finish the code required to support this? 🤔

Solution does not work for Debian11 Bullseye.
Is there a new workaround for this?
https://github.com/docker/swarmkit/issues/1244#issuecomment-285935430

Hey, @allfro
Have you found a solution? I need the exact same usage like you (for tun device). Did you switch to another solution or have you figured out a workaround?

I really like workaround from @BretFisher https://github.com/moby/swarmkit/issues/1244#issuecomment-394343097 and here is how I adapted it for nodes that require a device:

connect that special container to stack network to make it behave like part of the stack (shutting down, redeploying). Otherwise it will keep running forever and require manual removal
extract that service to separate docker-compose.yml and save it on each node that requires a device. Run command will be less messy

Putting it all together, your services will have to change from this:
services:
  my-service-starter:
    image: docker
    command: 'docker run --name <name> --device /dev/bus/usb -e TOKEN=1234 -p 5000:5000 <image>'
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    deploy:
      placement:
        constraints:
          - node.labels.device_required == true

to this:
services:
  my-service-handler:
    image: docker
    command: 'docker-compose -f /docker-compose.yml up'
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /home/ubuntu/docker-compose.yml:/docker-compose.yml
    deploy:
      placement:
        constraints:
          - node.labels.device_required == true

networks:
  default:
    name: my_network
    driver: overlay
    attachable: true

(on manager)
and
services:
  <name>:
    image: <image>
    restart: always
    container_name: <name>
    devices:
      - /dev/bus/usb
    environment:
      - TOKEN=1234
    ports:
      - 5000:5000

networks:
  default:
    name: my_network
    external: true

(/home/ubuntu/docker-compose.yml on nodes that require a device)

Hi @pjalusic ,
services:
  <name>:
    image: <image>
    restart: always
    container_name: <name>
    devices:
      - /dev/bus/usb
    environment:
      - TOKEN=1234
    ports:
      - 5000:5000

networks:
  default:
    name: my_network
    external: true

But in the worker node, I need to depend on another servcie from manager node.
Could you give me a example for the docker-compose.yml of worker node to add "depends_on" tag.

I developed a plugin in the end that allows me to map devices to containers: https://github.com/allfro/device-volume-driver. Hope it helps others. Unfortunately, it only works on systems that use cgroup v1 (alpine). I am looking for some help to develop the cgroupv2 support into the plugin. It works really well and I've used it to containerize x11 desktops that require access to fuse and the vmware graphics devices.

After planning to redo my home server setup with swarm (so I can have multiple nodes), I discovered that this wasn't supported, and I needed it for VAAPI.
After looking through things, it seemed to me like this was a plumbing (and developer-hour) problem. Basing things on a previous PR series which added ulimit support to swarm, here is a chain of PRs which add devices in the most boring way; just plumbing it through the API as-is, no special management or API. Just what docker already supported outside of swarm.

https://github.com/moby/swarmkit/pull/3106
https://github.com/moby/moby/pull/44695
https://github.com/docker/cli/pull/3930

I'm sure I've missed something, and I don't quite know how to get everything building together to test this (I typically run things from my package manager's installed docker), but maybe someone is willing to try the above out.

Hi, I'm trying the workaround in https://github.com/moby/swarmkit/issues/1244#issuecomment-1178706059 and it indeed works, but when I remove the stack the handler is successfully removed, while the privileged container in docker-compose.yml continue running and has to be killed manually using docker kill.
Any idea on what could be the issue?

The docs say to use:
deploy:
  resources:
    reservations:
      devices:
        - driver: nvidia
          count: 1
          capabilities: [gpu]

But unsurprisingly this is broken in swarm.
Time to move to k8s.

These docs are referencing the API for docker compose, not swarm (services or stacks). The correct (and functioning) API for a stack is:
services:
  my-gpu-service:
    ...
    deploy:
      resources:
        reservations:
          generic_resources:
            - discrete_resource_spec:
              kind: "NVIDIA-GPU"
              value: 2

This works if you've registered your GPUs in the /etc/docker/daemon.json file.
For anyone looking for device support for NVIDIA GPUs using Swarm I did a quick write up here summarizing two solutions. My write up was heavily inspired by the original gist I found on the subject here.

Hello, Thanks for the tip. Anyway in my case it doesn't work. After having investigated step by step, I've realized that I don't have a /sys/fs/cgroup/devices folder on any of my swarm nodes. Does anyone have an idea of where this can come from ?
sudo ls /sys/fs/cgroup
cgroup.controllers      cgroup.stat             cpuset.cpus.effective  dev-mqueue.mount  io.pressure       memory.stat                    sys-kernel-debug.mount
cgroup.max.depth        cgroup.subtree_control  cpuset.mems.effective  init.scope        io.stat           -.mount                        sys-kernel-tracing.mount
cgroup.max.descendants  cgroup.threads          cpu.stat               io.cost.model     memory.numa_stat  sys-fs-fuse-connections.mount  system.slice
cgroup.procs            cpu.pressure            dev-hugepages.mount    io.cost.qos       memory.pressure   sys-kernel-config.mount        user.slice

Also in this same situation. I was using this solution to passthrough the iGPU driver to PLEX on a dockerswarm host for hardware transcoding: https://pastebin.com/XY7GP18T
I had some new hardware which required running the latest version of ubuntu to recognise it but this uses the cgroups v2. At the moment I reverted back to using cgroups v1 via these instructions to get this working again: https://sleeplessbeastie.eu/2021/09/10/how-to-enable-control-group-v2/
Key bit:
$ sudo sed -i -e 's/^GRUB_CMDLINE_LINUX=""/GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=1"/' /etc/default/grub $ sudo update-grub $ sudo reboot
I will experiment with moving to cgroups v2 and a combination of generic resource advertising the iGPU to the service as soon as I have time via these two hints as outlined by @coltonbh :
https://gist.github.com/coltonbh/374c415517dbeb4a6aa92f462b9eb287
https://docs.docker.com/compose/gpu-support/#enabling-gpu-access-to-service-containers
If anyone has any idea how to correctly advertise a quicksync driver to a cgroup v2 using dockerswarm it would be highly appreciated. Alternatively, I guess I could migrate to kubernetes ;)

I've written this hack and tried it with plex and it seems to work: https://github.com/allfro/device-mapping-manager. Essentially it runs a privileged container which listens for docker create events and inspects the mount points. If a mount is within the /dev folder, it will walk the mount path for character and block devices and apply the necessary device rules to make the devices available. This doesn't work with fuse yet because the default apparmor profile blocks mounts (ugh!) but it does work with graphics cards and other devices that don't require operations that are blocked by Docker's apparmor profile. It is inspired by the previous comments.

If anyone has any idea how to correctly advertise a quicksync driver to a cgroup v2 using dockerswarm it would be highly appreciated. Alternatively, I guess I could migrate to kubernetes ;)

Check this out: https://github.com/allfro/device-mapping-manager

phenomenal -- 6years passed and yet i still have to have external device "enabler" in swarm :(

Also following this. Very interested in access to character devices (/dev/bus/usb/...) in a docker swarm. To help some others until this is supported by docker, a workaround for swarm + usb:
1. On the (linux) host(s), create a udev rule which creates a symlink to your device (in my case an ftdi device). e.g. /etc/udev/rules.d/99-libftdi.rules
   `SUBSYSTEMS=="usb", ATTRS{idVendor}=="xxxx", ATTRS{idProduct}=="xxxx", GROUP="dialout", MODE="0666", SYMLINK+="my_ftdi", RUN+="/usr/bin/setupdockerusb.sh"`
   Then reload udev rules:
   `sudo udevadm control --reload-rules`
   Upon connect of the usb device, the udev manager will create a symlink /dev/my_ftdi -> /dev/bus/usb/xxx/xxx and execute /usr/bin/setupdockerusb.sh

2. The /usr/bin/setupdockerusb.sh ([ref](https://groups.google.com/forum/#!topic/docker-user/SukMxOG0oP0))
   This script sets the character device permissions on (the first) container with given image name.

#!/bin/bash
USBDEV=`readlink -f /dev/my_ftdi`
read minor major < <(stat -c '%T %t' $USBDEV)
if [[ -z $minor || -z $major ]]; then
    echo 'Device not found'
    exit
fi
dminor=$((0x${minor}))
dmajor=$((0x${major}))
CID=`docker ps --no-trunc -q --filter ancestor=my/imagename|head -1`
if [[ -z $CID ]]; then
    echo 'CID not found'
    exit
fi
echo 'Setting permissions'
echo "c $dmajor:$dminor rwm" > /sys/fs/cgroup/devices/docker/$CID/devices.allow

3. Create the docker swarm with following options:
   `docker service create [...] --mount type=bind,source=/dev/bus/usb,target=/dev/bus/usb [...]`

4. Event listener (systemd service):
   Waits for a container to be started and sets permissions. Run with root permissions on host.

#!/bin/bash
docker events --filter 'event=start'| \
while read line; do
    /usr/bin/setupdockerusb.sh
done

Any ideas on how to make this work for cgroups2 ?
As far as I can make out, devices are treated differently than in cgroups1 (which I believe the above is targeting).

If anyone has any idea how to correctly advertise a quicksync driver to a cgroup v2 using dockerswarm it would be highly appreciated. Alternatively, I guess I could migrate to kubernetes ;)

@reisholmes check this out: https://github.com/allfro/device-mapping-manager

This did the trick for me for running linuxserver/plex with hardware acceleration using Intel QuickSync with Docker Swarm

If anyone has any idea how to correctly advertise a quicksync driver to a cgroup v2 using dockerswarm it would be highly appreciated. Alternatively, I guess I could migrate to kubernetes ;)

@reisholmes check this out: https://github.com/allfro/device-mapping-manager

This did the trick for me for running linuxserver/plex with hardware acceleration using Intel QuickSync with Docker Swarm

I might be a massive idiot - but I'm going to ask anyway - how did you make this work?
I still get stuck with this issue
Did you clone the repo and build the image yourself locally? Something else?

Hi @cdalvaro !

I still get stuck with this issue
Did you clone the repo and build the image yourself locally? Something else?

Hi @demaniak!
It's really straight forward! I follow these steps: https://github.com/allfro/device-mapping-manager/issues/12#issuecomment-2019301115
Create a service deployed in all machines with devices you want to allow in your services:
services:
  device-mapping-manager:
    image: alpinelinux/docker-cli
    entrypoint: docker
    command: |
      run
      --rm
      -i
      --name device-manager
      --privileged
      --cgroupns=host
      --pid=host
      --userns=host
      -v /sys:/host/sys
      -v /var/run/docker.sock:/var/run/docker.sock
      ghcr.io/allfro/allfro/device-mapping-manager:nightly
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    deploy:
      mode: global
      restart_policy:
        condition: any
      placement:
        constraints:
          - node.labels.dmm == true

I added the constrain node.labels.dmm == true since I don't want this service running in all my nodes. Just in some of them. Feel free to remove it if you want it in all your nodes.
This service will look for your containers (already created and newly created), and those devices mounted as volumes will be added to the allowed list of the container.
For example:
services:
  plex:
    image: ghcr.io/linuxserver/plex:latest
    container_name: plex
    hostname: "{{.Node.Hostname}}"
    networks:
      - traefik
    ports:
      - "32400:32400/tcp"
    volumes:
      - /dev/dri:/dev/dri # Hardware acceleration for Intel GPUs with Docker Swarm via DMM

Hope this can help you!

Thank you @cdalvaro , gonna try it out a bit later, but his looks like it should work!

Last I checked @skjnldsv, read-only doesn't work on the Docker socket, and it is still writable even when :ro is used.


[optimize] using Future or ListenableFuture instead of CompletableFut…

…ure.

Coverage decreased (-2.9%) to 93.939% when pulling 7c596d4c4e7219f80217ffa30b395c6abc805f60 on optimize-future into ac0ddf31d588bc6422ac78ed78c4d52490875073 on master.


Update Factorio.yaml

Not tested, just written up in github's editor.
Removed traps, they're bad for async.
Revamped tech costs to be reasonable
added recipe time, tech cost distribution and ramping tech cost
Personally I'd also recommend removing the peaceful mode starts. If someone wants to play with peaceful they can use the server console to change it.

While I agree with you on the peaceful mode stuff, enough people complain about it that I felt it was only fair to give them the option to start that way without having to enter in a command.

Personally I'd also recommend removing the peaceful mode starts. If someone wants to play with peaceful they can use the server console to change it.

does the reverse also apply? using console to switch it to non-peaceful? if so: id rather it be all peaceful starts and those who want to deal with biters can turn it on themselves

this probably needs updated with the new factorio things, doesnt it?

Probably does, good luck, please don't mess it up again.

Personally I'd also recommend removing the peaceful mode starts. If someone wants to play with peaceful they can use the server console to change it.

does the reverse also apply? using console to switch it to non-peaceful? if so: id rather it be all peaceful starts and those who want to deal with biters can turn it on themselves

No, the thing most people use to turn it peaceful removes the Biters from the map, which is irreversable.

Probably does, good luck, please don't mess it up again.

no i meant aren't there new options? i dunno jack about factorio
so was wondering if you wanted to update this with said new options before i do anything with this

This PR's purpose is to unfuck Factorio.


config:darwin → config:darwin_x86_64

Motivation

This makes room for if we ever want to define config:darwin_arm64.
Test plan

See included automated tests.

Current dependencies on/for this PR:

master

PR #6817   👈

This comment was auto-generated by Graphite.


[Transaction] Migrate transaction coordinator handleInitProducerId tests and fix behavior

Motivation
Currently, KoP support transaction but don't have units test to cover transaction coordinator code behavior, for future maintain, we should add units test from  Kafka.
Modification

Fix handleInitProducerId behavior.
Add units test to ensure behavior same as Kafka.

Ping @gaoran10 @BewareMyPower PTAL. Thanks.


[FEAT]: Remember volume

Description
Volume is reset when you quit and it's way too loud for me so every time it's a blast (risky with earphones depending on the channel). It would be great if tera could remember the volume level.
Additional Information
No response

It can be done by adding a function that writes to a config file and using that value when it starts.
Let's see what I can do....


Security.Test Bmc Ssh Security.Verify BMC SSH Weak Cipher And Algorithm test fails, but returns PASS

The test is designed to expect the ssh command to fail because login permission is denied, but if the ssh command fails for other reasons the test case still passes, even though it has not fulfilled the test case goal of checking the ciphers.
In my working environment (see below), the test case fails because the ssh command line doesn't contain '-o' in front of each of the second and third ssh option arguments, as demonstrated below.
failing code:
    ${ssh_cmd_buf}=  Catenate  ssh -o NumberOfPasswordPrompts=0 UserKnownHostsFile=/dev/null
    ...  StrictHostKeyChecking=no -vv ${OPENBMC_HOST} 2>&1

remediated code:
    ${ssh_cmd_buf}=  Catenate  ssh -o NumberOfPasswordPrompts=0 -o UserKnownHostsFile=/dev/null
    ...  -o StrictHostKeyChecking=no -vv ${OPENBMC_HOST} 2>&1

I observed this failure on Ubuntu 18.04 with OpenSSH 7.6p1 / OpenSSL 1.0.2 and Python 3.6.9.
I propose improving the test case by first verifying that the command returns the kind of information desired.
In my environment the remediated code returns the error 'Permission denied' since the ssh attempt requests no password prompt ('-o NumberOfPasswordPrompts=0). My refactored code expects that string to be returned, and returns SKIP if that criterion is not satisfied.
Here is the git diff showing how I have refactored the code:
diff --git a/security/test_bmc_ssh_security.robot b/security/test_bmc_ssh_security.robot
index fae06281..c70a3972 100644
--- a/security/test_bmc_ssh_security.robot
+++ b/security/test_bmc_ssh_security.robot
@@ -41,6 +41,8 @@ Verify BMC SSH Weak Cipher And Algorithm
     # debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm<EMAIL_ADDRESS>     #         hmac-sha2-512-etm<EMAIL_ADDRESS>     #         umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
+    #         ...
+    #         ... Permission denied (publickey,password).

     # Example of weak algorithms to check:
     # - encryption: triple-DES ("DES-CBC3").
@@ -49,7 +51,16 @@ Verify BMC SSH Weak Cipher And Algorithm
     # - KEX: diffie-hellman-group1(any) , (any) SHA1

     Printn
-    ${ssh_cmd_buf}=  Catenate  ssh -o NumberOfPasswordPrompts=0 UserKnownHostsFile=/dev/null
-    ...  StrictHostKeyChecking=no -vv ${OPENBMC_HOST} 2>&1
+    ${ssh_cmd_buf}=  Catenate  ssh -o NumberOfPasswordPrompts=0 -o UserKnownHostsFile=/dev/null
+    ...    -o StrictHostKeyChecking=no -vv ${OPENBMC_HOST} 2>&1
+
+    ${rc}  ${std_err}=  Shell Cmd  ! ${ssh_cmd_buf}
+    Log  std_err=${std_err}  console=yes
+    Log  rc=${rc} console=yes
+
+    ${has_it}=  Run Keyword And Return Status  Should Contain  ${std_err}  Permission denied
+    Skip If  not ${has_it}
+    ...  Skipping test case since response is not as expected
+
     Shell Cmd  ! ${ssh_cmd_buf} | egrep -- "${weak_key_regex}"
     Shell Cmd  ! ${ssh_cmd_buf} | egrep -- "${mac_key_regex}"

Fix https://gerrit.openbmc-project.xyz/c/openbmc/openbmc-test-automation/+/52110

@generatz   https://gerrit.openbmc-project.xyz/c/openbmc/openbmc-test-automation/+/52110
Take a look


feat: Simplify upgrade process

The wording "update" might be changed, as updating a container to a new version of the image requires recreating the container. So, it's a more destructive process than a simple "update"
Will close #41

Code side is done, just needs testing before I merge it


iOS: Timeline: use filled-in triangles instead of chevrons for expand/collapse

SF Symbols has some filled-in triangles.
Yes, this is different from Mail and some other apps, but I want to try it.

You mean Feeds right, not Timeline I presume.

Sheesh. Yes.