Feat/kraken connector

Before submitting this PR, please make sure:

[x] Your code builds clean without any errors or warnings
[x] You are using approved title ("feat/", "fix/", "docs/", "refactor/")

A description of the changes proposed in the pull request:
This PR create kraken connector to the latest standards.
Tests performed by the developer:
unittest
Tips for QA testing:
Normal QA test for spot connectors.

PR update:

Test commit 78fa258336c06

When adding kraken API key getting TypeError:2024-02-08 14:39:42,741 - 22239 - hummingbot.core.utils.async_utils - ERROR - Unhandled error in background task: KrakenExchange.__init__() got an unexpected keyword argument 'kraken_secret_key'
Traceback (most recent call last):
  File "/Users/rapcomia/github/hummingbot/6840/hummingbot/core/utils/async_utils.py", line 9, in safe_wrapper
    return await c
  File "/Users/rapcomia/github/hummingbot/6840/hummingbot/client/command/connect_command.py", line 61, in connect_exchange
    await self._perform_connect(connector_config, previous_keys)
  File "/Users/rapcomia/github/hummingbot/6840/hummingbot/client/command/connect_command.py", line 144, in _perform_connect
    err_msg = await self.validate_n_connect_connector(connector_name)
  File "/Users/rapcomia/github/hummingbot/6840/hummingbot/client/command/connect_command.py", line 122, in validate_n_connect_connector
    err_msg = await asyncio.wait_for(
  File "/Users/rapcomia/miniconda3/envs/hummingbot/lib/python3.10/asyncio/tasks.py", line 445, in wait_for
    return fut.result()
  File "/Users/rapcomia/github/hummingbot/6840/hummingbot/user/user_balances.py", line 89, in add_exchange
    market = UserBalances.connect_market(exchange, client_config_map, **api_details)
  File "/Users/rapcomia/github/hummingbot/6840/hummingbot/user/user_balances.py", line 46, in connect_market
    connector = connector_class(**init_params)
TypeError: KrakenExchange.__init__() got an unexpected keyword argument 'kraken_secret_key'

Compare results from development and this PR, we are able to add the keys ok successfully

Steps to reproduce:

Run connect kraken and add API keys

logs_hummingbot.log

PR update:

Test commit 454320b9948bb33bb6e52f240894785532357dc4
Connect API keys but still no go, this time getting different result

Compared to development branch's kraken, we are able to connect API keys successfully

PR update:

Test commit 37cae7118aa549c931ce3eded28d8d9bad911e5b

Reported issue on connector API keys have been fixed ✅
Setup simple PMM strategy:
Upon initializing start getting AttributeError:2024-02-20 18:50:36,056 - 87883 - hummingbot.client.hummingbot_application - INFO - start command initiated.
2024-02-20 18:50:36,516 - 87883 - hummingbot.connector.exchange.kraken.kraken_exchange.KrakenExchange - INFO - Network status has changed to NetworkStatus.CONNECTED. Starting networking...
2024-02-20 18:50:36,517 - 87883 - hummingbot.core.api_throttler.async_request_context_base - INFO - API rate limit on PublicEndpointLimitID (1 calls per 1s) has almost reached. Limits used is 1 in the last 1 seconds
2024-02-20 18:50:37,001 - 87883 - hummingbot.connector.exchange.kraken.kraken_exchange.KrakenExchange - ERROR - Error requesting time from Kraken server
Traceback (most recent call last):
File "/Users/rapcomia/github/hummingbot/6840/hummingbot/connector/exchange_py_base.py", line 799, in _update_time_synchronizer
time_provider=self.web_utils.get_current_server_time(
AttributeError: module 'hummingbot.connector.exchange.kraken.kraken_web_utils' has no attribute 'get_current_server_time'
2024-02-20 18:50:37,005 - 87883 - hummingbot.connector.exchange.kraken.kraken_exchange.KrakenExchange - NETWORK - Unexpected error while fetching account updates.
Traceback (most recent call last):
File "/Users/rapcomia/github/hummingbot/6840/hummingbot/connector/exchange_py_base.py", line 777, in _status_polling_loop
await self._update_time_synchronizer()
File "/Users/rapcomia/github/hummingbot/6840/hummingbot/connector/exchange_py_base.py", line 799, in _update_time_synchronizer
time_provider=self.web_utils.get_current_server_time(
AttributeError: module 'hummingbot.connector.exchange.kraken.kraken_web_utils' has no attribute 'get_current_server_time'
2024-02-20 18:50:37,005 - 87883 - hummingbot.connector.exchange.kraken.kraken_exchange.KrakenExchange - WARNING - Could not fetch account updates from Kraken. Check API key and network connection.

Tried to change change API tier from Starter, Intermediate and Pro still getting same error
Tried to check if increasing the nonce on API setup would fix, still the same error

logs_pmm_01.log

the problem fixed. @rapcmia

PR update:

Test commit 6344f5be00592043250d4a140df69547fd8d4578
Reported AttributeError has been fixed
Create simple PMM

Observe when initially started the strategy, getting Rate limited
Rate limit logs are noticeable on the connector, tried to change API nonce from 100 to 300 same results. Possible exchange related

Start strategy and order posted ok but there instances of errors:
2024-02-22 12:48:01,453 - 41276 - hummingbot.connector.exchange.kraken.kraken_exchange.KrakenExchange - ERROR - Error received from /0/private/QueryTrades. Response is {'error': ['EOrder:Invalid order']}.
2024-02-22 12:48:01,454 - 41276 - hummingbot.connector.exchange.kraken.kraken_exchange.KrakenExchange - WARNING - Failed to fetch trade updates for order HBOTBSLUT611f124b69bdf2250a018f8. Error: argument of type 'OSError' is not iterable
Traceback (most recent call last):
  File "/Users/rapcomia/github/hummingbot/6840/hummingbot/connector/exchange/kraken/kraken_exchange.py", line 597, in _all_trade_updates_for_order
    all_fills_response = await self._api_request_with_retry(
  File "/Users/rapcomia/github/hummingbot/6840/hummingbot/connector/exchange/kraken/kraken_exchange.py", line 408, in _api_request_with_retry
    raise e
  File "/Users/rapcomia/github/hummingbot/6840/hummingbot/connector/exchange/kraken/kraken_exchange.py", line 391, in _api_request_with_retry
    raise IOError({"error": response_json})
OSError: {'error': {'error': ['EOrder:Invalid order']}}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/rapcomia/github/hummingbot/6840/hummingbot/connector/exchange_py_base.py", line 954, in _update_orders_fills
    trade_updates = await self._all_trade_updates_for_order(order=order)
  File "/Users/rapcomia/github/hummingbot/6840/hummingbot/connector/exchange/kraken/kraken_exchange.py", line 615, in _all_trade_updates_for_order
    if "EOrder:Unknown order" in e or "EOrder:Invalid order" in e:
TypeError: argument of type 'OSError' is not iterable
2024-02-22 12:48:01,689 - 41276 - hummingbot.connector.exchange.kraken.kraken_exchange.KrakenExchange - ERROR - Error received from /0/private/QueryTrades. Response is {'error': ['EOrder:Invalid order']}.
2024-02-22 12:48:01,689 - 41276 - hummingbot.connector.exchange.kraken.kraken_exchange.KrakenExchange - WARNING - Failed to fetch trade updates for order HBOTBSLUT611f12113d8922250a018f8. Error: argument of type 'OSError' is not iterable
Traceback (most recent call last):
  File "/Users/rapcomia/github/hummingbot/6840/hummingbot/connector/exchange/kraken/kraken_exchange.py", line 597, in _all_trade_updates_for_order
    all_fills_response = await self._api_request_with_retry(
  File "/Users/rapcomia/github/hummingbot/6840/hummingbot/connector/exchange/kraken/kraken_exchange.py", line 408, in _api_request_with_retry
    raise e
  File "/Users/rapcomia/github/hummingbot/6840/hummingbot/connector/exchange/kraken/kraken_exchange.py", line 391, in _api_request_with_retry
    raise IOError({"error": response_json})
OSError: {'error': {'error': ['EOrder:Invalid order']}}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/rapcomia/github/hummingbot/6840/hummingbot/connector/exchange_py_base.py", line 954, in _update_orders_fills
    trade_updates = await self._all_trade_updates_for_order(order=order)
  File "/Users/rapcomia/github/hummingbot/6840/hummingbot/connector/exchange/kraken/kraken_exchange.py", line 615, in _all_trade_updates_for_order
    if "EOrder:Unknown order" in e or "EOrder:Invalid order" in e:
TypeError: argument of type 'OSError' is not iterable
2024-02-22 12:48:01,918 - 41276 - hummingbot.connector.exchange.kraken.kraken_exchange.KrakenExchange - ERROR - Error received from /0/private/QueryOrders. Response is {'error': ['EGeneral:Invalid arguments']}.
2024-02-22 12:48:01,919 - 41276 - hummingbot.connector.exchange.kraken.kraken_exchange.KrakenExchange - WARNING - Error fetching status update for the active order HBOTBSLUT611f124b69bdf2250a018f8: {'error': {'error': ['EGeneral:Invalid arguments']}}.
2024-02-22 12:48:12,739 - 41276 - hummingbot.core.api_throttler.async_request_context_base - INFO - API rate limit on /0/public/Ticker (1 calls per 1s) has almost reached. Limits used is 1 in the last 1 seconds

Rebalance assets and observe behavior for both buy and sell
Single order levels:

Observed same OSError occurred
There are instances of Unexpected error while processing event 2002024-02-22 13:23:13,001 - 41276 - hummingbot.strategy.pure_market_making.pure_market_making - INFO - (XRP-USDT) Creating 1 bid orders at (Size, Price): ['1E+1 XRP, 0.53683 USDT']
2024-02-22 13:23:13,001 - 41276 - hummingbot.strategy.pure_market_making.pure_market_making - INFO - (XRP-USDT) Creating 1 ask orders at (Size, Price): ['1E+1 XRP, 0.54767 USDT']
2024-02-22 13:23:13,283 - 41276 - hummingbot.connector.client_order_tracker - INFO - Created LIMIT_MAKER BUY order HBOTBXPUT611f1a3ca57de2250a018f8 for 10.00000000 XRP-USDT.
2024-02-22 13:23:13,283 - 41276 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp":<PHONE_NUMBER>.0, "type": "OrderType.LIMIT_MAKER", "trading_pair": "XRP-USDT", "amount": "10.00000000", "price": "0.53683", "order_id": "HBOTBXPUT611f1a3ca57de2250a018f8", "creation_timestamp":<PHONE_NUMBER>.0, "exchange_order_id": "OWYJJO-2L46L-EHX46Q", "leverage": 1, "position": "NIL", "event_name": "BuyOrderCreatedEvent", "event_source": "kraken"}
2024-02-22 13:23:13,287 - 41276 - hummingbot.connector.client_order_tracker - INFO - Created LIMIT_MAKER BUY order HBOTBXPUT611f1a3ca57de2250a018f8 for 10.00000000 XRP-USDT.
2024-02-22 13:23:13,288 - 41276 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp":<PHONE_NUMBER>.0, "type": "OrderType.LIMIT_MAKER", "trading_pair": "XRP-USDT", "amount": "10.00000000", "price": "0.53683", "order_id": "HBOTBXPUT611f1a3ca57de2250a018f8", "creation_timestamp":<PHONE_NUMBER>.0, "exchange_order_id": "OWYJJO-2L46L-EHX46Q", "leverage": 1, "position": "NIL", "event_name": "BuyOrderCreatedEvent", "event_source": "kraken"}
2024-02-22 13:23:13,290 - 41276 - hummingbot.connector.exchange.kraken.kraken_exchange.KrakenExchange - ERROR - Unexpected error while processing event 200.
Traceback (most recent call last):
  File "/Users/rapcomia/miniconda3/envs/hummingbot/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
    self.dialect.do_execute(
  File "/Users/rapcomia/miniconda3/envs/hummingbot/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
sqlite3.IntegrityError: UNIQUE constraint failed: Order.id

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "hummingbot/core/pubsub.pyx", line 165, in hummingbot.core.pubsub.PubSub.c_trigger_event
    typed_listener.c_call(arg)
  File "hummingbot/core/event/event_listener.pyx", line 25, in hummingbot.core.event.event_listener.EventListener.c_call
    self(arg)
  File "/Users/rapcomia/github/hummingbot/6840/hummingbot/core/event/event_forwarder.py", line 24, in __call__
    self._to_function(self.current_event_tag, self.current_event_caller, arg)
  File "/Users/rapcomia/github/hummingbot/6840/hummingbot/connector/markets_recorder.py", line 321, in _did_create_order
    self.save_market_states(self._config_file_path, market, session=session)
  File "/Users/rapcomia/github/hummingbot/6840/hummingbot/connector/markets_recorder.py", line 254, in save_market_states
    market_states: Optional[MarketState] = self.get_market_states(config_file_path, market, session=session)
  File "/Users/rapcomia/github/hummingbot/6840/hummingbot/connector/markets_recorder.py", line 282, in get_market_states
    market_states: Optional[MarketState] = query.one_or_none()
  File "/Users/rapcomia/miniconda3/envs/hummingbot/lib/python3.10/site-packages/sqlalchemy/orm/query.py", line 2850, in one_or_none
    return self._iter().one_or_none()
  File "/Users/rapcomia/miniconda3/envs/hummingbot/lib/python3.10/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter
    result = self.session.execute(
  File "/Users/rapcomia/miniconda3/envs/hummingbot/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1665, in execute
    ) = compile_state_cls.orm_pre_session_exec(
  File "/Users/rapcomia/miniconda3/envs/hummingbot/lib/python3.10/site-packages/sqlalchemy/orm/context.py", line 312, in orm_pre_session_exec
    session._autoflush()
  File "/Users/rapcomia/miniconda3/envs/hummingbot/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 2264, in _autoflush
    util.raise_(e, with_traceback=sys.exc_info()[2])
  File "/Users/rapcomia/miniconda3/envs/hummingbot/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/Users/rapcomia/miniconda3/envs/hummingbot/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 2253, in _autoflush
    self.flush()
  File "/Users/rapcomia/miniconda3/envs/hummingbot/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3449, in flush
    self._flush(objects)
  File "/Users/rapcomia/miniconda3/envs/hummingbot/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3588, in _flush
    with util.safe_reraise():
  File "/Users/rapcomia/miniconda3/envs/hummingbot/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
    compat.raise_(
  File "/Users/rapcomia/miniconda3/envs/hummingbot/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/Users/rapcomia/miniconda3/envs/hummingbot/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 3549, in _flush
    flush_context.execute()
  File "/Users/rapcomia/miniconda3/envs/hummingbot/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute
    rec.execute(self)
  File "/Users/rapcomia/miniconda3/envs/hummingbot/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute
    util.preloaded.orm_persistence.save_obj(
  File "/Users/rapcomia/miniconda3/envs/hummingbot/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj
    _emit_insert_statements(
  File "/Users/rapcomia/miniconda3/envs/hummingbot/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements
    c = connection._execute_20(
  File "/Users/rapcomia/miniconda3/envs/hummingbot/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1710, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
  File "/Users/rapcomia/miniconda3/envs/hummingbot/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
    return connection._execute_clauseelement(
  File "/Users/rapcomia/miniconda3/envs/hummingbot/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement
    ret = self._execute_context(
  File "/Users/rapcomia/miniconda3/envs/hummingbot/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1953, in _execute_context
    self._handle_dbapi_exception(
  File "/Users/rapcomia/miniconda3/envs/hummingbot/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2134, in _handle_dbapi_exception
    util.raise_(
  File "/Users/rapcomia/miniconda3/envs/hummingbot/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
    raise exception
  File "/Users/rapcomia/miniconda3/envs/hummingbot/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
    self.dialect.do_execute(
  File "/Users/rapcomia/miniconda3/envs/hummingbot/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely)
(sqlite3.IntegrityError) UNIQUE constraint failed: Order.id
[SQL: INSERT INTO "Order" (id, config_file_path, strategy, market, symbol, base_asset, quote_asset, creation_timestamp, order_type, amount, leverage, price, last_status, last_update_timestamp, exchange_order_id, position) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)]
[parameters: ('HBOTBXPUT611f1a3ca57de2250a018f8', 'pmm_kraken2.yml', 'pure_market_making', 'kraken', 'XRP-USDT', 'XRP', 'USDT',<PHONE_NUMBER>000, 'LIMIT_MAKER', 10000000, 1, 536830, 'BuyOrderCreated',<PHONE_NUMBER>000, 'OWYJJO-2L46L-EHX46Q', 'NIL')]
(Background on this error at: https://sqlalche.me/e/14/gkpj)

The issue occured while the connector was creating single orders, it created two buy orders
Both buy orders have similar order data

Found also Unexpected error while processing event 201.

Traceback can be found on attached log
Similar with the error reported above for event 200, this time the connector created two sell order causing this error
Both sell orders have similar order data

Steps to reproduce:

Add kraken API keys
Create simple pmm strategy then start
Observe order creation and cancelation of order

logs_pmm_kraken1a.log
logs_pmm_kraken2.log

Pending

Filled order event
Multiple order levels
V2 Strategy
Run long term bot

the above problems fixed.

PR update:

Test commit e0d0b4684dad066e1b8452fed645c96c355359ff
Reported issues on ...events 200 and 201 has been fixed
Start simple PMM but getting failed to cancel order:

2024-02-23 15:00:07,007 - 16357 - hummingbot.connector.exchange.kraken.kraken_exchange.KrakenExchange - ERROR - Failed to cancel order HBOTBSLUT612071b0b038db4280b1fde
Traceback (most recent call last):
  File "/Users/rapcomia/github/hummingbot/6840/hummingbot/connector/exchange_py_base.py", line 520, in _execute_order_cancel
    cancelled = await self._execute_order_cancel_and_process_update(order=order)
  File "/Users/rapcomia/github/hummingbot/6840/hummingbot/connector/exchange_py_base.py", line 540, in _execute_order_cancel_and_process_update
    cancelled = await self._place_cancel(order.client_order_id, order)
  File "/Users/rapcomia/github/hummingbot/6840/hummingbot/connector/exchange/kraken/kraken_exchange.py", line 417, in _place_cancel
    cancel_result = await self._api_request_with_retry(
  File "/Users/rapcomia/github/hummingbot/6840/hummingbot/connector/exchange/kraken/kraken_exchange.py", line 381, in _api_request_with_retry
    response_json = await self._api_request(path_url=path_url, method=method, params=params, data=data,
  File "/Users/rapcomia/github/hummingbot/6840/hummingbot/connector/exchange_py_base.py", line 905, in _api_request
    request_result = await rest_assistant.execute_request(
  File "/Users/rapcomia/github/hummingbot/6840/hummingbot/core/web_assistant/rest_assistant.py", line 47, in execute_request
    response = await self.execute_request_and_get_response(
  File "/Users/rapcomia/github/hummingbot/6840/hummingbot/core/web_assistant/rest_assistant.py", line 80, in execute_request_and_get_response
    data = json.dumps(data) if data is not None else data
  File "/Users/rapcomia/miniconda3/envs/hummingbot/lib/python3.10/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/Users/rapcomia/miniconda3/envs/hummingbot/lib/python3.10/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/Users/rapcomia/miniconda3/envs/hummingbot/lib/python3.10/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/Users/rapcomia/miniconda3/envs/hummingbot/lib/python3.10/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type coroutine is not JSON serializable

Run stop command, same error occurred.
Orders affected on this error is stuck on exchange
Ran another instances and observed when reaching refresh time, same error

Steps to reproduce:

Start PMM with Kraken and observe order cancelation
Check exchange, orders are stuck need to manual stop
When stop command initiated, same error occurred

logs_pmm_kraken1.log
logs_pmm_kraken2.log

fixed

PR update:

Test commit c903d5006079742c8384bb995a033238400cd4c2
Reported issue on failed to cancel order has been fixed
Ran tests on:
Setup with one side balance only
Setup with multiple order level with hanging order
For filled orders we observed getting Unexpected error in user stream listener loop..KeyError: 'status'2024-02-26 17:16:27,537 - 55260 - hummingbot.connector.client_order_tracker - INFO - Created LIMIT_MAKER SELL order HBOTSXPUT612455d43575ffdda881940 for 10.00000000 XRP-USDT.
2024-02-26 17:16:27,537 - 55260 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp":<PHONE_NUMBER>.0, "type": "OrderType.LIMIT_MAKER", "trading_pair": "XRP-USDT", "amount": "10.00000000", "price": "0.54089", "order_id": "HBOTSXPUT612455d43575ffdda881940", "creation_timestamp":<PHONE_NUMBER>.0, "exchange_order_id": "OKP477-GLJC4-2TSJ43", "leverage": 1, "position": "NIL", "event_name": "SellOrderCreatedEvent", "event_source": "kraken"}
2024-02-26 17:16:29,595 - 55260 - hummingbot.connector.client_order_tracker - INFO - Created LIMIT_MAKER BUY order HBOTBXPUT612455d4356c1fdda881940 for 10.00000000 XRP-USDT.
2024-02-26 17:16:29,595 - 55260 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp":<PHONE_NUMBER>.0, "type": "OrderType.LIMIT_MAKER", "trading_pair": "XRP-USDT", "amount": "10.00000000", "price": "0.53550", "order_id": "HBOTBXPUT612455d4356c1fdda881940", "creation_timestamp":<PHONE_NUMBER>.0, "exchange_order_id": "OEJMNS-MPQQG-5VL2MX", "leverage": 1, "position": "NIL", "event_name": "BuyOrderCreatedEvent", "event_source": "kraken"}
2024-02-26 17:21:08,807 - 55260 - hummingbot.connector.exchange.kraken.kraken_exchange.KrakenExchange - ERROR - Unexpected error in user stream listener loop.
Traceback (most recent call last):
  File "/Users/rapcomia/github/hummingbot/6840/hummingbot/connector/exchange/kraken/kraken_exchange.py", line 513, in _user_stream_event_listener
    self._process_order_message(event_message)
  File "/Users/rapcomia/github/hummingbot/6840/hummingbot/connector/exchange/kraken/kraken_exchange.py", line 588, in _process_order_message
    order_update = self._create_order_update_with_order_status_data(order_status=order_msg,
  File "/Users/rapcomia/github/hummingbot/6840/hummingbot/connector/exchange/kraken/kraken_exchange.py", line 573, in _create_order_update_with_order_status_data
    new_state=CONSTANTS.ORDER_STATE[order_status["status"]],
KeyError: 'status'
2024-02-26 17:21:13,810 - 55260 - hummingbot.connector.client_order_tracker - INFO - The BUY order HBOTBXPUT612455d4356c1fdda881940 amounting to 10.00000000/10.00000000 XRP has been filled.
2024-02-26 17:21:13,812 - 55260 - hummingbot.strategy.pure_market_making.pure_market_making - INFO - (XRP-USDT) Maker buy order of 10.00000000 XRP filled. [clock=2024-02-26 09:21:13+00:00]

Reproduced on 2/2 attempt on same market pair
No orders stuck nor when stop command is initiated
Trade data and orders matched between connector and exchange

Steps to reproduce:

Start PMM with kraken and observe connector when filled order happened

logs_pmm_kraken2.log

Pending:

Test on dmanv3 with kraken as candles data in progress
Pending test for dmanv4 and other v2 strategies not started

PR update:

Latest commit dda2d63bcbe12b66cd82cfc47ef5ddd7cde4cee4
Confirmed issues reported on filled order status error fixed
Running dmanv3 overnight on spot market wip

PR update:

When reading logs found an issue on filled order event and tried to reproduce for v2 strategy (dmanv3)

After a successfully filled order, got order filled updates did not arrive on time
Compare this behavior with V1 (PMM strategy) not able to reproduce the issue
This causes filled order not recorded on history or CSV file but available on exchange

Steps:

Copy the .yml file attached and paste in conf/script folder
Run start --script v2_dman_v3_config.py --conf the.yml

logs_conf_dmanv3a.log
conf_dmanv3a.yml.log remove .log

fixed

PR update:

Ran tests on latest commit 8d13b890be5da9...

Start same dmanv3 script and seems to create order on connector but not on exchange
Observed that the created order was already filled on exchange and nothing happened on connector. Took 1-2mins before getting the logs:

Order filled updates idid not arrived on time.. followed by ...order successfully filled

Ran stop command, got failed to cancel order error:2024-02-29 12:11:48,908 - 22038 - hummingbot.client.hummingbot_application - INFO - stop command initiated.
2024-02-29 12:11:49,223 - 22038 - hummingbot.connector.exchange.kraken.kraken_exchange.KrakenExchange - ERROR - Failed to cancel order HBOTSXPUT6127d6ee5dfe61e095fba3e
Traceback (most recent call last):
  File "/Users/rapcomia/github/hummingbot/6840/hummingbot/connector/exchange_py_base.py", line 520, in _execute_order_cancel
    cancelled = await self._execute_order_cancel_and_process_update(order=order)
  File "/Users/rapcomia/github/hummingbot/6840/hummingbot/connector/exchange_py_base.py", line 540, in _execute_order_cancel_and_process_update
    cancelled = await self._place_cancel(order.client_order_id, order)
  File "/Users/rapcomia/github/hummingbot/6840/hummingbot/connector/exchange/kraken/kraken_exchange.py", line 417, in _place_cancel
    cancel_result = await self._api_request_with_retry(
  File "/Users/rapcomia/github/hummingbot/6840/hummingbot/connector/exchange/kraken/kraken_exchange.py", line 407, in _api_request_with_retry
    raise e
  File "/Users/rapcomia/github/hummingbot/6840/hummingbot/connector/exchange/kraken/kraken_exchange.py", line 390, in _api_request_with_retry
    raise IOError({"error": response_json})
OSError: {'error': {'error': ['EOrder:Unknown order']}}

Observed when there is filled order event:

Order fill updates did not arrived on time happened but also added order fill is successfully completed
However it does not indicate anything on history nor CSV
Confirmed the order created on the connector was filled already on exchange
After reaching the set timelimit (300s)

Connector tries to place a close order to complete position executor however:
Connector returned an order price NaN ammount: 0 to trigger script max-retries

Steps to reproduce:

Copy the .yml file attached and paste in conf/script folder
Run start --script v2_dman_v3_config.py --conf the.yml

logs_conf_dmanv3a.log
conf_dmanv3a.yml.log remove .log

fixed

Have deleted

The new code removes the KrakenInFlightOrder and uses "get_new_numeric_client_order_id" to produce the "client_order_id".

@rapcmia code looks good, just do a minimal test to validate that everything is working after removing the inflight order

PR update:
- Ran tests on v1, all ok
- Ran tests by using refactored v2 strategy using pmm_simple controller
- Ran few more tests specifically on filled order events
- Trade data between client, csv and exchange history matched
- Setup long term bot for 1-2hrs and observe behavior
- Orders created and cancelled ok
- Script notified if insufficient balance, same behavior other connectors ✅


 Are you using a build from Alcatraz, or did you build and install yourself? If the latter, did you by any chance build from the nonblocking branch?

It's latest version from Alcatraz 2.1.1

Installed from nonblocking, seems ok

I just wanted to make sure you are not runing nonblocking, as it is not stable, and this crash is known to occur there (because filtering is asynchronous).

But with nonblocking it's ok and just swiched back to Alcatraz to test and crash again.
So I think I'll try to use nonblocking branch for a while

by the way, I have compilation error in DVTCompletingTextView.h file on
@property(retain) id <DVTCompletingTextViewDelegate> delegate; // @dynamic delegate;
line. Just commented out it and built

There was no crash, because plugin didn't load.
Added UDID, still :ok_hand:

meh...now it mess up while I'm defining property loginLabel :tired_face:
@property (nonatomic, strong) TTTAttributedLabel *loglebaLni;

This should no longer crash on master. Yes, I know that nonblocking is broken, that is why I wanted to make sure you are not using it by any chance.


Getting this error with the raft branch on RethinkDB 2.1

The changefeed on server_status returned an error: ReqlServerError: The connection was closed before the query could be completed for:
Aug 11 19:16:48 aidax-batcher-1 aidax[2580]: r.db("rethinkdb").table("server_status").union(["feedSeparator"]).union(r.db("rethinkdb").table("server_status").changes())
Aug 11 19:17:25 aidax-batcher-1 aidax[2580]: Unhandled rejection TypeError: Cannot read property 'reject' of undefined
Aug 11 19:17:25 aidax-batcher-1 aidax[2580]: at /app/releases/20150811164745/node_modules/rethinkdbdash/lib/connection.js:533:25
Aug 11 19:17:25 aidax-batcher-1 aidax[2580]: at Object.tryCatch (/app/releases/20150811164745/node_modules/rethinkdbdash/lib/helper.js:170:3)
Aug 11 19:17:25 aidax-batcher-1 aidax[2580]: at Connection._send (/app/releases/20150811164745/node_modules/rethinkdbdash/lib/connection.js:530:10)
Aug 11 19:17:25 aidax-batcher-1 aidax[2580]: at /app/releases/20150811164745/node_modules/rethinkdbdash/lib/term.js:199:32
Aug 11 19:17:25 aidax-batcher-1 aidax[2580]: at tryCatcher (/app/releases/20150811164745/node_modules/bluebird/js/main/util.js:26:23)
Aug 11 19:17:25 aidax-batcher-1 aidax[2580]: at Promise._settlePromiseFromHandler (/app/releases/20150811164745/node_modules/bluebird/js/main/promise.js:503:31)
Aug 11 19:17:25 aidax-batcher-1 aidax[2580]: at Promise._settlePromiseAt (/app/releases/20150811164745/node_modules/bluebird/js/main/promise.js:577:18)
Aug 11 19:17:25 aidax-batcher-1 aidax[2580]: at Promise._settlePromiseAtPostResolution (/app/releases/20150811164745/node_modules/bluebird/js/main/promise.js:244:10)
Aug 11 19:17:25 aidax-batcher-1 aidax[2580]: at Async._drainQueue (/app/releases/20150811164745/node_modules/bluebird/js/main/async.js:128:12)
Aug 11 19:17:25 aidax-batcher-1 aidax[2580]: at Async._drainQueues (/app/releases/20150811164745/node_modules/bluebird/js/main/async.js:133:10)
Aug 11 19:17:25 aidax-batcher-1 aidax[2580]: at Immediate.Async.drainQueues [as _onImmediate] (/app/releases/20150811164745/node_modules/bluebird/js/main/async.js:15:14)
Aug 11 19:17:25 aidax-batcher-1 aidax[2580]: at processImmediate [as _immediateCallback] (timers.js:368:17)
Aug 11 19:18:07 aidax-batcher-1 aidax[2580]: Unhandled rejection TypeError: Cannot read property 'reject' of undefined
Aug 11 19:18:07 aidax-batcher-1 aidax[2580]: at /app/releases/20150811164745/node_modules/rethinkdbdash/lib/connection.js:533:25
Aug 11 19:18:07 aidax-batcher-1 aidax[2580]: at Object.tryCatch (/app/releases/20150811164745/node_modules/rethinkdbdash/lib/helper.js:170:3)
Aug 11 19:18:07 aidax-batcher-1 aidax[2580]: at Connection._send (/app/releases/20150811164745/node_modules/rethinkdbdash/lib/connection.js:530:10)
Aug 11 19:18:07 aidax-batcher-1 aidax[2580]: at /app/releases/20150811164745/node_modules/rethinkdbdash/lib/term.js:199:32
Aug 11 19:18:07 aidax-batcher-1 aidax[2580]: at tryCatcher (/app/releases/20150811164745/node_modules/bluebird/js/main/util.js:26:23)
Aug 11 19:18:07 aidax-batcher-1 aidax[2580]: at Promise._settlePromiseFromHandler (/app/releases/20150811164745/node_modules/bluebird/js/main/promise.js:503:31)
Aug 11 19:18:07 aidax-batcher-1 aidax[2580]: at Promise._settlePromiseAt (/app/releases/20150811164745/node_modules/bluebird/js/main/promise.js:577:18)
Aug 11 19:18:07 aidax-batcher-1 aidax[2580]: at Promise._settlePromiseAtPostResolution (/app/releases/20150811164745/node_modules/bluebird/js/main/promise.js:244:10)
Aug 11 19:18:07 aidax-batcher-1 aidax[2580]: at Async._drainQueue (/app/releases/20150811164745/node_modules/bluebird/js/main/async.js:128:12)
Aug 11 19:18:07 aidax-batcher-1 aidax[2580]: at Async._drainQueues (/app/releases/20150811164745/node_modules/bluebird/js/main/async.js:133:10)
Aug 11 19:18:07 aidax-batcher-1 aidax[2580]: at Immediate.Async.drainQueues [as _onImmediate] (/app/releases/20150811164745/node_modules/bluebird/js/main/async.js:15:14)
Aug 11 19:18:07 aidax-batcher-1 aidax[2580]: at processImmediate [as _immediateCallback] (timers.js:368:17)

The message was just printed right?
This is expected if the connection to the server where the feeds is opened dies.


Enumerable.ToDictionary Method

https://docs.microsoft.com/en-us/dotnet/api/system.linq.enumerable.todictionary?view=netcore-3.1

Will implement this.

You got it! Happy hacking 🎃


Update blog navigation pages

On the blog navigation page, add blog post pictures with excerpts

This is completed in staging and will be out in the next update


Add the ability to adjust the sector size while creating Disks

This fixes #133 .
As discussed on the issue above, per recommendation I added an additional parameter to the diskfs.Create() function that allows customization of sector size - currently only in the case of regular files.
Questions here:

Should we also override the sector size for block devices in this case? I wasn't sure so I left it to auto-detect always.
Naming. It's hard for me to find good names for the values!

Please be gentle, Go is still not very native for me.

Thanks! Adjusted!

Thanks @sil2100


RRuntimeError: Unable to find an inherited method for function ‘assay’ for signature ‘"dgCMatrix", "character"’

I'm trying to troubleshoot a bug associate with a package I'm working on with @LuckyMD (https://github.com/singlecellopenproblems/SingleCellOpenProblems). I tried using toy data and am getting this error
import anndata
import numpy as np
import scIB.preprocessing

adata = anndata.AnnData(np.random.normal(100,1, size=(3000,1000)))
scIB.preprocessing.normalize(adata)

raises
---------------------------------------------------------------------------
RRuntimeError                             Traceback (most recent call last)
<ipython-input-1-68c695ef7288> in <module>
      4
      5 adata = anndata.AnnData(np.random.normal(100,1, size=(3000,1000)))
----> 6 scIB.preprocessing.normalize(adata)

~/.local/lib/python3.8/site-packages/scIB/preprocessing.py in normalize(adata, min_mean)
    168     ro.globalenv['data_mat'] = adata.X.T
    169     ro.globalenv['input_groups'] = adata_pp.obs['groups']
--> 170     size_factors = ro.r(f'computeSumFactors(data_mat, clusters = input_groups, min.mean = {min_mean})')
    171     del adata_pp
    172

/usr/lib/python3.8/site-packages/rpy2/robjects/__init__.py in __call__(self, string)
    414     def __call__(self, string):
    415         p = rinterface.parse(string)
--> 416         res = self.eval(p)
    417         return conversion.rpy2py(res)
    418

/usr/lib/python3.8/site-packages/rpy2/robjects/functions.py in __call__(self, *args, **kwargs)
    195                 v = kwargs.pop(k)
    196                 kwargs[r_k] = v
--> 197         return (super(SignatureTranslatedFunction, self)
    198                 .__call__(*args, **kwargs))
    199

/usr/lib/python3.8/site-packages/rpy2/robjects/functions.py in __call__(self, *args, **kwargs)
    123             else:
    124                 new_kwargs[k] = conversion.py2rpy(v)
--> 125         res = super(Function, self).__call__(*new_args, **new_kwargs)
    126         res = conversion.rpy2py(res)
    127         return res

/usr/lib/python3.8/site-packages/rpy2/rinterface_lib/conversion.py in _(*args, **kwargs)
     42 def _cdata_res_to_rinterface(function):
     43     def _(*args, **kwargs):
---> 44         cdata = function(*args, **kwargs)
     45         # TODO: test cdata is of the expected CType
     46         return _cdata_to_rinterface(cdata)

/usr/lib/python3.8/site-packages/rpy2/rinterface.py in __call__(self, *args, **kwargs)
    622                     error_occured))
    623             if error_occured[0]:
--> 624                 raise embedded.RRuntimeError(_rinterface._geterrmessage())
    625         return res
    626

RRuntimeError: Error in (function (classes, fdef, mtable)  :
  unable to find an inherited method for function ‘assay’ for signature ‘"dgCMatrix", "character"’

I'm have the following package versions installed
anndata 0.7.4
scanpy 1.4.5.post3
rpy2 3.3.5
anndata2ri 1.0.3
scanorama 1.6
memory_profiler 0.57.0
networkx 2.4
igraph 0.8.2
louvain 0.6.1

As a note, this is reproducible with real data
from openproblems.tasks.label_projection.datasets import  zebrafish_labels
import scIB.preprocessing

adata = zebrafish_labels(test=False)
scIB.preprocessing.normalize(adata)

raises
---------------------------------------------------------------------------
RRuntimeError                             Traceback (most recent call last)
<ipython-input-16-d43d74662bad> in <module>
      3
      4 adata = zebrafish_labels(test=False)
----> 5 log_scran_pooling(adata)

~/.local/lib/python3.8/site-packages/openproblems/tools/utils.py in normalize(adata, obsm, obs, var, *args, **kwargs)
     22                 adata.X = adata.layers[func.__name__]
     23             else:
---> 24                 func(adata, *args, **kwargs)
     25                 adata.layers[func.__name__] = adata.X
     26

~/.local/lib/python3.8/site-packages/openproblems/tools/normalize.py in log_scran_pooling(adata)
     12     scprep.run.install_bioconductor("scran")
     13     # Normalize via scran-pooling with own clustering at res=0.5
---> 14     scIB.preprocessing.normalize(adata)
     15
     16     # Make lightweight

~/.local/lib/python3.8/site-packages/scIB/preprocessing.py in normalize(adata, min_mean)
    168     ro.globalenv['data_mat'] = adata.X.T
    169     ro.globalenv['input_groups'] = adata_pp.obs['groups']
--> 170     size_factors = ro.r(f'computeSumFactors(data_mat, clusters = input_groups, min.mean = {min_mean})')
    171     del adata_pp
    172

/usr/lib/python3.8/site-packages/rpy2/robjects/__init__.py in __call__(self, string)
    414     def __call__(self, string):
    415         p = rinterface.parse(string)
--> 416         res = self.eval(p)
    417         return conversion.rpy2py(res)
    418

/usr/lib/python3.8/site-packages/rpy2/robjects/functions.py in __call__(self, *args, **kwargs)
    195                 v = kwargs.pop(k)
    196                 kwargs[r_k] = v
--> 197         return (super(SignatureTranslatedFunction, self)
    198                 .__call__(*args, **kwargs))
    199

/usr/lib/python3.8/site-packages/rpy2/robjects/functions.py in __call__(self, *args, **kwargs)
    123             else:
    124                 new_kwargs[k] = conversion.py2rpy(v)
--> 125         res = super(Function, self).__call__(*new_args, **new_kwargs)
    126         res = conversion.rpy2py(res)
    127         return res

/usr/lib/python3.8/site-packages/rpy2/rinterface_lib/conversion.py in _(*args, **kwargs)
     42 def _cdata_res_to_rinterface(function):
     43     def _(*args, **kwargs):
---> 44         cdata = function(*args, **kwargs)
     45         # TODO: test cdata is of the expected CType
     46         return _cdata_to_rinterface(cdata)

/usr/lib/python3.8/site-packages/rpy2/rinterface.py in __call__(self, *args, **kwargs)
    622                     error_occured))
    623             if error_occured[0]:
--> 624                 raise embedded.RRuntimeError(_rinterface._geterrmessage())
    625         return res
    626

RRuntimeError: Error in (function (classes, fdef, mtable)  :
  unable to find an inherited method for function ‘assay’ for signature ‘"dgRMatrix", "character"’

So in version 3.11 scran requires an sce object as input. A pity, but okay... will test whether this is consistent in 3.10 as well on my end and then merge. Thanks @scottgigante !

I'd make a sassy comment but you already know how I feel about tools not accepting standard data formats as input ;)

Haha, i don't mind as long as it's consistent tbh... but yeah :/

BiocGenerics::sizeFactors(
  scran::computeSumFactors(
    SingleCellExperiment::SingleCellExperiment(list(
      counts=matrix(runif(10, 100, 100), ncol=10)
    ))
  )
)

Why the BiocGenerics::sizeFactors() wrapping?

Otherwise it returns the sce rather than the sizeFactors vector.

So it turns out I'm running bioconductor version 3.9. And:
size_factors = ro.r(f'computeSumFactors(SingleCellExperiment::SingleCellExperiment(list(data_mat)), clusters = input_groups, min.mean = {min_mean})')

give me the error:
RRuntimeError: Error in assay(x, i = assay.type) :
  'assay(<SingleCellExperiment>, i="character", ...)' invalid subscript 'i'
'counts' not in names(assays(<SingleCellExperiment>))
Calls: <Anonymous> ... .local -> .computeSumFactors -> ncol -> assay -> assay

I really love bioconductor version updates :/

Can you try SingleCellExperiment::SingleCellExperiment(list(counts=data_mat))?

damn, I thought that's what I was doing the whole time. So it seems as though they introduced a new function calculateSumFactors() which uses the data matrix input. computeSumFactors() now only takes SingleCellExperiment objects where it did both before. So either, move to sce for everything, or write a bioconductor version check to see if it's >=3.11.

Okay, works now. Thanks a lot for catching that typo! The error message wasn't really helping and I clearly wasn't looking at what I put in there :/

I just saw you had a PR for this already... no idea what I was doing there...

Lol no worries. Thanks for acting on this so quickly!
On Fri, 11 Sep 2020, 12:27 pm MalteDLuecken<EMAIL_ADDRESS>wrote:

I just saw you had a PR for this already... no idea what I was doing
there...
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/theislab/scib/issues/141#issuecomment-691193042, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ACA3DX6UBL6UA2BFAJ6TCE3SFJFW3ANCNFSM4QDPGNZQ
.

Just merged your PR #157 after reverting mine... credit where credit's due...

hah thanks :D


[manifests] Is there a way to specify dependency's version?

I didn't find any mention of such useful thing in the documentation file

It will come very soon... Take a look at PRs #11758.


net.* performance

With SO_REUSEPORT we should be able to run multiple listeners for incoming messages and handle those. This comes with a couple of ifs:

I don't yet know how to do this
Maybe Go's net.* is not fast enough - fix that?
There might still be lots of low hanging Go userspace code in CoreDNS and Go DNS, so that we don't need to go this route.

See some experimental stuff from @asergeyev
https://github.com/asergeyev/go-toys/tree/master/reuse which looks very interesting.


Add onOpenAutoFocus prop to Dialog component

Could be nice, but I worry about putting these props on the root component rather than just exposing each part of the Dialog instead.

Can you merge this until then?

Can you merge this until then?

No. One-off requests like this is how a library gets very messy. Let's solve the base problem.

👍


Fix TranslatedText translate default behavior

TranslatedText translate used to just returns key if not present if map of/ given language code.
Now it attempts to look for the key with default language code of en_us, returns key if that also fails.

I'll publish releases when I've got time (probably next week somewhere)


my Card

@grvv  Looks like there is a simple conflict with the commit .Would you like to resolve it ? If you don't then i would do it for you ! Its just a simple { bracket conflict .

done!

great :) .Thanks for contributing


Package scrhack Warning: unknown \newfloat definition found! (KOMA-script)

Prerequisites

[ X ] Put an X between the brackets on this line if you have done all of the following:

Check that you are using the latest version of kaobook.
Check that your issue isn't already filed: https://github.com/fmarotta/kaobook/issues

Description
This might be specific to KOMA-script, but I am unfamiliar with it, and I'm encountering it through kaobook.
running latexmk -pdf  mwe.tex with the MWE below succeeds, but there is a warning:
[...]
(/usr/local/texlive/2021/texmf-dist/tex/latex/koma-script/scrhack.sty
(/usr/local/texlive/2021/texmf-dist/tex/latex/koma-script/float.hak

Package scrhack Warning: unknown \newfloat definition found!
(scrhack)                Maybe you are using a unsupported float version.

) (/usr/local/texlive/2021/texmf-dist/tex/latex/koma-script/floatrow.hak)
(/usr/local/texlive/2021/texmf-dist/tex/latex/koma-script/listings.hak)
[...]

Minimal Working Example
[You can use the following document to help you create a minimal working example.]
\documentclass{kaobook}

\usepackage{mwe}

\begin{document}
\mainmatter

\begin{figure}
	\centering
	\includegraphics[width=0.5\columnwidth]{example-image}
	\caption{
		A figure caption
	}
\end{figure}

\end{document}

Additional Information
$ pdflatex --version
pdfTeX 3.141592653-2.6-1.40.22 (TeX Live 2021)
kpathsea version 6.3.3
Copyright 2021 Han The Thanh (pdfTeX) et al.
There is NO warranty.  Redistribution of this software is
covered by the terms of both the pdfTeX copyright and
the Lesser GNU General Public License.
For more information about these matters, see the file
named COPYING and the pdfTeX source.
Primary author of pdfTeX: Han The Thanh (pdfTeX) et al.
Compiled with libpng 1.6.37; using libpng 1.6.37
Compiled with zlib 1.2.11; using zlib 1.2.11
Compiled with xpdf version 4.03

It doesn't seem to be kao-specific, as it occurs even with this document, which uses scrbook:
\documentclass{scrbook}

\usepackage{floatrow}
\usepackage{scrhack}

\begin{document}

\end{document}

It is enough to load floatrow and scrhack at the same time to trigger the warning. scrhack tries to make floatrow compatible with KOMAScript, but it supports only some versions of the latter. Probably it supports only older versions of floatrow, so one possibility would be to downgrade floatrow.

Thanks! I opened an issue here: https://sourceforge.net/p/koma-script/tickets/9/

I seems that kaobook loads scrhack after floatrow. It is recommended to load it before loading any of the packages, that should be patched.
Nevertheless, loading float.hak if floatrow has already been loaded is a bug in scrhack that will be fixed in KOMA-Script 3.34.

Thank you! I was confused by this post on TeX.SE. Now scrhack is loaded before any of the patched packages.


[Feature]: Manual override during leaderboard testing

Description
In current testing environment there is no way to intervene in vehicle action.
Due to non perfect implementation of perception, action and planning the vehicle often gets stuck on obstacles.
Also in different discussions with other students the wish for "reproducible" scenarios came up.
I therefore suggest implementing a manual override functionality. The operator should be able to maneuver the vehicle manually.
In best case scenario this could be done by keyboard or by a game-pad. A simpler solution would be to implement a qt-panel for rviz for simple steering and throttle.
Definition of Done

[ ]  vehicle can be maneuvered manually

As described in #420 this currently relies on catkin_make beeing executed manually if any changes want to be made. Probably should not be merged until the mentioned issue is resolved.

Current implementation (7edac9d120777744023958505a10ae3fefb1adeb)

There is also a implementation from carla:
https://carla.readthedocs.io/projects/ros-bridge/en/latest/carla_manual_control/
In leaderdboard testing I couldn't get it to work yet. The pygame window opens and pressing 'B' will change Modes. There are however to topics beeing published to vehicle_cmd.

There is also a implementation from carla: https://carla.readthedocs.io/projects/ros-bridge/en/latest/carla_manual_control/
In leaderdboard testing I couldn't get it to work yet. The pygame window opens and pressing 'B' will change Modes. There are however to topics beeing published to vehicle_cmd.

If you launch with the carla manual control, you would have to disable the original vehicle control command or change the topic to a dead sink. Remapping the topic to a new dead sink topic should be possible with a dedicated .launch file.


Add exclude tags and operations attributes to openapi annotation

Purpose

Fix issues with tag and operation filtering. Add the extra fields excludeTags and excludeOperations to Opeanapi annotation. This use to filter the tags and operations that exclude validating.

Samples

@openapi:ServiceInfo {
contract: "resources/petstore.yaml",
excludeTags: ["user"]
excludeOperations: ["getOrderById"]
}

Check List

[ ] Read the Contributing Guide
[ ] Updated Change Log
[ ] Checked Tooling Support (#)
[ ] Added necessary tests

[ ] Unit Tests
[ ] Spec Conformance Tests
[ ] Integration Tests
[ ] Ballerina By Example Tests

[ ] Increased Test Coverage
[ ] Added necessary documentation

[ ] API documentation
[ ] Module documentation in Module.md files
[ ] Ballerina By Examples

Codecov Report

Merging #23573 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #23573   +/-   ##
=======================================
  Coverage   14.59%   14.59%
=======================================
  Files          51       51
  Lines        1411     1411
  Branches      219      219
=======================================
  Hits          206      206
  Misses       1189     1189
  Partials       16       16

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c2c976f...3c4d959. Read the comment docs.


Pointless

What I want Is the future bank holidays..
for next ten years.. to populate my db..

def update_bank_holidays(y):

    ## new years day
    d = datetime.date(year=y, month=1, day=1)
    if d.weekday() > 4:
        da = 1 if d.weekday() == 6 else 2
        d.weekday(), da, d.weekday() - 5
        d = d + datetime.timedelta(days=da)
    hols_update(d, "New Year's Day")

    ## Easter
    d = easter.easter(y)  # easter sunday
    hols_update(d + datetime.timedelta(days=-2), "Good Friday")
    hols_update(d + datetime.timedelta(days=+1), "Easter Monday")

    ## Early May Bank Holiday - first monday on may (maypole)
    if y == 2020 or y == 1995:
        # VE Day UK gov.tweak
        d = datetime.date(year=y, month=5, day=8)
        ll = "VE Day Bank Holiday"
    else:
        d = datetime.date(year=y, month=5, day=1)
        if d.weekday() != 0:
            d = d + datetime.timedelta(days=7 - d.weekday())
        ll = "Early May Bank Holiday"
    hols_update(d, ll)

    ## Spring Bank Holiday - last monday on may
    d = datetime.date(year=y, month=5, day=31)
    if d.weekday() != 0:
        d = d + datetime.timedelta(days=d.weekday() * -1)
    hols_update(d, "Spring Bank Holiday")

    ## Summer Bank Holiday - last monday on aug
    d = datetime.date(year=y, month=8, day=31)
    if d.weekday() != 0:
        d = d + datetime.timedelta(days=d.weekday() * -1)
    hols_update(d, "Summer Bank Holiday")

    ## Xmas
    d = datetime.date(year=y, month=12, day=25)
    if d.weekday() > 4:
        d = d + datetime.timedelta(days=7 - d.weekday())
    hols_update(d, "Christmas Day")

    ## Boxing
    d = d + datetime.timedelta(days=+1)
    if d.weekday() > 4:
        d = d + datetime.timedelta(days=7 - d.weekday())
    hols_update(d, "Boxing Day")

hi @pedromorgan! you're right, it seems that UK Government Digital Service only provides bank holiday information until 2021. this library simply exposes their data (from https://www.gov.uk/bank-holidays), so i'm not sure what we could do really. would it make sense to you to have a way of estimating when bank holidays will be until they're confirmed by the government?

@ushkarev The rules for setting a bank holiday are clear, as in code above.. Only snag is if guv change some dates, eg VE day


Interface cleanup

Separate basic, more-, oneway- and upgrade-calls in client interface
Use std::error_code for varlink error type to always only return ["parameters"] of the reply
Code formatting
GCC 7 compatibility
Cleanup of inner stream processing

Merging this as it's the current de-facto state


Staging - User is never asked if they came by bike.

Repro steps

List yourself as a biker
Start you first session that day

Expected Result
"Did you come by bike today? Please reply using /yes or /no"
Actual Result
Even after user is listed with /biker, the question is never asked
However it is possible to add yourself as biker by doing /yes independently of being asked or not.
System Info

crontab job wasn't installed on staging server yet


Add airport styling

Add styling for airports. See #188.
Zoomed out
At zoom level <=10 and when the long name doesn't fit, the IATA or ICAO code is shown.

Large airport

Smaller airport

Very small airport

Gate numbers
Bonus points if you recognize this airport

Remaining Issues

Military air bases are not styled differently--see #232.
Should international airports stand out more somehow?

I think the outline without runways looks a bit empty. How about having the runways show up at the same zoom as the outline? They're pretty big features and I think they'd look good at low zoom.
Likewise, taxiways show up at a somewhat high zoom and airports look empty next to residential streets. What if we bump taxiways down to where runways are currently showing up?

I do like how everything looks overall, though! Fantastic work.

Agreed, I think that looks good.

I really like the use of IATA codes when zoomed out, and I think that combined with the runways will make it clear these represent airports at low zoom, even without an airplane icon. I just want to point out that this is the first non-car transport-related rendering, and also the first use of purple on the map, so this might set a precedent for purple being a transportation-related color (similar to Carto). Not that this is a bad thing (I think it looks good), but something to consider going forward, especially if purple was being considered for any other large features.

I like the stroke width and fill transparency, but a different color would help distinguish aeroways from boundaries. The shades of purple in this PR are very close to the ones used for administrative boundaries, which can interact with runways and taxiways confusingly:

Changed to a more pink color to avoid confusion with boundaries:

And made airport labels without IATA and ICAO codes not show up until higher zoom levels:

OpenMapTiles doesn't expose the runway surface tag.

The pink doesn't feel quite right to me.  I think the original purple shading was better, but there may be other colors that work as well.
The intersection of boundaries and airports feels like a bit of an esoteric case to me.  Perhaps we can monkey with the draw order and not necessarily show all parts of the boundary drawing.  I'm not sure why a map user would care about the precise position of a town boundary through an airport anyways.
I'm also thinking that when we're down to the IATA codes that it should have an airplane icon, though I'm happy to defer that to a future issue if the author isn't graphically inclined.

I'd have to agree with Brian that the purple "looked better". This style already uses a lot of red tones and I'm wondering if a red tone would also be appropriate for military areas. I like the idea of using a consistent color-range for public transit broadly (like busways, airports, ferry terminals, train & bus stations). I don't know that choosing a public-transport color needs to block this PR, but maybe we can take an initial stab (purple or otherwise) and then shift it later as more features get added.

I just noticed that highway tunnels are rendering on top of runways and taxi-ways (LAV on localhost, ORD on localhost):

These lines should be moved after the highway tunnels:
  lyrAeroway.runway,
  lyrAeroway.runwayArea,
  lyrAeroway.taxiway,

Reverted the color and added icons:

Looks like the only question is boundary collisions.  How would people feel about putting airport areas on top of boundaries?
Also bear in mind that at some point I anticipate putting labels on boundaries to indicate what territory is on either side.  This would be awfully cluttered to have a bunch of boundary labels running through ATL.

France, fix your administrative boundaries...

If we do that for runways it should be done for roads also, and IMO that would only really make sense at the highest zooms.

Although I like the skumorphic idea, I think it would make more sense in a map style going for a realistic look and feel.   The highway map aesthetic we are going for in this style.is more abstract and symbolic.

I like the look of this so far.  Definitely also a fan of the cooler purple over the warmer pink.  However, a boundary running through an airport does look rather muddled with the similar shade of purple.  I wonder if desaturating the purple further or even moving entirely to a shade of gray would work.
I noticed that the North Carolina state highway map shows the runway structure of airports in the city insets.  White with a black outline is used.  Some examples below.  If anyone else has maps that show airport details like this, please share.

Exploring grayscale as suggested:

I played around with changing the layer ordering to see what we could do about airfield/boundary collisions.  The results weren't great.  First, I tried moving the boundary casing below the airfield fill.  This had the effect of keeping the boundaries on top of the airfield, but making them incoherent as you lose the state/county/city distinctions.  I also tried moving the boundary casing and fill below the airfield, and this kept the airfield intact no problem but it creates a love triangle between airfield, water, and boundary.  Airfield wants to be above boundary, boundary wants to be above water, and water wants to be above airfield (for example, small retaining ponds and so forth).  And that's the ball game.
  So we probably have to live with those collisions unless someone has a magic sauce solution here.
I'd propose that we come to a consensus on what color scheme we like and push ahead.

What about grayscale runways/taxiways and purple airport boundaries? I like the grayscale runways, and maybe that alone is enough to make them conflict with administrative boundaries less.

More distinguishable from boundaries, harder to tell what's a road

Of note, boundaries on top of airports is silly on osm-carto as well :-D
https://www.openstreetmap.org/search?query=CDG airport#map=14/49.0135/2.5557

The main issue with the screenshot in https://github.com/ZeLonewolf/openstreetmap-americana/pull/318#issuecomment-1126258309 is that we chose to put tertiary roads in gray, hence the conflict.  If we could undo that, we might have this nailed.

Go team!

I really like the color work from @zekefarwell in https://github.com/ZeLonewolf/openstreetmap-americana/pull/318#discussion_r872921754.  I think we have a winner there unless anyone objects.
I also like the skeuomorphic runway lines that @claysmalley demonstrated, but only at the highest zoom where it's clear that we're looking at dashed lines.  But, we can do that as a separate PR
I agree with the suggestion by @claysmalley for square end caps https://github.com/ZeLonewolf/openstreetmap-americana/pull/318#discussion_r872984695

Thanks for spearheading this @AnonymousAlligatorOSM, this is the first significant small-area feature and it will set the stage for other area features.

I'm away from a computer and can't check myself, but just be sure that
there runway dashes don't conflict with unpaved roads. I can't quite tell
from the screenshots posted if that is an issue or not.
On Sat, May 14, 2022, 7:36 PM Brian Sperlongano @.***>
wrote:

@.**** approved this pull request.
I think this is ready to ship in this state. We might consider some
improvements at zoom 10 as far as which features to show for what types of
airports, but I think those are tweaks that can be separately investigated.
—
Reply to this email directly, view it on GitHub
https://github.com/ZeLonewolf/openstreetmap-americana/pull/318#pullrequestreview-973107082,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAGFGT27ZBAF4MONIRI3QLVKA2HBANCNFSM5VM54PUQ
.
You are receiving this because you commented.Message ID:
@.***>

@adamfranco the runway dashes are not included in this PR.  @claysmalley only posted samples.


19 task 3 test cases has some fails

Probably this will make any progress with #19

Probably the error was the wrong status code. It should be 400 for error responses, but the implementation was sending 200.


Add typst language support

Added support for Typst

I built from source and tested it. File extension successfully is being recognized, but Icon is not showing properly. But I cannot find the reason.


Configure CI for PRs

[x] Run go test
[x] Run mkdocs build

Completed by 0e6a5549779c834a2d6bc8a67452c85a8c372a52.


DEV-57/58 Chore: axios zustand

https://linear.app/poklist/issue/DEV-57/axios-and-zustand
https://linear.app/poklist/issue/DEV-58/component-alert
Description
Replace this text with a detailed description of what this PR does.
Related issue #(ticket number) / None
Type of Change

[ ] Bug fix (non-breaking change that fixes an issue)
[ ] New feature (non-breaking change that adds functionality)
[ ] Breaking change (fix or feature that may cause existing functionality to not work as expected)
[ ] This change requires a documentation update

Affected Pages

[ ] Home
[ ] Profile and Settings
[ ] Other (please specify)

Related Project Task (e.g., Redmine)
(If there are related tasks, please provide links; if not, delete this line and write "None")
Checklist

[ ] I have manually tested this locally to ensure functionality and stability
[ ] I have commented my code, especially in hard-to-understand areas
[ ] I have updated the documentation, or my changes do not require documentation updates
[ ] My PR has a clear title and description

https://linear.app/poklist/issue/DEV-57/axios-and-zustand https://linear.app/poklist/issue/DEV-58/component-alert
Description
Replace this text with a detailed description of what this PR does.
Related issue #(ticket number) / None
Type of Change

[ ]  Bug fix (non-breaking change that fixes an issue)
[ ]  New feature (non-breaking change that adds functionality)
[ ]  Breaking change (fix or feature that may cause existing functionality to not work as expected)
[ ]  This change requires a documentation update

Affected Pages

[ ]  Home
[ ]  Profile and Settings
[ ]  Other (please specify)

Related Project Task (e.g., Redmine)
(If there are related tasks, please provide links; if not, delete this line and write "None")
Checklist

[ ]  I have manually tested this locally to ensure functionality and stability
[ ]  I have commented my code, especially in hard-to-understand areas
[ ]  I have updated the documentation, or my changes do not require documentation updates
[ ]  My PR has a clear title and description

你沒有勾選XD


default use large_utf8 in arrow to avoid 2G size limit

This commit is worth adding to the changelog.md , please add it

added in CHANGELOG.md:

Optimized to support billion-level PSI scenarios.


[Enhancement] Icon / Image Area

While the badges are useful, users may also want to include some sort of icon or image alongside or instead of the badge to denote the control's meaning (e.g. an exclamation point instead of "Warning" text).  My proposed solution is to include a ContentControl that sits to the left of the badge in an Auto-sixing column that would allow users to insert a graphic (e.g. and icon or perhaps a logged-in user image).
If it's agreed upon, I will mock it up and submit a PR for approval.

Sorry for late response. Sound good to me 👍


fix shiftclicking craft bug

This not only fixes shiftcrafting not working correctly, but also caches the last recipe similar to fastworkbench so repeated shift crafts don't stall the server.

Thanks!


Datasette.render_template() method

Refs #577

Needs documentation.

More importantly, I need to prototype up a plugin that actually uses this method - probably datasette-upload-csvs - to make sure it serves its purpose.

It doesn't make sense for extra body scripts to be available for all template renders (including from plugins) but extra CSS and JS to only be handled by the BaseView render method - especially since extra body scripts are likely to depend on extra JS files.

I'm going to ship this without documentation, get a couple of plugins out that use it and then add documentation once I'm confident it's right.

See #576 for existing ticket covering Datasette class documentation.


feat(storage): log tx opening location

To be able to catch the leaked transactions, we need to know their origin. Having a caller tracked up to the provider and the transaction ID should be enough.
2024-01-08T17:03:06.811863Z DEBUG storage::db::mdbx: Transaction opened caller=/Users/shekhirin/Projects/reth/crates/storage/provider/src/providers/database/mod.rs:250:14 id=305 mode=read-only
2024-01-08T17:03:06.817450Z  INFO reth::commands::node::events: Executing stage pipeline_stages=3/13 stage=Bodies checkpoint=222000 target=695895 stage_progress=31.90%
2024-01-08T17:03:06.817483Z DEBUG storage::db::mdbx: Transaction opened caller=/Users/shekhirin/Projects/reth/crates/stages/src/pipeline/mod.rs:351:53 id=306 mode=read-write
2024-01-08T17:03:07.006589Z  INFO reth::commands::node::events: pipeline_stages=3/13 stage=Bodies checkpoint=223000 target=695895 stage_progress=32.05% Stage committed progress
2024-01-08T17:03:07.020313Z DEBUG storage::db::mdbx: Transaction opened caller=/Users/shekhirin/Projects/reth/crates/storage/provider/src/providers/database/mod.rs:432:14 id=306 mode=read-only

The runtime cost of #[track_caller] should be negligible. No stack walk happens during the caller retrieval, and instead the caller is passed as an additional argument to the function. See https://rustc-dev-guide.rust-lang.org/backend/implicit-caller-location.html#generating-code-for-track_caller-callees for more information.

concerned about slowdown during Execution stage, I see a lot of logs about opening new read-only transactions. Investigating.


It's Clippy time!

Compliance with all default hard errors and crates that are explicitly
using #![deny(warnings)].
There are some legitimate warnings like println!("") -> println!() but
they aren't considered in this PR.

To remove all hard errors, more research is needed, specially inside the declarative decl_runtime_apis! macro. Nevertheless, the ones covered in this PR are a great start IMO

@c410-f3r you need to merge master.

@bkchr Merged

Ty

Thanks


EPIC: Preventing and preparing for non-determinism between app versions

Currently, we have minor version bump tests in knuu. This test is great because it performs a simple test for non-determinism between minor versions. However, we likely need to either expand on this test or create new tests that expand on what portions of the state-machine and block validity rules are covered. While not required, ideally we would be able to run some tests upon each push, or at least after being merged to main.

[x] https://github.com/celestiaorg/celestia-app/issues/2414
[x] https://github.com/celestiaorg/celestia-app/issues/3540

Unless we are 100% confident that the tests cover every case, then we also need to be prepared to debug non-determinism issues.

[ ] https://github.com/celestiaorg/cosmos-sdk/issues/397
[ ] https://github.com/celestiaorg/cosmos-sdk/issues/398

closing as completed


Add colons to unhandled messages first param

client.command_handler.addHandler("ERR_UNKNOWNCOMMAND", (cmd) => { /* emit error here */ });

I must be missing something, but that's the reason for this, sorry?

dont understand your comment @astorije; it's a really easy fix that @xPaw commented

@MaxLeiter I think he's missing an explaining what this issue wants.

ah, the error should probably be ignore: unknown command, rather than ignore Unknown command

How about switching it around to be Unknown command: ignore? That feels more natural to me.


Deleting SIM contacts with name but no number or with number but no name always fails

Problem
The SimContactsDelete API fails to delete SIM contacts that either have;

non-null name and null number
null name and non-null number

This is due to the following selection clause;
"tag='${name}' AND number='${number}'"

The tag and number should not be present if the value on the right hand side of the = is null. Having tag='null' or number='null' causes the delete to fail.
Solution
Exclude tag or number from the selection clause if the right hand side value is null.

This is included in 0.2.4!


FAILED! => {"censored": ... 'no_log: true' was specified for this result", "changed": true}

Description of problem
Task dokku ssh-keys:add for user outputs the error:
fatal: [myhost]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true}

How reproducible
Run this role with a valid user and a valid ssh key.
Steps to Reproduce

Create a user and a ssh-key in dokku host
Set dokku_users in playbook variables:

vars:
    dokku_users:
      - name: "My Name"
        username: myusername
        ssh_key: "{{lookup('file', '~/.ssh/id_rsa.pub')}}"

Run the playbook using this role

Actual Results
TASK [dokku_bot.ansible_dokku : add ssh key for user {{ item.username }}] **************************************************************
included: /home/leandro/.ansible/roles/dokku_bot.ansible_dokku/tasks/ssh-key.yml for <IP_ADDRESS> => (item={'name': 'Leandro Ramos', 'username': 'leandro', 'ssh_key': 'ssh-rsa a lot of chars here'})

TASK [dokku_bot.ansible_dokku : store sha256 hash of ssh key for user leandro] *********************************************************
ok: [<IP_ADDRESS>]

TASK [dokku_bot.ansible_dokku : dokku ssh-keys:add for user leandro] *******************************************************************
fatal: [<IP_ADDRESS>]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true}

Expected Results
No fatal error.
Environment Information

dokku report APP_NAME output
-----> uname: Linux dokku 5.10.0-14-amd64 #1 SMP Debian 5.10.113-1 (2022-04-29) x86_64 GNU/Linux
-----> memory:
                      total        used        free      shared  buff/cache   available
       Mem.:           1442         188         251           0        1002        1103
       Swap:            974           0         974
-----> docker version:
       Client: Docker Engine - Community
        Version:           20.10.16
        API version:       1.41
        Go version:        go1.17.10
        Git commit:        aa7e414
        Built:             Thu May 12 09:17:39 2022
        OS/Arch:           linux/amd64
        Context:           default
        Experimental:      true

       Server: Docker Engine - Community
        Engine:
         Version:          20.10.16
         API version:      1.41 (minimum version 1.12)
         Go version:       go1.17.10
         Git commit:       f756502
         Built:            Thu May 12 09:15:42 2022
         OS/Arch:          linux/amd64
         Experimental:     false
        containerd:
         Version:          1.6.4
         GitCommit:        212e8b6fa2f44b9c21b2798135fc6fb7c53efc16
        runc:
         Version:          1.1.1
         GitCommit:        v1.1.1-0-g52de29d
        docker-init:
         Version:          0.19.0
         GitCommit:        de40ad0
-----> docker daemon info:
       Client:
        Context:    default
        Debug Mode: true
        Plugins:
         app: Docker App (Docker Inc., v0.9.1-beta3)
         buildx: Docker Buildx (Docker Inc., v0.8.2-docker)
         scan: Docker Scan (Docker Inc., v0.17.0)

       Server:
        Containers: 1
         Running: 0
         Paused: 0
         Stopped: 1
        Images: 21
        Server Version: 20.10.16
        Storage Driver: overlay2
         Backing Filesystem: extfs
         Supports d_type: true
         Native Overlay Diff: true
         userxattr: false
        Logging Driver: json-file
        Cgroup Driver: systemd
        Cgroup Version: 2
        Plugins:
         Volume: local
         Network: bridge host ipvlan macvlan null overlay
         Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
        Swarm: inactive
        Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
        Default Runtime: runc
        Init Binary: docker-init
        containerd version: 212e8b6fa2f44b9c21b2798135fc6fb7c53efc16
        runc version: v1.1.1-0-g52de29d
        init version: de40ad0
        Security Options:
         apparmor
         seccomp
          Profile: default
         cgroupns
        Kernel Version: 5.10.0-14-amd64
        Operating System: Debian GNU/Linux 11 (bullseye)
        OSType: linux
        Architecture: x86_64
        CPUs: 2
        Total Memory: 1.409GiB
        Name: dokku
        ID: QDBG:VLYU:3QPR:46DQ:EDIW:CMLX:IP2K:XC5H:BXTO:VJ56:UVGD:U4CX
        Docker Root Dir: /var/lib/docker
        Debug Mode: false
        Registry: https://index.docker.io/v1/
        Labels:
        Experimental: false
        Insecure Registries:
         <IP_ADDRESS>/8
        Live Restore Enabled: false

-----> git version: git version 2.30.2
-----> sigil version: 0.8.1build+dce1c9e
-----> herokuish version:
       herokuish: 0.5.34
       buildpacks:
         heroku-buildpack-multi     v1.0.0
         heroku-buildpack-ruby      v236
         heroku-buildpack-nodejs    v189
         heroku-buildpack-clojure   v87
         heroku-buildpack-python    v205
         heroku-buildpack-java      v69
         heroku-buildpack-gradle    v36
         heroku-buildpack-scala     v91
         heroku-buildpack-play      v26
         heroku-buildpack-php       v205
         heroku-buildpack-go        v160
         heroku-buildpack-nginx     v16
         buildpack-null             v3
-----> dokku version: dokku version 0.27.1
-----> plugn version: plugn: 0.8.2build+1b2e2b4
-----> dokku plugins:
         00_dokku-standard    0.27.1 enabled    dokku core standard plugin
         20_events            0.27.1 enabled    dokku core events logging plugin
         app-json             0.27.1 enabled    dokku core app-json plugin
         apps                 0.27.1 enabled    dokku core apps plugin
         builder              0.27.1 enabled    dokku core builder plugin
         builder-dockerfile   0.27.1 enabled    dokku core builder-dockerfile plugin
         builder-herokuish    0.27.1 enabled    dokku core builder-herokuish plugin
         builder-null         0.27.1 enabled    dokku core builder-null plugin
         builder-pack         0.27.1 enabled    dokku core builder-pack plugin
         buildpacks           0.27.1 enabled    dokku core buildpacks plugin
         certs                0.27.1 enabled    dokku core certificate management plugin
         checks               0.27.1 enabled    dokku core checks plugin
         common               0.27.1 enabled    dokku core common plugin
         config               0.27.1 enabled    dokku core config plugin
         cron                 0.27.1 enabled    dokku core cron plugin
         docker-options       0.27.1 enabled    dokku core docker-options plugin
         domains              0.27.1 enabled    dokku core domains plugin
         enter                0.27.1 enabled    dokku core enter plugin
         git                  0.27.1 enabled    dokku core git plugin
         logs                 0.27.1 enabled    dokku core logs plugin
         network              0.27.1 enabled    dokku core network plugin
         nginx-vhosts         0.27.1 enabled    dokku core nginx-vhosts plugin
         plugin               0.27.1 enabled    dokku core plugin plugin
         proxy                0.27.1 enabled    dokku core proxy plugin
         ps                   0.27.1 enabled    dokku core ps plugin
         registry             0.27.1 enabled    dokku core registry plugin
         repo                 0.27.1 enabled    dokku core repo plugin
         resource             0.27.1 enabled    dokku core resource plugin
         run                  0.27.1 enabled    dokku core run plugin
         scheduler            0.25.6 enabled    dokku core scheduler plugin
         scheduler-docker-local 0.27.1 enabled    dokku core scheduler-docker-local plugin
         scheduler-null       0.27.1 enabled    dokku core scheduler-null plugin
         shell                0.27.1 enabled    dokku core shell plugin
         ssh-keys             0.27.1 enabled    dokku core ssh-keys plugin
         storage              0.27.1 enabled    dokku core storage plugin
         trace                0.27.1 enabled    dokku core trace plugin
 !     App nextcloud does not exist
2022/05/18 13:22:35 exit status 20

How (deb/make/rpm) and where (AWS, VirtualBox, physical, etc.) was Dokku installed?:

deb
Virt-Manager (KVM, QEMU)

Additional information
No additional information, app was not deployed.

Hi @leandroramos , thanks for the report.
I'm currently no longer using dokku and my time to work on this role is very limited.
The continuous integration tests of this role include a user being set up
https://github.com/dokku/ansible-dokku/blob/7e047c63395bfca4ede366a8c1b59ceeda023472/molecule/default/converge.yml#L21-L24
i.e. the issue you saw either is specific to your setup or it is an issue that appeared since the CI tests were last run.
You could try editing the role here
https://github.com/dokku/ansible-dokku/blob/7e047c63395bfca4ede366a8c1b59ceeda023472/tasks/ssh-key.yml#L11
and see whether the log output provides any hints.

Thank you, @ltalirz. It was, in fact, an issue with my setup. I recreated the virtual machine and everything worked fine.


document.validate fails if ds:x509certificate is not embedded in the SAML Response

We are building an app that is going to do SSO with Ping Federate using ruby-saml, and I am told that the IdP is emitting valid SAMLv2 responses. However, the responses do not contain  the ds:x509certificate stanza. The certificate is available in the metadata.xml file, and it matches the signature in the SAML response. I am not sure why the cert would have to be embedded in the response, when it is already available in the metadata. We do have an option for "anchored" response that does include the certificate. So this is partially to help someone else in the future who runs into this same issue and partially to figure out why the certificate would be required in the SAML response. Our initial problem is solved, but it has left me wondering why. Hopefully someone can help.
~tommy

Quote from: http://stackoverflow.com/questions/1703301/saml-why-is-the-certificate-within-the-signature

SAML responses do not require including the public key for that signature. Section 5.4.5 of the SAML2  spec states "XML Signature defines usage of the ds:KeyInfo element. SAML does not require the use of ds:KeyInfo, nor does it impose any restrictions on its use. Therefore, ds:KeyInfo MAY be absent." You can verify the signature if the public key has been provided to you through other means, e.g. stored in your local certificate store prior to implementing the SAML consumer. –  Sam.Rueby Nov 5 '14 at 16:01

The ruby-saml gem expects a Signed SAML Response, This Signature may carry a cert.
In our signature validation checking we extract the x509certificate from the SAMLResponse and compare it with the settings.idp_cert or the settings.idp_fingerprint.
Most of the Service Providers implementations works in this way.
The main reason is that most SAML implementations delegate on XMLSignature implementations that requires the KeyInfo element, but I'm agree that the spec says that SAML does not require it (. Section 5.4.5).
Other reason for doing that is for avoiding DOS attacks (saml-sec-consider 5.2.1 Denial of Service).
Is fast/easy discard a SAMLResponse without cert or an invalid cert than verify the signature.

@pitbulk would you consider accepting a pull request that makes it optional in the response document if settings.idp_cert is defined?

@TimMoore, I posted why we implemented all our toolkits requiring the x509certificate data.
I think is easier configuring the Identity Provider side to inject the ds:x509certificate.
99% of Identity Provider implementations do that.

In our case, I was told that "it is a valid SAMLv2 response" by the IdP admins, which according to what I found above, is correct. They acted like including the certificate made it an "anchored" SAML assertion, which changed something about the validation. I basically just responded with a link to the code and a "text" description of the algorithm, which seemed to satisfy them. They "checked a checkbox" (?) and everything started working. It seems that, by default, they do not normally include the x509 cert in their assertions.  Maybe not. I would say that, in order to be "compliant" with the standards as written, it should be coded to be optional. I don't think it would require significant code changes. Unfortunately, we are also off on #206 as well, due to jRuby requirements, so we are maybe not the best test case, but I would be willing to try if someone submitted it.
In any case, this issue should help someone else who encounters a similar issue, even if we don't change anything about the validation. :)

@TJM I appreciate it. Right now there are other priorities in the toolkit development process, but we will consider to add a settings to allow SAMLResponse without fingerprint, but this specific change means several changes on the code.

One thing we noticed if you generate the metadata xml before you set the idp_cert in settings, the generated metadata claims that signed response is not required.
so, if the IDP admin generates the metadata xml first (before setting the idp_cert), the response won't be signed. But once they changed the configure to respond with signed request everything works.
instead of relaxing SAML response to accept unsigned request, maybe always generate the metadata to require signed response.

@patcheng, The toolkit reject any unsigned request.

I think that's the issue.
When processing the response, the toolkit rejects anything that's unsigned. So the Metadata XML should always be WantAssertionsSigned="true"
But in https://github.com/onelogin/ruby-saml/blob/master/lib/onelogin/ruby-saml/metadata.rb#L29
WantAssertionsSigned is set to true only when settings.idp_cert_fingerprint or settings.idp_cert is set. In our workflow at least, user will most likely generates the metadata before setting the idp certs.

@patcheng, We can have the WantAssertionsSigned=false and the IdP then add the Signature to the whole SAML Response message.

For the record, we've also hit this problem --- a customer's IdP is not including the certificate in each SAML response, only in their metadata.
First, I'd like to say thanks for providing this great library and the excellent documentation --- we definitely would not have gotten this far without it! However, we are a bit stuck without this option, since we don't have any control over what the IdP provides, and from the discussion above, it appears that they are already compliant with the standard.

Just ran into this as well.  If the vendor can't inject the cert into their response we'll have to resort to a one-off fork of this gem to get it working, which is unfortunate.  I echo @jdleesmiller here w/ the "Thanks, but we're a little stuck."

We're running in to the exact issue as @TJM as well. It should not be a required validation.

@Bauerpauer , @jdleesmiller , @TJM , @TimMoore
Can you review this new branch that includes support on SAMLResponse that does not contain ds:x509certificate?
https://github.com/onelogin/ruby-saml/tree/no_x509certificate

Thanks @pitbulk , this worked for me.

@luisvm can you validate it before merge?

@pitbulk I am sorry, the project that I was working with on when we came across that requirement is no longer an active customer. Hopefully @jsom or someone can still reproduce it?

Running into this problem. Any update on when this will be shipped?

Is merged.


GH Workflow for Asana PR Sync

Description
[Describe what this change achieves]
Issues Resolved
[List any issues this PR will resolve]
Check List

[ ] New functionality includes testing.

[ ] All tests pass, including unit test, integration test and doctest

[ ] New functionality has been documented.

[ ] New functionality has javadoc added
[ ] New functionality has user manual doc added

[ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@pjfitzgibbons Thanks for the PR. I think we need to add asana-token: ${{ secrets.ASANA_ACCESS_TOKEN }} to the repo.


Support for 1.51 Symphony API features

Add support to new field 'subType' on Streams API

The latest 1.51 spec has forced an upgrade to swagger-gen 2.3.0 because of the integer values in arrays (known bug in prior swagger versions).  This change will impact all Boolean getters on the symphony-java-api build from "get" to "is".  This will be hidden from anyone using the Sym objects, but if you are utilizing native objects you will be impacted (minor). SNAPSHOT of 1.51.0 native language bindings have been issued to Sonatype.
There are many other little changes in 1.51 spec, so starting to role them all in as part of SJC 1.1.5.

Actually, I'm not seeing anything in the spec related to 'subType" can you please verify :
https://github.com/symphonyoss/symphony-api-spec/blob/rc-current-r51/pod/pod-api-public-deprecated.yaml

I've verified it and found out that 'subType' field will be added on 1.52 instead of 1.51.

@ftbb
The API specs for r52 were released. https://github.com/symphonyoss/symphony-api-spec/tree/rc-current-r52/pod
Can you add support to subType field (V3RoomAttributes) and the new '/v1/files/allowedTypes' API?


Numerical Integrator: Interface with SpaceObject Class

Inconsistent format of the state variable between propagator.py and SpaceObject.py Pick one and stick to it.

Make it so that the numerical integrator only writes to the ephemeris once every "output frequency". Currently it outputs all the state vectors.

Get rid of self.atmospheric density -> this is now fetched in the force model of the ingtegrator routine.
Get rid of self.sgp4_epoch
Get rid of anything related to sgp4
Do some code profiling.

Done #54


unknown argument: '-mno-fused-madd'

Getting the following error which is the same as the one in https://github.com/emre/storm/issues/73
clang: error: unknown argument: '-mno-fused-madd'

error: command 'cc' failed with exit status 1

However the workaround in https://github.com/emre/storm/issues/73#issuecomment-42154651 did not work.
What I ended up doing was:
sh-3.2# export CFLAGS=-Qunused-arguments
sh-3.2# export CPPFLAGS=-Qunused-arguments
sh-3.2# ARCHFLAGS=-Wno-error=unused-command-line-argument-hard-error-in-future pip install pycrypto

Just thought I'd put it here for posterity. We should probably put this in the documentation.

thanks.
can you add a section to documentation for troubleshoot on installation? :santa:
we should probably put #97 too.

@emre ok will do that.


Refactor button component

Move all of the variables into the class and use better view helpers to create the tags.

@andysellick not necessarily, if the component is tested sufficiently in the view tests. View tests are preferable over unit tests in this case anyway, since it allows refactoring without changing tests.


Missing py.typed

Hi! I'm using the PEX class in my package that is linted with Mypy (for it's PEX.resolve() method). It seems pex is typed, but is missing the py.typed file and Mypy complains about missing type hints for the package.

Pex is only supported as a binary, not a library. Can you explain your use case?

Ah! I've been using PEX.resolve() to find all the console_scripts entry points and export them as shell wrappers after building the PEX, to allow the PEX to subprocess.run() console_scripts from other requirements.
I'm using the CLI to build the PEX, but have not found a way to enable this use case with the PEX cli.
        console_scripts = _get_console_scripts(PEX(str(self.output_file.get())))
        if console_scripts:
            console_scripts_dir.mkdir(parents=True, exist_ok=True)
            self.logger.info("Exporting %d console_scripts in PEX to %s", len(console_scripts), console_scripts_dir)
            for script in console_scripts:
                file = console_scripts_dir / script
                file.write_text(
                    f'#!/bin/sh\nPEX_SCRIPT={script} PATH="{console_scripts_dir.absolute()}:$PATH" '
                    f'"{self.output_file.get().absolute()}" "$@"\n'
                    "exit $?\n"
                )
                file.chmod(0o777)
        else:
            self.logger.info("note: PEX has no console_scripts")

# ...

def _get_console_scripts(pex: PEX) -> set[str]:
    """Return all entry points registered under `console_script` for this PEX."""

    result = set()
    for dist in pex.resolve():
        result.update(dist.get_entry_map().get("console_scripts", {}).keys())
    return result

There is the --venv option and PEX_VENV=true PEX_VENV_BIN_PATH=prepend variables, but I could not find a way to "embed" those variables in the PEX (--inject-var doesn't seem to work with these PEX_ variables).

You found the right answer, build the PEX with --venv prepend. Or maybe you missed that --venv optionally takes {prepend,append} args?

Oh, I did miss that! I'll try this right away, thank you. Are there any performance implications for enabling the --venv mode?

It works, unfortunately it has one drawback in my particular case in that the python executable also resolve to that of the environment that the PEX is unpacked into. So when I run Mypy as a PEX, and pass --python-executable python, it doesn't find any type hints for the project I actually want to type-check.
I've tried passing shutil.which("python") to --python-executable to get the full path to the Python interpreter that I actually want it to use. On my system that resolves to a Pyenv shim though, which later on when Mypy accesses it still resolves to the Python of the PEX's unpacked venv. 🤦

@NiklasRosenstein if you can provide a working example using Pip / venv, I'm sure I can translate that to a working Pex example for you.

@NiklasRosenstein I'll close this as an answered question unless you can provide an example set up repro steps that lead to a situation the CLI can't handle but your existing API use can.

Alright @NiklasRosenstein you've been silent; so I'll close this as answered, but please speak up if you do find a situation the CLI can't handle that you expect it could / should / the (unsupported) APIs do.


Support for other formats like Smile or CBOR

As a tormented developer trying to improve logging in my company I realised that ElasticSearch supports Smile and CBOR as input format.
I wonder if it is in your plans to allow output in these formats
So there could be an improvement in disk space, network usage and cpu usage by using binary formats instead of pure JSON.
If it's not and you guys think it could be a good idea, and you want to lead me to the right classes, I might give it a try.

Fortunately, jackson makes it easy to support additional formats by providing implementations of JsonFactory that write in those other formats.  For example SmileFactory and CBORFactory are subclasses of JsonFactory
Unfortunately, logstash-logback-encoder currently depends on MappingJsonFactory instead of just the base JsonFactory, and MappingJsonFactory is JSON specific.
The easiest way to support other formats would be to have a JsonFactoryDecorator that returned an implementation of JsonFactory that writes in the desired format.  However, since JsonFactoryDecorator currently operates on a MappingJsonFactory, this cannot be done without making a backwards incompatible change to JsonFactoryDecorator.
I just pushed a branch named dataformats that makes this backwards incompatible change, and I was able to write in Smile and CBOR using a JsonFactoryDecorator that returned the respective factories.  For example:
import com.fasterxml.jackson.core.JsonFactory;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.dataformat.smile.SmileFactory;

public class SmileJsonFactoryDecorator implements JsonFactoryDecorator {

    @Override
    public JsonFactory decorate(JsonFactory factory) {
        SmileFactory smileFactory = new SmileFactory();
        ObjectMapper mapper = new ObjectMapper(smileFactory);
        smileFactory.setCodec(mapper);
        return smileFactory;
    }
}

Since I just released 5.0 with several backwards incompatible changes, I'm hesitant to release another backwards incompatible version so quickly.
I'll need to see if there is an alternative option that is backwards compatible.

Awesome, thanks, let us know if there is anything we can do to help.

I was able to make this somewhat backwards compatible in the dataformats branch by providing default methods on the JsonFactoryDecorator interface.  I had to bump the minimum java version to 1.8 though.  So still will need a major version bump.

Available in 6.0


[auto-bump] [no-release-notes] dependency by tbantle22

:coffee: An Automated Dependency Version Bump PR :crown:
Initial Changes
The changes contained in this PR were produced by `go get`ing the dependency.
```bash
go get github.com/dolthub/[dependency]/go@[commit]
```

Fixed in https://github.com/dolthub/doltgresql/pull/675


Drop minimum python version to 3.7

It's already set to 3.7 in tox.ini, so this just brings the rest of the repo in line with that.  We can drop 3.7 support when typer drops it.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>0
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 27.059%

Totals

Change from base Build<PHONE_NUMBER>7:
0.0%

Covered Lines:
23

Relevant Lines:
85

💛  - Coveralls


Columns Component

I noticed that the columns component on Sunstar Engineering has margin bottom top properties but not on Sunstar.com.

For example on the following page: https://main--sunstar--hlxsites.hlx.live/brands the space is too narrow between columns and the section itself.
Can we set it as it was for Sunstar Engineering please. Thanks.

This was removed for the history page via https://github.com/hlxsites/sunstar/pull/121 by @dnbute
I think there this big bottom margin was looking a bit big. @ehrro , would it be ok to have that bi margin there? If yes, we can reinstate that margin. Else please suggest a common margin bottom that can be used globally.

We can use the same logic as we used for sunstar-engineering.com. If we create 1 column table with several rows then apply small margin if there is 1 table for row then apply the big margin. What do you think?

We can use the same logic as we used for sunstar-engineering.com. If we create 1 column table with several rows then apply small margin if there is 1 table for row then apply the big margin. What do you think?

Yeah, I'd rather have a single column block with multiple rows instead of multiple column blocks in the Word doc.
@JiangLong2019 , looks like you worked on this page. Do you think it is possible to combine multiple columns blocks here into one here. That'd take care of styling and also be much easier for authors.

Hi @ehrro, I changed the styling, can you please help to check if it's ok?
https://issue-166--sunstar--hlxsites.hlx.page/brands
@sdmcraft @dnbute Let me know if this change affect your pages.

It kinda messes with this
https://issue-166--sunstar--hlxsites.hlx.page/about/history

But I think a no-bottom-margin variant would do the trick, I have an open PR where I can add it

@dnbute Ops, thanks for sharing that. I agree, it would be great  if a new variant can resolve the problem.

How about you add the text below the columns block to the block itself as another row in it spanning across 2 columns?

So this is what I managed to get by adding a :last-of-type to the css that adds margin-bottom: 7.5rem, and it looks good, apart from the pretty big margin at the bottom; so basically I guess I just moved the problematic margin a bit lower unfortunately :(

would it help if you put the grey shade in the block instead of the section?

I think this is the only instance where a columns block is used like this, regardless of how I author this page we'll need @JiangLong2019's changes; I'll wait for the PR to take a closer look at what changed exactly and then I'll figure something out since I think it'll be easier that way, sounds good?

The spacing is now good. I have one more request is to horizontally align the image and the content on the right.

In addition to this I also opened another ticket to address the issue on mobile when putting margin on a container with 100% width.

You can fix that be either deleting the width here:

Or using padding instead of margin:

Thank you.

@ehrro Thanks for the feedback. I've addressed all the above, please help to check.
https://issue-166--sunstar--hlxsites.hlx.page/brands
@dnbute Sure, let's discuss. All changes are on branch "issue-166". I think it's easy to merge because only few lines had been changed, but we need to make sure it won't affect other pages.

Hello @JiangLong2019 it's good for me.

Thanks @ehrro


Markdown - differentiate the styles of h1~h6

Currently they are the same.

h1 ~ h6 in markdown should have something indicative to differentiate, since stacking like h3 h4 h5 is common.
Could refer to what Github has which is to have a bottom border for h1 ~ h3 but not h4 ~ h6.
Should be different from the normal h1 ~ h6. For example, h1 ~h6 has colors but not the normal one.
Do not color the text, use the bottom border line instead

Duplicate of #138