CCXT Exchanges: Binance, KuCoin, Cryptopia

Dear Catalyst Team,
Environment

Operating System: Linux Conda environment with python 2.7, pip install enigma-catalyst

Description of Issue:

Current Volume is displaying the volume in the market currency instead of the base currency. For example, eth_btc reports current_volume= data.current(coin, 'volume') in eth volume instead of BTC volume. I believe same error occurs in data.history. The need for base currency value is in order to properly scale and compare values.

What did you expect to happen? current daily/24h volume in BTC
What happened instead? got current daily/24h volume in ETH (market currency)

CCXT Kucoin: Many request error when getting the history of the pairs.
Error: ccxt.base.errors.RequestTimeout: kucoin GET https://api.kucoin.com/v1/open/chart/history?from=1516062779&symbol=CAT-BTC&type=30&resolution=30&to=1516667579 request timeout

CCXT Cryptopia: Cannot get all BTC pairs. Mainly because some pair symbols are all numbers and it it converting the symbol into an int instead of a string.
Error: AttributeError: 'int' object has no attribute 'lower'
catalyst/exchange/ccxt/ccxt_exchange.py", line 534, Line 534 creates the error.

What did you expect to happen?
What happened instead?

Reproduction Steps

Run code using: python <filename.py> -x <exchange_name>

from datetime import timedelta

import numpy as np
import pandas as pd
import argparse

from catalyst import run_algorithm
from catalyst.exchange.utils.exchange_utils import get_exchange_symbols
from catalyst.api import (symbols, )

def initialize(context):
    context.i = -1  # minute counter
    context.exchange = context.exchanges.values()[0].name.lower()
    context.base_currency = context.exchanges.values()[0].base_currency.lower()
    # print(context.coins)

def handle_data(context, data):
    context.i += 1
    lookback_days = 7  # 7 days

    # current date & time in each iteration formatted into a string
    now = pd.to_datetime(data.current_dt, utc=True)
    date, time = now.strftime('%Y-%m-%d %H:%M:%S').split(' ')
    lookback_date = now - timedelta(days=lookback_days)

    one_day_in_minutes = 1440  # 60 * 24 assumes data_frequency='minute'
    # update universe everyday at midnight
    if not context.i % one_day_in_minutes:
        context.universe = universe(context, lookback_date, now)

    # get data every 30 minutes
    minutes = 30
    # get lookback_days of history data: that is 'lookback' number of bins
    lookback = one_day_in_minutes / minutes * lookback_days
    if not context.i % minutes and context.universe:
        # we iterate for every pair in the current universe
        for coin in context.coins:
            pair = str(coin.symbol)

            # Get 30 minute interval OHLCV data. This is the standard data
            # required for candlestick or indicators/signals. Return Pandas
            # DataFrames. 30T means 30-minute re-sampling of one minute data.
            # Adjust it to your desired time interval as needed.
            current_volume = data.current(coin, 'volume')

            opened = fill(data.history(coin, 'open',
                                  bar_count=lookback, frequency='30T')).values
            high = fill(data.history(coin, 'high',
                                  bar_count=lookback, frequency='30T')).values
            low = fill(data.history(coin, 'low',
                                  bar_count=lookback, frequency='30T')).values
            close = fill(data.history(coin, 'price',
                                  bar_count=lookback, frequency='30T')).values
            volume = fill(data.history(coin, 'volume',
                                  bar_count=lookback, frequency='30T')).values

            # close[-1] is the last value in the set, which is the equivalent
            # to current price (as in the most recent value)
            # displays the minute price for each pair every 30 minutes
            print('{now}: {pair} -\tO:{o},\tH:{h},\tL:{c},\tC:{c},\tV:{v},\tCV:{v2}'.format(
                    now=now,
                    pair=pair,
                    o=opened[-1],
                    h=high[-1],
                    l=low[-1],
                    c=close[-1],
                    v=volume[-1],
                    v2=current_volume,
                 ))

            # -------------------------------------------------------------
            # --------------- Insert Your Strategy Here -------------------
            # -------------------------------------------------------------

def analyze(context=None, results=None):
    pass

# Get the universe for a given exchange and a given base_currency market
# Example: Poloniex BTC Market
def universe(context, lookback_date, current_date):
    # get all the pairs for the given exchange and given market
    context.coins = context.exchanges[context.exchange].assets
    context.coins = [c for c in context.coins if c.quote_currency == context.base_currency]

    return context.coins

# Replace all NA, NAN or infinite values with its nearest value
def fill(series):
    if isinstance(series, pd.Series):
        return series.replace([np.inf, -np.inf], np.nan).ffill().bfill()
    elif isinstance(series, np.ndarray):
        return pd.Series(series).replace(
                     [np.inf, -np.inf], np.nan
                    ).ffill().bfill().values
    else:
        return series

if __name__ == '__main__':
    parser = argparse.ArgumentParser()

    # Directory Parameters:
    parser.add_argument('--exchange', '-x', type=str, default='poloniex',
                        help='Specify exchange')

    start_date = pd.to_datetime('2018-01-01', utc=True)
    end_date = pd.to_datetime('2018-01-22', utc=True)

    args = parser.parse_args()
    performance = run_algorithm(# start=start_date, end=end_date,
                                capital_base=1.0,  # amount of base_currency
                                initialize=initialize,
                                handle_data=handle_data,
                                analyze=analyze,
                                exchange_name=args.exchange,
                                data_frequency='minute',
                                base_currency='btc',
                                live_graph=False,
                                live=True,
                                simulate_orders=True,
                                algo_namespace='simple_universe')

Anything else?
Trying to get all of these 3 exchanges to run paper trading. The above script is similar to simple_universe.py and will serve to test other exchange data are retrieved properly and compatible with Catalyst. Currently, only Binance (out of binance, kucoin, cryptopia) are working out of the box, in terms of providing me with history data.
Sincerely,
avn3r

Any Updates on this bug? Still unable to get Cryptopia to paper trade.

Expect something this week for sure
On Fri, Feb 23, 2018 at 4:50 PM Abner Ayala-Acevedo <
<EMAIL_ADDRESS>wrote:

Any Updates on this bug? Still unable to get Cryptopia to paper trade.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/enigmampc/catalyst/issues/182#issuecomment-368180814,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABZ-QqO9CXHXr79zCh-eTWZnIPp5mzoaks5tX1zFgaJpZM4Ro4nf
.

Actively testing this at the moment. This took longer than expected due to complexity of the data. Using CCXT as a common interface help, but we had quite a few edge cases to deal with.

Since you you use ccxt (which is not documented anywhere) should it be possible to ingest data from a ccxt exchange besides beside polo, bitfinex and bittrex? Trying for example:
catalyst ingest-exchange -x binance -i btc_usdt
does not work. Thank you

Hi @gitlines, catalyst is indeed using CCXT which is also mentioned in our documentation. We are still actively working to add more historical data (like binance) to our servers, to enable users to ingest it and backtest their strategies with. Unfortunately it takes much longer than anticipated but hopefully we will be able to release it soon.


Update raven and some other general improvements

In this PR

Updated raven to the latest version
Changed from underscore to lodash
Store all default options in this.defaults object
Merge user options with defaults using lodash _.defaultsDeep() for compatibility with future raven features

Breaking Change
All options passed to winston-sentry now mirror those used by raven. This means the previous globalTags option is now named tags.

@guzru @shaharke Can somebody take a look at this PR and merge it?

@adambiggs I like the refactoring. Notice that this change is not backwards compatible since globalTags is ignored. I suggest we make it backwards compatible before merging.

@shaharke the reason for the breaking change is that options are now passed directly to the raven client. So the winston-sentry options API should match raven's API.

@adambiggs I understand, but that would break users' code. The need to move globalTags into tags.

@shaharke ok, how about if i add something like options.tags = options.tags || options.globalTags for backward compatibility.

@adambiggs published as 0.1.2

Thanks! 🍻


How to generate text label files ? (instead of xml)

Hello,
I saved the labeled images by pressing Save
then in the image folder I get an XML file

But that's it!  no text file ?
So, how can I get a text file instead of an xml file ?
Thanks a lot

Ok I found out, just need to press the button below the Save button (on the left) in order to toggle the output mode


updated readme with contribution guidelines [Issue #79]

fixes: #79
WHAT kind of change does this PR introduce?
Update to the readme to include contributing guidelines
HOW is this accomplished?
updating the readme
Checklist

[x] Docs have been added / updated to reflect the changes
[x] I have reviewed and tested the changes :white_check_mark:
[x] I have squashed my commits to have a reasonable amount of commits
[x] All of the code I have written adhere to the coding conventions outlined in the coding conventions section of the wiki

✅| Styling is adequate and follows coding conventions outlined by the Kotlin Foundation
N/A
✅| New methods and functionalities have appropriate tests (testing edge cases + good coverage)
N/A
✅| Branch has an acceptable amount of commits with descriptive names and
N/A
✅| PR description matches the code outlined a has mentions to issues
N/A
✅| This PR attempts to implement a specific feature (no code blob or epic pr)
N/A
Status: Approved
Comments:


Selected day is reset when user performs search

Select a date from the date picker which is not today
Perform a search
Notice that the selected date gets reset to today

Completed in 06c89a2ccee2da4fc3ff3d004191ddc60dbd7f9b


Fixed flag taking same space

Fixed flag component taking same space regardless of its size prop. This was due to "scale()" default css behavior.

cool, that's helpful!


Add hide_facets_by_default to all content finder

Trello card: https://trello.com/c/Fy8KPoWV/501-all-content-show-hide-facets

@bilbof currently on 2nd line, are you happy to do the merge from rummager to resolve this change? I'll close this PR.


Loading JSON schemas from a file for validation

Hi,
I was checking the API to see if there is an option to load a defined JSON schema from a file, and then using this to validate JSON input (say in a string), but have not seen such an option.
Is this currently supported?

Yes, it is supported.  In fact, this is how the JSON Schema Test Suite is validated.
To get an instance of the schema, you can use one of two methods

JsonSchemaRegistry.Get() - For this, you need to pass a Uri instance, which means that you'll need to format the file path with the file:/// protocol.var schema = JsonSchemaRegistry.Get("file:///c:\schema.json");

Manual parse & load - You can also manually read the file text, parse the JSON and then deserialize a schema from that.  (This is done internally in method 1.)var schemaText = File.ReadAllText("c:\schema.json");
var schemaJson = JsonValue.Parse(schemaText);
var schema = new JsonSerializer().Deserialize<IJsonSchema>(schemaJson);

Regardless of which method you choose, be sure you're getting the right schema draft.  Manatee.Json will attempt to figure out which version is correct based on the the content of the schema, but you can be sure by setting the default draft.
Once you have the schema, you need to pass it a JsonValue of the JSON you want to validate.  To get this, you can use JsonValue.Parse().
var json = "<json text>";
var results = schema.Validate(json);

For more info, you can check the tests in this repository and in the docs.  I'll try to expand the docs to show more extensive usage of schema.
For what it's worth, I typically use method 2 because I don't like formatting file URIs.

Thank you for the prompt response.
Method 2 fits better to what I'm trying with a couple of other libraries, so I will probably use that.
With regards to schema draft, I think draft 6 will be better for my purpose. My current schema uses the "propertyNames": see "Additions and backwards-compatible changes" here: http://json-schema.org/draft-06/json-schema-release-notes.html
Does the library support this?

Yes, drafts 4, 6, and 7 are fully supported.
Please feel free to read the documentation for more information.


Remove gulp-load-plugins usage.

Codecov Report

Merging #2550 (27dfcd4) into main (44013c3) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #2550   +/-   ##
=======================================
  Coverage   81.18%   81.18%
=======================================
  Files         637      637
  Lines       23199    23199
=======================================
  Hits        18835    18835
  Misses       4364     4364

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 44013c3...27dfcd4. Read the comment docs.


White screen on accessing pages other than homepage

Steps to recreate the bug

Go to localhost:5173/me

White screen appears

The pages that have the white screen of death:

my profile page, localhost:5173/me
a. blank because when the page component tries to get user from local storage and userr is not logged in so its returning undefined.
b. useEffect with [] is calling populateStateWithUserById() which then calls getUserById
c. intended behaviour should be userId can be undefined, send it to the backend, and backend throw 500
browse pages, /browse-user and /browse-admin
a. both are loading but when trying to get questions, auth.js middleware is getting called to authenticate the user.
b. since user is not logged in, auth.js is throwing a 401
update user page, seems to have the same behaviour as /me page
/update-question got selectedQuestion undefined, should be same behaviour as me page
/session too

the plan to solve this.
create a generic error page that will be shown for different error types. depending on the error code, can show different content. like maybe 401 will say oh u havent log in and show a login button that redirects users, then 500 is just something went wrong with a go to home page button etc


Fixes/type errors

Since you completely ignored my last PR which contained one of these fixes, I assume that you will also ignore this one. But alas I submit this PR anyway. That one even had full tests and everything! In this one I didn't bother updating the tests with the latest fixes because I don't trust you to even look at this so why bother? Why bother having code of conduct and PR templates if no one even reviews the PRs? 🙄
Anyway, your types are wrong, please update them. Your own backend does not always return the types that you use here. You can pretend like it does, but it does not.
Shopify:  🙈🙉🙊
Me: 🤯

Hi there 👋
Sorry for the sluggish responses on this. Tldr; we do want to get this merged. And we do appreciate you taking the time to find all these type errors.
We have a ton of upstream friction with the REST resources, and we would ideally make these changes upstream, like we outline in our contribution guidelines.
Though we are planning on merging this when the January API release goes out, and apply these changes.

@lizkenyon I already did sign the CLA. Just rebased and made the same changes to 2024-01

Thanks for your contribution!


Lot of warning when i update

💡 Summary
For some time now I have a lot of "Warning" that are present when I update Misskey, is this normal?
☹ Actual Behavior
yarn install v1.22.5
[1/4] Resolving packages...
warning Resolution field<EMAIL_ADDRESS>is incompatible with requested version<EMAIL_ADDRESS>[2/4] Fetching packages...
info<EMAIL_ADDRESS>The platform "linux" is incompatible with this module.
info<EMAIL_ADDRESS>is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
warning " ><EMAIL_ADDRESS>has unmet peer dependency "<EMAIL_ADDRESS>warning "@babel/plugin-transform-runtime ><EMAIL_ADDRESS>has unmet peer dependency "<EMAIL_ADDRESS>warning "@babel/plugin-transform-runtime ><EMAIL_ADDRESS>has unmet peer dependency "<EMAIL_ADDRESS>warning "@babel/plugin-transform-runtime ><EMAIL_ADDRESS>has unmet peer dependency "<EMAIL_ADDRESS>warning "@babel/plugin-transform-runtime > babel-plugin-polyfill-corejs2 ><EMAIL_ADDRESS>has unmet peer dependency "<EMAIL_ADDRESS>warning "@babel/plugin-transform-runtime > babel-plugin-polyfill-corejs2 > @babel/helper-define-polyfill-provider ><EMAIL_ADDRESS>has unmet peer dependency "<EMAIL_ADDRESS>warning " ><EMAIL_ADDRESS>has incorrect peer dependency<EMAIL_ADDRESS>warning " ><EMAIL_ADDRESS>has incorrect peer dependency<EMAIL_ADDRESS>|| ^3.0".

And
yarn run v1.22.5
$ webpack && gulp build
assets by chunk 3.94 MiB (id hint: vendors)
  assets by status 3.4 MiB [big]
    asset 8<IP_ADDRESS>.js 1.94 MiB [emitted] [minimized] [big] (id hint: vendors)
    asset 2<IP_ADDRESS>.js 600 KiB [emitted] [minimized] [big] (id hint: vendors)
    asset 59<IP_ADDRESS>.js 462 KiB [emitted] [minimized] [big] (id hint: vendors) 1 related asset
    asset 45<IP_ADDRESS>.js 429 KiB [emitted] [minimized] [big] (id hint: vendors) 1 related asset
  10 assets
256 assets
orphan modules 3.42 MiB [orphan] 1814 modules
runtime modules 9.96 KiB 16 modules
built modules 12 MiB [built]
  modules by path ./src/ 3.72 MiB
    modules by path ./src/client/ 3.51 MiB 733 modules
    modules by path ./src/misc/ 2.98 KiB 6 modules
    modules by path ./src/prelude/*.ts 3.89 KiB 3 modules
    3 modules
  modules by path ./node_modules/ 8.29 MiB
    javascript modules 8.26 MiB 328 modules
    json modules 29.4 KiB
<EMAIL_ADDRESS>941 bytes [built] [code generated]
      ./node_modules/markdown-it/node_modules/entities/lib/maps/entities.json 28.4 KiB [built] [code generated]
  ../../src/methods/node.js (ignored) 15 bytes [built] [code generated]
  crypto (ignored) 15 bytes [optional] [built] [code generated]

WARNING in ./node_modules/matter-js/build/matter.js 30:88-110
Module not found: Error: Can't resolve 'poly-decomp' in '/home/misskey/misskey/node_modules/matter-js/build'
 @ ./src/client/scripts/physics.ts 1:0-36 12:19-39 19:19-39 30:4-21 32:19-39 33:4-21 35:19-42 42:4-20 52:18-38 58:18-41 66:4-20 68:18-37 69:28-57 78:4-20 107:16-35 113:12-30
 @ ./node_modules/ts-loader/index.js??clonedRuleSet-7.use[0]!./node_modules/vue-loader/dist/index.js??ruleSet[0].use[0]!./src/client/pages/about-misskey.vue?vue&type=script&lang=ts 10:0-47 105:35-42
 @ ./src/client/pages/about-misskey.vue?vue&type=script&lang=ts 1:0-203 1:0-203 1:204-396 1:204-396
 @ ./src/client/pages/about-misskey.vue 2:0-64 3:0-59 3:0-59 6:0-13 7:0-16 9:15-21
 @ ./src/client/pages/ lazy ^\.\/.*\.vue$ namespace object
 @ ./src/client/router.ts 8:18-47
 @ ./src/client/init.ts

WARNING in ./node_modules/moment/locale/en-SG.js
There are multiple modules with names that only differ in casing.
This can lead to unexpected behavior when compiling on a filesystem with other case-semantic.
Use equal casing.
This can impact web performance.
Assets:
  app.12.72.0.js (639 KiB)
  45<IP_ADDRESS>.js (429 KiB)
  59<IP_ADDRESS>.js (462 KiB)
  2<IP_ADDRESS>.js (600 KiB)
  8<IP_ADDRESS>.js (1.94 MiB)

WARNING in entrypoint size limit: The following entrypoint(s) combined asset size exceeds the recommended limit (244 KiB). This can impact web performance.
Entrypoints:
  app (639 KiB)
      app.12.72.0.js

1 warning has detailed information that is not shown.
Use 'stats.errorDetails: true' resp. '--stats-error-details' to show it.

📌 Environment
Debian 10
Node : 14.15.5
Yarn : 1.22.5
Misskey 12.72.0

Wish we can help fix this. Similar issue since Misskey 12.53.0 https://github.com/syuilo/misskey/issues/6808


[bn] Translate turnkey-solutions.md

Translated turnkey-solutions.md for Bengali Documentation
Localizes content/bn/docs/setup/production-environment/turnkey-solutions.md
issues: #31677

/retitle [bn] Translate turnkey-solutions.md

/language bn

/merge

/merge


Fix potential out-of-order execution with Drawable scheduling method

In master, performing Drawable.Schedule with no transform delay prior to a regular Scheduler.Add does not make it execute in order, as the first gets enqueued to timedTasks while the second gets enqueued to runQueue directly, causing the second to execute before the first.
At first I was going to resolve this by making Scheduler.AddDelayed(..., 0) enqueue to runQueue directly, but looking at it again, I felt that to be weird and unnecessary logic compared to updating usages itself to use Add if there's no delay.
I'm honestly not sure which to pick at this point. For reference, here's a branch of the alternative direction (https://github.com/frenzibyte/osu-framework/compare/drawable-schedule-no-delay).

[ ] Depends on #5058

Can you explain how you ran into this?
Can this have a test case attached?

Can you explain how you ran into this?

Pretty much the same as I mentioned in the first paragraph minus unnecessary details, basically:

Action A was scheduled with Drawable.Schedule (which translated to Scheduler.AddDelayed(..., 0)).
Action B was scheduled with Scheduler.Add.

Action B got executed before Action A, when none of them had any delays and Action A should have been executed first.

I'm unsure if a test case is required as it would be testing performing a Drawable.Schedule followed by Scheduler.Add with ensuring execution order to be correct, which feels a bit weird... could add one though, I guess it still makes sense to have one.

but what were you doing that triggered this? i'm not interested in a more abstract explanation with "action A" and "action B" of what the failure case is. i see what the failure case is. i was interested what specifically you were doing where this surfaced, because apparently this order matters in some respect in whatever it is that you're doing.

I'm unsure if a test case is required as it would be testing performing a Drawable.Schedule followed by Scheduler.Add with ensuring execution order to be correct, which feels a bit weird...

that's literally what "adding a test case" means? you're arguing that you're not sure if you should add a test case because you'd have to test the behaviour you introduced here? i don't get it.
if this ordering is something we really want to uphold then this is an assumption that should be tested.

I didn't think such details will be necessary as I had thought scheduled delegates should always be executed in order...
...but, on DrawableChannel, chat lines are populated on BDL by a method named newMessagesArrived, and that method schedules itself using Drawable.Schedule.
I have wrote a method that queries all chat lines to find which chat line to highlight, and I made it so that it gets scheduled if DrawableChannel is not loaded yet, so that the chat lines are populated correctly via the initial newMessagesArrived call.
I have used Scheduler.Add<T>(action, data) to schedule the highlighting method, but then I have realized the highlighting method is actually being executed before the scheduled intial newMessagesArrived logic, which is contrary to what I had expected (since newMessagesArrived ran first).
After debugging through, I've found out Drawable.Schedule does not execute before a regular Scheduler.Add<T>(action, data) due to using AddDelayed regardless of whether TransformDelay = 0, and that's why I ended up PR'ing this.

i was asking because while i agree that this ordering is probably more correct, relying on it is probably not the wisest. and i'd say that in your use case relying on scheduling order seems kind of sketchy. i'd ideally like a solution that does not rely on something like that as it's potentially really easy to break - and this change could end up being potentially breaking (although i'd hope we don't rely on the old ordering anywhere either).

I'm not 100% sure on that since the initial newMessagesArrived call happens on BDL, which should be invoked before any side methods like highlighting a message get called.

I have added test coverage as requested.

I'm fine with this, until we come across a case that requires things to behave differently (hopefully never).


[BUG] Issue with Chapter Download

Currently there seems to be a minor problem when downloading chapters. From my experience, you cannot download a second time from the same manga title without closing and reopening the chapter list first.
Check for yourself... just open a random manga with several chapters and download one of them, doesn't matter which. You can also select multiple chapters. This download will work just fine. Now, once the queue is finished, select another chapter and try to download that one. Everything looks normal and Mangadesk seems to work, it will also tell you that the second download is okay after some time, but when you look in the manga folder there won't be a new subfolder for the second download. This behavior persists until you close the chapter list in Mangadesk and reopen it,

hmmm okay i tried doing what you said but it seems to download fine for me?
Steps I did:

Open a chapter page.
Select some chapters to download.
Download finishes, add more selections.
Try downloading again.
Check files, sees new chapters.

Perhaps there's a step that I'm doing wrong to replicate this issue?

current version (0.7.5) Win 8.1 x64, guest mode
Steps are like you listed them.

Search for Manga, pick title from list
open title/chapter list
go to a chapter (lets call it chapter A), press enter and confirm to download.
wait for finish message, don't leave chapter list. Chapter is marked as downloaded
select another chapter (chapter B), press enter again and start download - still looks like it's working
wait for the finish message, then check results:
-> DDL status marker on chapter B is not "Y"
-> in manga folder chapter A subfolder (and content) is present but chapter B subfolder is missing

I don't know what causes this. If it doesn't happen for you it be an OS issue. Thing is, I'm still using the same system and I do know it did not happen in some prior versions. Anyhow, I'm not sure when this happened first. It's not a new one. Might have popped up in 0.6.0 or 0.6.2

found the issue, thank you!

Ah, I understand. Setting a mark (CTRL+E) on the chapter first circumvents this...
I normally won't be marking a single chapter :smile_cat:


[SHOWCASE]

Title
My NAS Dashboard
Screenshot

Would you like your name/ username included?
Yes
Link to your Website/ Profile/ Twitter (optional)
https://aemoncao.github.io/
Description (Optional)
No response

Looking good!


Feature/milestone2

What I Did:

I added props that displayed labels on the calculator buttons.
I added props for default result in the display of the calculator.

Please watch this loom video.
https://www.loom.com/share/f2c82f6b601f4f74b066774274ad1910


Crash: vscode crashes after attempting to delete firebase/functions file from vscode

VSCode Version: 1.52.1 (user setup)
OS Version: Windows_NT x64 10.0.19042

Steps to Reproduce:

run firebase init functions
right-click on the functions directory and click delete
vscode crashes and is not responsive to scrolling and keyboard input
but the mouse still works and can open and close files
the mouse is a text cursor instead of a mouse-cursor
had to end task of vscode in task manager and re-opened vscode
vscode is now functioning well
had to delete the folder in the file explorer instead of vscode

Not sure if this issue occurs when all extensions are disabled as I have already deleted the file but it should still crash as it is not a problem with the extensions.
The functions file is quite big so maybe that's the reason why vscode crashed.

I think this is likely https://github.com/microsoft/vscode/issues/71243

The issue that has been referenced states that VSCode crashed attempting to delete a 10GB file while the file I tried deleting was a mere 20-30MB.
Is there a specific reason why this happens?
Thanks in advance.

@jeffzy15 how exactly do you delete the file, from the explorer?

In the explorer I right click on the directory and click delete which moves it to the recycle bin. But when I right click to delete the folder in VSCode to do the same, it crashes while the file explorer handled it well without crashing.

@jeffzy15 explain what "crashes" means, can you record a screen recoridng?

The initial crash was longer and didn't eventually delete the folder.

Ok thanks, that video is useful. I refer to @deepak1556 for how to diagnose this further as it seems to be an issue from the Electron API we use to move an item to the trash.

Clarification:
Folder size is 46MB.
Folder size on disk is 60MB.

Why not just send delete command to the background task and free up vscode instance? Why it so done in the first place?
This issue is so old

Why not just send delete command to the background task and free up vscode instance? Why it so done in the first place?
This issue is so old

@FDiskas could you show how to send delete command to the background task and free up VSCode instance.

@FDiskas could you show how to send delete command to the background task and free up VSCode instance.

Are you shure that I'm the right person to ask?
I would use google or stackoverflow.
There is something like "node detach child process"

Are you shure that I'm the right person to ask?
I would use google or stackoverflow.
There is something like "node detach child process"

Can you attach a crash dump for that scenario above? https://github.com/microsoft/vscode/wiki/Native-Crash-Issues
I'm unable to reproduce the issue locally.

Reproducible on windows

Reproducible on windows


Option to not steal focus when switching PRs

When githubPullRequests.focusedMode is set to "overview", if I checkout a pull request and then focus the terminal—either because I used the terminal to checkout the PR or because I now want to run a command such as installing dependencies—when the overview opens up it steals focus from the terminal. I would like an option to disable this focus stealing behavior, or alternatively, support an additional option for this setting that shows the PR details in the sidebar but doesn't open the full overview.

It's actually not the overview stealing focus, it's the sidebar.


Add APP_NAME support.

Consider adding support for APP_NAME to the connection string?

On vacation right now. Coming back to work on Monday, taking a look asap next week.

Released in v0.5.16


BZ1249559: Improve undo functionality in IE10 / IE11 by executing com…

…mands in refreshCanvasForIE as a group, rather than separately.
Tiho - will you take a look at this and see what you think. It improves the Undo functionality in IE10/IE11 to make it usable, but it's still not perfect. For each action that's been done in a process, the Undo command can be run twice - once to undo the selection of the selected activity, and once to undo the actual action that was done.

Jeremy and I discussed about this and decided to implement a global strategy for this that can help all commands. His implementation here is fine but we will look at the ability to omit all/any commands as needed from the undo stack app wide


[UI] Adopt fundamental styles - view Logs

Target

[ ] Adopt the fundamental styles in Logs view

Use https://github.com/dirigiblelabs/ide-next as reference

Issue cloned to https://github.com/eclipse/dirigible/issues/1504


Several simultaneous deploys

Feature to several simultaneous deploys. :smile:

This is duplicate issue from #12 .
There is a workaround but it require a little bit more work.
https://github.com/amoussard/sftp-deployment/issues/12#issuecomment-140293991

I think we should close this issue since is a duplicate.

Ok, thanks! :+1:


How to splice the other features needed into Bert and fine-tune them together?

In the process of using bert, we don't want to give up our hand-crafted features, such as the length of the sentence, the number of nouns in the sentence, the average length of the words in the sentence, and so on. I know that bert's create_model function in run_classifier has output_layer = model.get_pooled_output() and the output size is [batch_size, hidden_size]. I wander if  it could directly make my above three features to shape [batch_size, 3], Thus concat into [batch_size, hidden_size+3], and then fine-tuned, the problem is that the create_model function can not control the step size of batch_size.
I also think that using the extract_features method mentioned in https://github.com/google-research/bert/issues/201 is not so reasonable. This method seems to use the bert to extract the feature vector separately, then concat to three features together, then Training and validing alone, but if this method lost the role of bert fine-tuning?
Is there anyone realize the concat custom features? Thank you very much!!!

I probably have the same problem. "Merge" BERT text embedding to my hand-crafted features about novelty and diversity. I would like to implement something that looks like an new intermediate layer that blends the two, using the output as the input for a classic fine tuning operation in BERT. Not something like the #201 .
I'm trying to understand if this is possible with small changes.

I tried another way to solve this at #201 maybe help to you.

I probably have the same problem. "Merge" BERT text embedding to my hand-crafted features about novelty and diversity. I would like to implement something that looks like an new intermediate layer that blends the two, using the output as the input for a classic fine tuning operation in BERT. Not something like the #201 .
I'm trying to understand if this is possible with small changes.


ignore case of "Content-Type" and "Content-type"

ignore case of "Content-Type"  and  "Content-type"
Most JS libraries  use "Content-type".

Case sensitive  in JS,  but  not in HTTP protocol .
WTF !

#102 is a better one


Native iOS example doesn't seem to show the video

When testing the native demo app for ios, video doesn't seem show up. There seems no error in the log.

2015-04-23 15:42:24.580 NativeDemo[31953:4332609] [OpenWebRTC] initialized correctly!
2015-04-23 15:42:24.706 NativeDemo[31953:4332609] Video settings: {
bitrate = 1024000;
framerate = 25;
height = 480;
mode = VGA;
width = 640;
}
2015-04-23 15:42:24.919 NativeDemo[31953:4332609] Can't find keyplane that supports type 4 for keyboard iPhone-Portrait-NumberPad; using<PHONE_NUMBER>_Portrait_iPhone-Simple-Pad_Default
2015-04-23 15:44:07.379 NativeDemo[31953:4332609] Joining room with ID: 111111
[audio/capture] Default audio input
2015-04-23 15:44:07.386 NativeDemo[31953:4332651] gotLocalSourcesWithNames: (
"Default audio input"
)

The simulator does not support video sources I'm afraid. You will need to run on a real device.


[bug] Can't resolve 'events'; Can't resolve 'buffer'; Can't resolve 'lit-html'

Is there an existing issue for this?

[X] I have searched the existing issues

RainbowKit Version
2.1.3
wagmi Version
2.10.10
Current Behavior
I use below command to create a new react app:
yarn create react-app my-app
and then installed rainbowkit using below command:
yarn add @rainbow-me/rainbowkit wagmi<EMAIL_ADDRESS>@tanstack/react-query
Update src/index.js file to below:
import React from 'react';
import ReactDOM from 'react-dom/client';
import './index.css';
import App from './App';
import reportWebVitals from './reportWebVitals';
import<EMAIL_ADDRESS>import {
  getDefaultConfig,
  RainbowKitProvider,
} from '@rainbow-me/rainbowkit';
import { WagmiProvider } from 'wagmi';
import {
  mainnet,
  polygon,
  optimism,
  arbitrum,
  base,
} from 'wagmi/chains';
import {
  QueryClientProvider,
  QueryClient,
} from "@tanstack/react-query";

const config = getDefaultConfig({
  appName: 'My RainbowKit App',
  projectId: 'YOUR_PROJECT_ID',
  chains: [mainnet, polygon, optimism, arbitrum, base],
  ssr: true, // If your dApp uses server side rendering (SSR)
});

const queryClient = new QueryClient();

const root = ReactDOM.createRoot(document.getElementById('root'));
root.render(
  <React.StrictMode>
    <WagmiProvider config={config}>
      <QueryClientProvider client={queryClient}>
        <RainbowKitProvider>
          <App />
        </RainbowKitProvider>
      </QueryClientProvider>
    </WagmiProvider>
  </React.StrictMode>
);

// If you want to start measuring performance in your app, pass a function
// to log results (for example: reportWebVitals(console.log))
// or send to an analytics endpoint. Learn more: https://bit.ly/CRA-vitals
reportWebVitals();

Then try to run the project using below command:
yarn start
Below errors will appear.

Expected Behavior
No error.
Steps To Reproduce
yarn create react-app my-app
cd my-app
yarn add @rainbow-me/rainbowkit wagmi<EMAIL_ADDRESS>@tanstack/react-query
    Update src/index.js file;

yarn start
Link to Minimal Reproducible Example (CodeSandbox, StackBlitz, etc.)
https://codesandbox.io/p/github/linghuccc/rainbowkit-error/main
Anything else?
No response

It looks like you're missing some polyfills for your project. Reference the src/polyfills.ts file in this Create React App example and accompanying buffer dependency in the package.json: https://github.com/rainbow-me/rainbowkit/tree/b25db9a1f1f5acc96e28a12e38c2156e4fdbf51d/examples/with-create-react-app

It looks like you're missing some polyfills for your project. Reference the src/polyfills.ts file in this Create React App example and accompanying buffer dependency in the package.json: https://github.com/rainbow-me/rainbowkit/tree/b25db9a1f1f5acc96e28a12e38c2156e4fdbf51d/examples/with-create-react-app

Thanks, after I created polyfills.js file, installed buffer, events and lit-html modules, most of the errors disappeared, only the below error persists:

Ensure that you've followed the Wagmi installation guide and installed the Tanstack peer dependency https://wagmi.sh/react/getting-started#manual-installation

Ensure that you've followed the Wagmi installation guide and installed the Tanstack peer dependency https://wagmi.sh/react/getting-started#manual-installation

I suspect the error comes from yarn, but I am not sure how to fix it.

You might have conflicting peer dependencies installed. Clean your package.json, reference the peer versions here, and give that a try: https://github.com/rainbow-me/rainbowkit/blob/main/examples/with-next/package.json

You might have conflicting peer dependencies installed. Clean your package.json, reference the peer versions here, and give that a try: https://github.com/rainbow-me/rainbowkit/blob/main/examples/with-next/package.json

I have tried to use the package json file from the example, but was not successful. Later I changed yarn from v4.3.1 to v1.22.19 and there is no above errors.
Frankly speaking, your reply speed is very fast, but the product has a lot of bugs. Simply following the installation commands also fails. I really do not think rainbowkit is a product ready for use.

@linghuccc Can you delete node_modules and yarn.lock, then do yarn install again ?

@linghuccc Can you delete node_modules and yarn.lock, then do yarn install again ?

Done, still the same. I think the error originated from yarn peer dependency issue.

@linghuccc Yes that's right. It's a peer dependency issue, but i don't see why this would cause your app to crash. If you run your app is it working again ?

@linghuccc Yes that's right. It's a peer dependency issue, but i don't see why this would cause your app to crash. If you run your app is it working again ?

When you say "run your app", I suppose you mean "open the web page" because I am trying to build a react project. The index page will never show up and the error in below reply will always come up:
https://github.com/rainbow-me/rainbowkit/issues/2098#issuecomment-2229995195

@linghuccc Gotcha. Can you re-upload your code to codesandbox or github repo and i'll have a look ?

@linghuccc Gotcha. Can you re-upload your code to codesandbox or github repo and i'll have a look ?

Sure, the codes is in below URL:
https://github.com/linghuccc/rainbowkit-error

@linghuccc Gotcha. Can you re-upload your code to codesandbox or github repo and i'll have a look ?

Sure, the codes are in below URL: https://github.com/linghuccc/rainbowkit-error

Thanks! Will take a look and get back.

@linghuccc Apologies for the delay. I just checked out your code and doesn't seem like i'm getting any errors. I can see some warnings, but you can modify your start command like so.
"start": "GENERATE_SOURCEMAP=false react-scripts start"

The reason why this is happening is because the project uses create-react-app which isn't stable anymore and you should definetely look into using another library such as vite.

@linghuccc Apologies for the delay. I just checked out your code and doesn't seem like i'm getting any errors. I can see some warnings, but you can modify your start command like so.
"start": "GENERATE_SOURCEMAP=false react-scripts start"

The reason why this is happening is because the project uses create-react-app which isn't stable anymore and you should definetely look into using another library such as vite.

Thank you for your reply. I have updated start script as you suggested and below is the error I encountered:

I also need to point out, this error does not only appear when using "create-react-app". In fact, in my other post, I have reported similar errors when using "yarn create @rainbow-me/rainbowkit": https://github.com/rainbow-me/rainbowkit/issues/2097

@linghuccc I was not able to reproduce this for some reason. Did you do npm install and npm start ?

@linghuccc I was not able to reproduce this for some reason. Did you do npm install and npm start ?

As I described in the steps, I used "yarn" command, and did not use "npm" command.

@linghuccc Yes i did yarn install and yarn start. Everything worked fine for me.

What is your yarn version?

@linghuccc 1.22.22

@linghuccc 1.22.22

I have tried one more time with<EMAIL_ADDRESS>but the error remains.

@linghuccc Can you confirm that this is the right repo ? https://github.com/linghuccc/rainbowkit-error

@linghuccc Can you confirm that this is the right repo ? https://github.com/linghuccc/rainbowkit-error

Please refer to below repo for the latest error (yarn version 1.22.22)
https://github.com/linghuccc/rainbowkit-yarn.1.22.22
Thank you.

@linghuccc In your App.js you need to add client to <QueryClientProvider> like this.
const config = getDefaultConfig({
  appName: 'Uniswap Example',
  projectId: process.env.REACT_APP_PROJECT_ID,
  chains: [mainnet, sepolia],
  ssr: true, // If your dApp uses server side rendering (SSR)
  transports: {
    [sepolia.id]: http('https://sepolia.infura.io/v3/' + process.env.REACT_APP_INFURA_KEY),
  }
});

const queryClient = new QueryClient(); // <- here

const root = ReactDOM.createRoot(document.getElementById("root"));

root.render(
  <React.StrictMode>
    <WagmiProvider config={config}>
      <QueryClientProvider client={queryClient}>
        <RainbowKitProvider>
          <App />
        </RainbowKitProvider>
      </QueryClientProvider>
    </WagmiProvider>
  </React.StrictMode>
);

@linghuccc In your App.js you need to add client to <QueryClientProvider> like this.
const config = getDefaultConfig({
  appName: 'Uniswap Example',
  projectId: process.env.REACT_APP_PROJECT_ID,
  chains: [mainnet, sepolia],
  ssr: true, // If your dApp uses server side rendering (SSR)
  transports: {
    [sepolia.id]: http('https://sepolia.infura.io/v3/' + process.env.REACT_APP_INFURA_KEY),
  }
});

const queryClient = new QueryClient(); // <- here

const root = ReactDOM.createRoot(document.getElementById("root"));

root.render(
  <React.StrictMode>
    <WagmiProvider config={config}>
      <QueryClientProvider client={queryClient}>
        <RainbowKitProvider>
          <App />
        </RainbowKitProvider>
      </QueryClientProvider>
    </WagmiProvider>
  </React.StrictMode>
);

You are right! I must have missed it during copy and paste!
I immediately went to check projects initialized using "yarn create @rainbow-me/rainbowkit" and "yarn create vite". Luckily I did not make the same mistake for those two projects. Even luckier I found out that those two projects no longer has similar error as before. Well, all at a sudden, I am all good.
Thank you for pointing this out.

Thank you so much, magiziz!


Highlighting the downloads links in the nav

I'm using this branch for us to try different ways of highlighting the downloads links in the nav, now that the content is in the nav. This first attempt is using simple emoji. It makes them stand out, but doesn't necessarily fit with the rest of the brand style?
Thoughts and suggestions to talk this one through are welcome. I figure this is one of those where we need to see some iterations to see what we want to go with. Not a priority for launch, but starting the discussion while it's in our heads.

I like having icons there to show they're a different type of link. The XP & XC save icons work for me.
I'm not sure about the Send & OC elements though, it feels like it works for registration but not for existing users. Maybe we should have 2 links for those products, something like

Create an account on OC/Send for free
Log into OC/Send portal

I like having icons there to show they're a different type of link. The XP & XC save icons work for me.

What about the icons used for OrderCloud/Send?

I'm not sure about the Send & OC elements though, it feels like it works for registration but not for existing users. Maybe we should have 2 links for those products, something like

Create an account on OC/Send for free
Log into OC/Send portal

I'm fine with us updating the nav to have more elements, but for this particular PR we should contain to the highlighting portion. Unless you are meaning that would impact how you would want those highlighted if there were now two links showing?
NOTE: I also tried using the SVG that is in 'waffle' nav menu at the top to indicate that it goes somewhere else, but I lacked the CSS skills to get it to line up appropriately, just putting that out there as another option to show that something 'different' is there.

Yeah, I'm fine with combining the two if thats what people prefer. If we do combine them, then I think using a static arrow like @markvanaalst suggested makes more sense.

I've updated with the latest from Main, merged the OrderCloud and Send back together into a single link, and added an emoji 'arrow'. Is this the idea folks were thinking might work?
Or do we want to use a non-emoji arrow like the one in the Waffle menu?

I think I prefer using the same SVG as used in the burger menu. I just submitted a change where you can now mark a link as being external and the nav component will render the SVG with it.
What do you guys think of this approach?

I like what is here, but since it's my PR, I can't "approve" this review. I want to say that in dark mode, the icon used by Rob looks better, but the one Mark has is consistent with the waffle menu icons. Let's go with that for now.
@markvanaalst or @robearlam , whomever sees this first, can you do the final "approval" and then merge to main?


Controlled vocabularie to be used for adms:representationTechnique

Why is in this profile an INSPIRE codelist  http://inspire.ec.europa.eu/metadata-codelist/SpatialRepresentationType  required instead of the widely used ISO19139 codelist MD_SpatialRepresentationTypeCode?
This INSPIRE codelist is not mandated by INSPIRE so if it must be used in geoDCAT explanation is required or allow to use the ISO19139 codelist.

same issue for  the property:
dcat:hadRole with codelist http://inspire.ec.europa.eu/metadata-codelist/ResponsiblePartyRole,
geodcatap:resourceType with codelist http://inspire.ec.europa.eu/metadata-codelist/ResourceType
geodcatap:topicCategory   with codelist http://inspire.ec.europa.eu/metadata-codelist/TopicCategory

This is because GeoDCAT-AP is an RDF (Linked Data) representation of the metadata, where controlled vocabularies and their items are identified by their IRIs, which is not the case for ISO codelists. That is why there are RDF (Linked Data) based codelist counterparts published in the INSPIRE registry, which use IRIs for identification.
Content-wise, the http://inspire.ec.europa.eu/metadata-codelist/SpatialRepresentationType seems the same as the MD_SpatialRepresentationTypeCode, and the other corresponding codelists, should be identical, so there should be no issue in the mapping.


Fishy registration

I have worked on registration to make it more robust to the cuttlefish data as well as Jo's plates.
I had to relax the constraints on both angle and scale for particle stds, and then increase the number of particles since there's now more degrees of freedom.
I also switched out the top hat filter in spot detection for a Laplacian of Gaussian filter, and added an extra check that enough spots have been detected. If not, min repeatability will be decreased.
There are multiple fails in 2020-05-02-13-49-02-COVID_May2_JVassay_FcIgGplate_images, e.g. D12 and F1. As much as I welcome all kinds of image artifacts and smears that will help make the algorithm more robust, I think at some point for particularly bad images we'll have to start flagging fails.
Right now I issue a warning if the final particle filter fails, but we should instead at that point stop analyzing, or return NaNs or something. I'd love to hear your thoughts on this.
Oh and by the way, @bryantChhun, I'm sorry but I renamed c to constants, turns out I can't handle single letter variables outside single functions well (so hard to search for!)...

Codecov Report

Merging #60 into master will decrease coverage by 0.77%.
The diff coverage is 39.23%.

@@            Coverage Diff             @@
##           master      #60      +/-   ##
==========================================
- Coverage   36.57%   35.80%   -0.78%
==========================================
  Files          23       23
  Lines        1553     1606      +53
==========================================
+ Hits          568      575       +7
- Misses        985     1031      +46

Flag
Coverage Δ

#unittests
35.80% <39.23%> (-0.78%)
:arrow_down:

Impacted Files
Coverage Δ

array_analyzer/load/debug_plots.py
14.43% <0.00%> (ø)

array_analyzer/transform/array_generation.py
8.45% <0.00%> (ø)

array_analyzer/load/report.py
14.28% <7.14%> (-0.35%)
:arrow_down:

array_analyzer/extract/img_processing.py
12.75% <7.81%> (-0.83%)
:arrow_down:

array_analyzer/workflows/registration_workflow.py
20.22% <9.09%> (-3.39%)
:arrow_down:

array_analyzer/workflows/interpolation_wf.py
24.24% <20.00%> (ø)

array_analyzer/workflows/well_wf.py
22.41% <40.00%> (+0.74%)
:arrow_up:

array_analyzer/transform/point_registration.py
7.86% <50.00%> (ø)

pysero.py
53.84% <66.66%> (+1.21%)
:arrow_up:

array_analyzer/extract/metadata.py
86.44% <96.61%> (+0.23%)
:arrow_up:

... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 30324f1...4dbef22. Read the comment docs.

@bryantChhun @smguo @mattersoflight I've tuned the spot registration again to work better with the May 2 data.
Main changes is that

I adapted spot detection settings to the imaging params instead of pixel sizes,
tuned LoG filters,
Added well detection and cropping prior to spot detection
made spot detection a class,
will stop analysis if registration fails (ODs will not be written in these cases)

With the May 2 data it looks get_spot_intensity is a bit all over the place, but I didn't have time to investigate.
Please take a look again. Thanks!

Codecov Report

Merging #60 into master will decrease coverage by 0.77%.
The diff coverage is 39.23%.

@@            Coverage Diff             @@
##           master      #60      +/-   ##
==========================================
- Coverage   36.57%   35.80%   -0.78%
==========================================
  Files          23       23
  Lines        1553     1606      +53
==========================================
+ Hits          568      575       +7
- Misses        985     1031      +46

Flag
Coverage Δ

#unittests
35.80% <39.23%> (-0.78%)
:arrow_down:

Impacted Files
Coverage Δ

array_analyzer/load/debug_plots.py
14.43% <0.00%> (ø)

array_analyzer/transform/array_generation.py
8.45% <0.00%> (ø)

array_analyzer/load/report.py
14.28% <7.14%> (-0.35%)
:arrow_down:

array_analyzer/extract/img_processing.py
12.75% <7.81%> (-0.83%)
:arrow_down:

array_analyzer/workflows/registration_workflow.py
20.22% <9.09%> (-3.39%)
:arrow_down:

array_analyzer/workflows/interpolation_wf.py
24.24% <20.00%> (ø)

array_analyzer/workflows/well_wf.py
22.41% <40.00%> (+0.74%)
:arrow_up:

array_analyzer/transform/point_registration.py
7.86% <50.00%> (ø)

pysero.py
53.84% <66.66%> (+1.21%)
:arrow_up:

array_analyzer/extract/metadata.py
86.44% <96.61%> (+0.23%)
:arrow_up:

... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 30324f1...4dbef22. Read the comment docs.


Property:alternativeCollectionName

Label
Alternative Collection Name

Definition
A short title different to the one given in ObjectGroup.collectionName that is used to summarise the collection objects contained within the ObjectGroup.

Usage

Existing property

Existing class

Existing property identifier

Format
Text

Required
No

Repeatable
No

Constraints

Examples
Darwin's fossil mammal collection, The Hubricht Molluscan Collection

Notes

Created in response to #418


Add this cool feature to github.io

Could we have the table of content buttoin also on github.io that would be awesome

Hmm, do you mean the gh-pages? Not every page is the same, and I'm not sure if the rendered markdown classes are the same. Do you have an example page?

The following page here:
http://gabyx.github.io/GRSFramework/
this index.html directly loads the Home.md from the wiki and then renders the markdown :-)
It would be the most handiest, if I could send the inner html of my

...

to your java script function which (wenn the TOC button is pressed)
reads the innerHTML of
...
 , parses the stuff into a TOC and renders the TOC.
So would it be possible to include all the java script directly on my index.html page
and then call some function (which one? :-)) which displayes the TOC?
Would be awesome to have that. But I am not really JavaScript geek.
Maybee one could add a simple wiki entry to show which steps need to be taken to make that happen.

Honestly, because you're using gh-pages, I think you might be happier with a side menu. Something like:

http://takien.github.io/jPushMenu/
https://www.berriart.com/sidr/
http://callmenick.com/post/slide-and-push-menus-with-css3-transitions
http://getbootstrap.com/javascript/#scrollspy (a bit more complicated to set up)

I could get you started... I'll work on something when I get some free time.

Done: https://github.com/gabyx/GRSFramework/pull/1

Thanks for your input :-)
Due to that, I am no able to fully parse the rendered markdown into a sidebar TOC :-) pretty awesome, I added some +/- expand buttons on top =), thanks for your help .-)
http://gabyx.github.io/GRSFramework/

Nice work! :grinning:


Connection Support for Parameters.json

I am finding when I run my unit tests it errors on the connections.json file because its linking to a parameters.json file instead of the appsettings.
I am wanting to confirm if the code supports both? I had to do a work around in updating the connections.json to instead use appsettings and replace the authentication with the value from the parameters file. (connections.json is the original file and connections-unittest.json is the rewrite).
I have found when attempting to run the Logic Apps with the updated connections.json it doesn't work live anymore with the rewrite. To be able to test I have to rename the connections-unittest.json to connections.json and then back again once I am done testing. I am wondering if the library could be updated to support both?
connections-unittest.json
connections.json
parameters.json

Hi @ChristinaHebertHCS . Yes, you are right, the framework only supports connection runtime URLs that are inline in the connectionRuntimeUrl attribute in the connections.json file, or parameterised using @appSetting(). It does not support parameterisation using @parameters().
When I wrote this part of the framework, the Standard Logic App extension wasn't using @parameters() to parameterise the connections runtime URLs, so the framework code worked. But now the extension uses @parameters(), hence your observation.
I'll add this to the backlog for the next version.


Suggestion: Updating the Sage > Invalid PHP Version Error to correct minimum version

I know it's nitpicky, but currently the error when PHP is below the minimum requirement it simply states:

You must be using PHP 7.1 or greater.

But in the documentation states the minimum is 7.1.3
I recently told a new host the minimum was 7.1 so they installed 7.1.28... not sure if the theme should work at that version or not because 7.1.3 hasn't been installed yet, but if the theme truly is broken unless at least at 7.1.3 then I think the error should reflect that.
Sage4Lyfe!! 🙌

Hi @buckedup,
It looks like the issue template is missing from this issue. Please take a look at the Contribution Guidelines, which will tell you exactly what your ticket has to contain in order to be processable.
Please do not use the issue tracker for personal support requests (use the Roots Discourse to ask the Roots Community for help, or if you want the Roots Team to dedicate some time to your issue, we offer our services as well).

28 > 3
just FYI


build problems of gemm_matmul

@shu1chen I am running into a CMake build issue in the 'other' repo for the CPU:
Selecting Windows SDK version 10.0.22621.0 to target Windows 10.0.22631.
The C compiler identification is MSVC 19.38.33135.0
The CXX compiler identification is MSVC 19.38.33135.0
Detecting C compiler ABI info
Detecting C compiler ABI info - done
Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Professional/VC/Tools/MSVC/14.38.33130/bin/Hostx64/x64/cl.exe - skipped
Detecting C compile features
Detecting C compile features - done
Detecting CXX compiler ABI info
Detecting CXX compiler ABI info - done
Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Professional/VC/Tools/MSVC/14.38.33130/bin/Hostx64/x64/cl.exe - skipped
Detecting CXX compile features
Detecting CXX compile features - done
CMake Warning at libs/intgemm/CMakeLists.txt:58 (message):
  Your compiler is too old to support AVX2 AVX512BW AVX512VNNI.
  Multiplication will be slower on CPUs that support these instructions.  For
  details rerun cmake with --debug-trycompile then try to build in
  compile_tests/CMakeFiles/CMakeTmp.

DNNL_TARGET_ARCH: X64
DNNL_LIBRARY_NAME: dnnl
Performing Test CMAKE_HAVE_LIBC_PTHREAD
Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
Looking for pthread_create in pthreads
Looking for pthread_create in pthreads - not found
Looking for pthread_create in pthread
Looking for pthread_create in pthread - not found
Found Threads: TRUE
Found OpenMP_C: -openmp (found version "2.0")
Could NOT find OpenMP_CXX (missing: OpenMP_CXX_FLAGS OpenMP_CXX_LIB_NAMES)
Could NOT find OpenMP (missing: OpenMP_CXX_FOUND) (found version "2.0")
CMake Error at C:/Program Files/CMake/share/cmake-3.26/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find BLAS (missing: BLAS_LIBRARIES)
Call Stack (most recent call first):
  C:/Program Files/CMake/share/cmake-3.26/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
  libs/oneDNN/cmake/FindBLAS.cmake:953 (find_package_handle_standard_args)
  libs/oneDNN/cmake/blas.cmake:66 (find_package)
  libs/oneDNN/CMakeLists.txt:126 (include)

I have installed both MKL and AOCL but the BLAS is not found. Is there a variable I need to set?

@Ravenwater Thank you for being interested in this repo for GEMM performance comparison on GPU and taking the time to try it. However, the compilation and function cannot currently work and there is still much to be done. Since this is my personal exploration, I still need some time to finish this.

@shu1chen understood


Hardcoded tasks.json and launch.json

Currently both files "content" is hardcoded in KS, if possible it would be nice to have them generated so one can tinker with them.

Both are now generated using the regular vscode extension methods.


Go To Line functionality: Implementation

Can anyone help with fixing the GTL Function so it doesn't crash the program?

Found fix:
def go_to_line(self):     line_number, ok = QInputDialog.getInt(self, "Go to Line", "Enter line number:", value=1)     cursor = self.textedit.textCursor()     cursor.movePosition(QTextCursor.MoveOperation.Start)     cursor.movePosition(QTextCursor.MoveOperation.Down, QTextCursor.MoveMode.MoveAnchor, line_number - 1)     # Set the cursor as the new cursor for the QTextEdit     self.textedit.setTextCursor(cursor)     # Ensure the target line is visible     self.textedit.ensureCursorVisible()

Will submit in pull request


Automate deployment

Add a deploy workflow (see MDA and others)

upload to testpypi when release tagging
Upload to PyPi when making a gh release
include a simple twine check in CI
update development docs

@IAlibay has a template for a suitable workflow. Either copy or turn it into an action and use the action.  This workflow needs testpypi and PyPi tokens as gh environment variables.

Let me see if I can copy & paste from @IAlibay 's work on the other repos...


export/ generate sql plugin

[x] I agree to follow the Code of Conduct that this project adheres to.
[x] I have searched the issue tracker for a feature request that matches the one I want to file, without success.

Is your feature request related to a problem? Please describe.
I would like an extension that lets you generate sql from an er diagram. Also have the option to select database type: sqlite, mysql, postgres, sql server.
Describe the solution you'd like
File > export as > sql option, then select database type and a preview window of the sql generated
Describe alternatives you've considered
lucid charts has this, I've also put work in making an extension work with mermaidjs markdown erdiagrams to sql see

Please don't stale this one. It's a very useful request. Please have this feature added. Especially as draw.io is available as an extension for VSCode.

Please consider integrating this feature.

+1

Thanks for your interest in the issue. However, we've decided this does not fall within the scope of the project as we see it.

The dumbest solution I've seen is not to add such a useful plugin

@lastlink how to install and run it?


Columnstore indexes support

the version is said to be 12.0.2008 (SQL2014) but SQL edition. This implies that the columnstore index features would be the same as 2014 where there had been so much improvement with 2016,2017.
For example, the following feature would then NOT be available?? Can this be documented?
Batch exec for single thread queries
snapshot isolation and read-commited snapshot isolation
specification of columnstore index when creating a table
columnstore index on a memory-optimized table

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 28ca1f7f-5a34-0f23-10ce-3ff24c76db97
Version Independent ID: 5c4a22c4-78fc-911d-9fa0-8dfab1653b6e
Content: Azure SQL Database Managed Instance Overview
Content Source: articles/sql-database/sql-database-managed-instance.md
Service: sql-database
GitHub Login: @bonova
Microsoft Alias: bonova

@kitkatneko the version number for Azure SQL Database has no relation to the on-premise SQL Server product. In fact, Azure SQL Database generally has newer code than the SQL Server product itself.

I see. still would be nice to see what features are supported.

@kitkatneko See https://docs.microsoft.com/en-us/azure/sql-database/sql-database-features Every feature of column store is supported https://docs.microsoft.com/sql/relational-databases/indexes/columnstore-indexes-overview Any limitations would be noted specifically.

@kitkatneko We will now proceed to close this thread. If there are further questions regarding this matter, please reopen it and we will gladly continue the discussion.


Add Count, Image fields to Spec

This PR adds the Count and Image fields to the spec, representing the amount of instances, and the docker image to run.
Additionally, it removes a hyphen from the pluralization of the ServiceGroup CR, used as part of the k8s group identifier. That's because official k8s groups don't have hyphens between words (e.g. apiextensions).
And finally, it changes the group name for the CR, to resemble the k8s.io format used by the official components.
Closes #9.

PTALA.


Remove trigger rule questions

This removes a long list of questions about ingredients ("trigger rule questions") as these are now covered by a single "Upload list of poisonous ingredients" step.
Review app: https://cosmetics-remove-trigger-rules-submit-web.london.cloudapps.digital
https://regulatorydelivery.atlassian.net/jira/software/projects/COSBETA/boards/24?selectedIssue=COSBETA-651
https://regulatorydelivery.atlassian.net/jira/software/projects/COSBETA/boards/24?selectedIssue=COSBETA-650

I'm wondering why there is a separate trigger_questions_controller when it simply follows on from the questions in component_build_controller. I guess we don't need to fix this now but it becomes even more silly when there's only two questions.
We need to ensure the test changes I made in trigger_questions_controller_spec in #1397 are not lost in the deletion after merging.

@slorek yeah, it no longer makes any sense to have the two controllers - especially as the TriggerQuestionsController now no longer has any interaction with the TriggerQuestion models... 😂

#1397 is now merged so please rebase and bring in my changes to the trigger questions controller ;)

👍 Looks good to me

Sorry, I just spotted on the product details page that this template https://github.com/UKGovernmentBEIS/beis-opss/blob/master/cosmetics-web/app/views/notifications/_trigger_question_details.html.slim is rendered which will display the trigger question answers when viewing a previously created notification. I think we need to check and test that this page renders correctly and remove the redundant code from there.

Ignore the above comment; I think this is desired behaviour still for historical/bulk uploads.

@slorek yeah, I think that's still required for the bulk upload notifications (which do still generate trigger question rows in the database).


CLI doesn't remove configMaps

If configuration has been added to the airy.yaml file, then the cli creates the appropriate configMaps when running airy config apply.
However, if we then delete the configuration for a particular source, then the configMap of the particular source will not be removed and the pods for the source will continue to run.
Probably the solution would involve refactoring the logic and the configuration.

In order to delete a configMap when a configuration is removed, we must have a refference point where all the components of the Airy platform are listed, so that we can iterate over them to sync the configuratio between the config file and the kubernetes cluster.
One way would be to hardcode them in the cli, for a particular version.
Another way would be to hardcode the components inside the ServiceDiscovery app, in the backend (as we have at this moment) for the airy status cli.
I think a good approach would be to have a configMap mounted on the ServiceDiscovery app, presenting a json configuraiton file, where all the components and their configurationItems are listed.
The ServiceDiscovery app can read from there and the configu=
{
  "components": {
    "core": [
      {
        "name": "facebook",
        "type": "source",
        "configItems": {
          "appId": null,
          "appKey": null
        }
      },
      {
        "name": "twilio",
        "type": "source",
        "configItems": {
          "number": 0,
          "twilioKey": null
        }
      },
      {
        "name": "webhook",
        "type": "integration",
        "configItems": "siteName"
      },
      {
        "name": "media-resolver",
        "type": "storage",
        "configItems": {
          "s3Bucket": null,
          "accessKey": null
        }
      }
    ],
    "enterprise": [
      {
        "name": "search",
        "type": "ui",
        "configItems": "key"
      }
    ]
  }
}

This way, we would also be able to generate the airy.yaml file, as the cli will have from where to fetch all the components and their configuration items.
If we change the configmap on the fly, we need to reload the serviceDiscovery app.
Also we can have a convention for the components, for the configMaps and names of the apps - {type}-{name} (source-facebook)
This concept can work at the moment and if we switch to CustomResourcesDefinitions in the future.

I think I like the idea the "only central" place is a config map (something called components-config?). That way, as you say, we can refactor the service discovery class in the client.config endpoint

As we are going to use helm on the long run, I'm just thinking out loud if it makes sense to also use it for the things that the  controller does? Because if we change the configuration part we do in config apply and run helm upgrade all the configuration will be updated, configMaps created or deleted and the affected pods will be restarted. Not sure what will not work.
It has this out of the box and then we don't even need to do this issue for deleting the configmaps.
I guess I always had a sense that helm is temporary, that in the future we would want to have our own Airy CRD + operator and didn't consider this option at all so far. But going this direction, if we make use of all of the helm possibilities in a proper way - everything will work out of the box with very little code on our side. Perhaps best option would be to to create a kubernetes job to run helm inside the cluster and apply the configuration.
We can also use the it for the airy create command and it will be consistent:

Install the components - helm install Provisioning is on a helm hook post-install (jobs that we run only once).
Apply the configuration - map the airy.yaml file somewhere, we write the logic in helm on what/how needs to run and helm upgrade

Maybe helm upgrade is not intended in the first place to apply configuration, but at this moment it looks to me as everything will just work properly. Also the other issue that we have for restarting statefulsets will also be solved. Maybe it will be good to split the charts
helm install prerequisites
helm install core
helm upgrade core

so that we can upgrade only the components and not the prerequisites. But this is a different discussion.
This is the way we can use helm to restart the deployments/statefulsets on configMap changes, by adding an annotation to the deployment/statefulset:
spec:
  template:
    metadata:
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/configMaps.yaml") . | sha256sum }}

For whether to create or not a particular component, we can use an if block
{{- if .Values.comonent.enabled }}
... (kubernetes resources) ...
{{- end }}

While it's appealing to "delete the controller" and just rely on some other component doing that, it feels really wrong to rely on helm upgrade. Helm is a package manager after all and we do rely on it a lot to install things right now. Soon enough, we'll have to provide upgrade guides (hopefully people will be using airy core in production and want the shiny new features a new version brings). In that context, I'm relatively sure we'll end up using helm upgrade the "right way". Once we do this, I'm not sure we'll like the idea we mixed into a package upgrade our own knowledge of how the different components can/should be configured.
tl;dr I think we should keep helm out of Airy Core configuration because 1) we'd be heavily misusing an helm feature we plan to use 2) we should keep ourselves in control (pun intended) of how airy config exactly works

I also think that it is better that we have better control over the config process and do it ourselves. Also having our own K8s controller can help us for sure in fine-tuning our configuration process.
The main reason why I brought this up is because at this moment I am not aware of any features which our Airy controller should have in the foreseeing future (configuration-wise) and which cannot be covered with helm upgrade. Also as things are now, we cannot really use airy config unless we do helm install first, so that is why helm upgrade did not sound that wrong to me.
But I also have a hunch that relying on our own controller will be better on the long run and I vote for sticking with the Airy k8s controller for configuration purposes :+1:

We will create one configMap with all the mandatory and optional components with helm, as part of the airy create process.

After reviewing the PR https://github.com/airyhq/airy/pull/1510 and discussing the concept, me, ... agreed on the following:

We don't need the concept of configItems and we will not put any additional annotations in the apps
Inside the apps, we will put the labels:

core.airy.co/component
core.airy.co/mandatory

Related to airy config apply whatever we find under

components:
  x:
    y:

we write in a configmap x-y

When we create the configmaps from airy config apply, we also label them with a label airy.co/component, so that we know to remove them when configuration is not present in airy.yaml
Also we agreed on introspecting the kubernetes cluster in realtime for the airy status command, so that we can get what comopnents are currently deployed and running in the system
In that sense, we don't need a single configMap which is generated at install-time
We decided to create an internal endpoint on the controller, which returns the current running components in the system - the app will look for deployments having a specific label, something like kubectl get deployment -l core.airy.co/component -o jsonpath='{.items[*].metadata.labels.core\.airy\.co/component}'
The current ServiceDiscovery java app, will talk to the internal controller endpoint and expose that information to a public endpoint

For the jwtsecret, token, allowedOrigins and similar configuration values, we agree do put them under a top-level secrtion security, because they are relevant for more components.


Tuya component broke after upgrading

Home Assistant release with the issue: 0.77.1

Last working Home Assistant release (if known): 0.76.3
Operating environment (Hass.io/Docker/Windows/etc.): PIP/python installation of HA in Ubuntu 17.10

Component/platform: Tuya

Description of problem:
Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):
tuya:
  username: !secret tuya_user
  password: !secret tuya_password
  platform: tuya
  country_code: 1

Traceback (if applicable):
2018-09-11 11:13:53 ERROR (MainThread) [homeassistant.setup] Error during setup of component tuya
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connection.py", line 171, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/util/connection.py", line 56, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 849, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connection.py", line 314, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connection.py", line 180, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7f26094a3748>: Failed to establish a new connection: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/homeassistant/setup.py", line 148, in _async_setup_component
    component.setup, hass, processed_config)  # type: ignore
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.6/dist-packages/homeassistant/components/tuya.py", line 65, in setup
    tuya.init(username, password, country_code, platform)
  File "/home/hass/.homeassistant/deps/lib/python3.6/site-packages/tuyapy/tuyaapi.py", line 44, in init
    self.get_access_token()
  File "/home/hass/.homeassistant/deps/lib/python3.6/site-packages/tuyapy/tuyaapi.py", line 59, in get_access_token
    'from': 'tuya'
  File "/usr/local/lib/python3.6/dist-packages/requests/api.py", line 112, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 512, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 622, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 513, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='px1.tuyaus.com', port=443): Max retries exceeded with url: /homeassistant/auth.do (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f26094a3748>: Failed to establish a new connection: [Errno -2] Name or service not known',))

Additional information: I see that @huangyupeng is the authority on the subject. I'm sending my username privately to you and would appreciate any help provided... Thank you!!!

Could you remove the param and have a try again? This param will be set to 'tuya' by default. This error looks like a dns problem. And several people have this problem but I still can't figure it out...

Yep continued to work after a restart of HA and with the platform attribute removed. Very strange... but appreciate the help!


Update SQL planning in carbon-spark

This PR contains code clean up for carbon-spark SQL planning:

unify late materialization planning
code clean up for strategy related code, remove Raw related naming in Strategy and RDD

retest this please

@jackylk I will merge this PR and I will raise another PR to clean up the core package.


Create a content on Linux Mint

Go for it @kodepirate

I have completed my work, but I am unable to create PR,
Help!

@1Spirit1 It will be great if you merge my PR

I was a bit busy past few days, or else I would've helped you.
Happy to see that you resolved it on your own 🙂

@1Spirit1 No problem, its no big deal.😉✌


Missing information for Service Hooks consumption and limits

With this documentation It is unclear to me how are the service hooks affecting the global consumption limit. I understand that all actions performed by a user have related TSTUs. My guess is that when a service hook is triggered it will also have some TSTUs related but I didn't find anything specific to the service hooks consumption and limits here or in the service hook documentation. Specifically I think it would make sense to specify the following in this document (or perhaps in the Service hook document):
-Do Service hooks have a resource consumption limit too? If so, is it per trigger or different?
-Do all the Service hooks on a single project share the same resource consumption or each of them is treated as an individual entity with its own resource consumption (like the pipelines) or perhaps they share the consumption with the user who created them (or even triggered them)?

Is it possible to see the usage of the Service Hooks somewhere?
If a Service hook is triggered many times, can it be blocked/delayed because of exceding a rate limit?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 5c2c2355-60a5-d53f-2c72-8f032c1b8cf2
Version Independent ID: 711bf22d-99c9-51b2-382c-e095bdb2c37f
Content: Resource limits & constraints - Azure DevOps Services
Content Source: docs/integrate/concepts/rate-limits.md
Product: devops
Technology: devops-ecosystem
GitHub Login: @roofers
Microsoft Alias: kaelli

@operick8 Thank you for the feedback.
Please see the service hooks category starting with the overview.  Many examples are included here, such as webhooks.  If you need further examples or have any suggestions please comment below.
I don't see anything about the rate limits, assigning this to the author for review.

Hi @msebolt thanks for the prompt response. I do understand how to use service hooks and for the moment I don't need examples about them. What I do want to know is about the service hooks resource consumption. I think it is something important that should be included somewhere in the documentation.  @roofers Could you help answering the questions? Thanks in advance!

Do Service hooks have a resource consumption limit too? If so, is it per trigger or different?
Do all the Service hooks on a single project share the same resource consumption or each of them is treated as an individual entity with its own resource consumption (like the pipelines) or perhaps they share the consumption with the user who created them (or even triggered them)?
Is it possible to see the usage of the Service Hooks somewhere?
If a Service hook is triggered many times, can it be blocked/delayed because of exceding a rate limit?

@operick8 - Thank you for your inquiry and apologies for the delay in response. I checked with the product team and they indicated that Rate Limits don't apply to Service Hooks.

Thanks for the response, appreciate it! It would be nice to add that information in the rate limits documentation.(https://docs.microsoft.com/en-us/azure/devops/integrate/concepts/rate-limits?view=azure-devops)


Add generator for root disk unlocking

Add generator for root disk unlocking
Currently, root disk unlocking in the initramfs has to be triggered by setting a kernel argument rd.luks... in the Ignition config. This has two disadvantages. First, it would be nicer for the users if they did not have to set a kernel argument to make root disk unlocking work. Second, during first boot, it leads to an unnecessary systemd-cryptsetup unit that (at least sometimes) fails. While this failing unit causes no harm, it does make the tests fail and might confuse users.
The systemd-cryptsetup-generator does not generate because we don't have appropriate entries in /etc/crypttab in the initramfs. This PR attempts to add a custom generator that works around this issue by generating a crypttab file and then calling systemd-cryptsetup-generator on that crypttab file.
How to use
Root disk decryption set up with storage.luks in Ignition should work without passing special kernel arguments in the Ignition config.
Testing done
This PR is not ready for review yet, as I need to test this generator locally first. The current generator will not work because it relies on files in /dev/ but generators in the initramfs run very early, before files for block devices in /dev/ are created (I think).

[ ] Changelog entries added in the respective changelog/ directory (user-facing change, bug fix, security fix, update)
[ ] Inspected CI output for image differences: /boot and /usr size, packages, list files for any missing binaries, kernel modules, config files, kernel modules, etc.

It turns out that the systemd-generators in the initramfs run very early, even before device files for block devices are created in /dev/. So far, I couldn't come up with a way for the generator to figure out the UUID of the ROOT partition under these circumstances. Any ideas would be very welcome.
If we decide that it's not feasible to implement this generator, we could instead implement a systemd service that runs in the initramfs or even during late user space boot. That service would determines the UUID of the ROOT partition and add rd.luks.uuid=$uuid to the kernel cmdline parameters in grub.cfg. It would bring the same benefits as the generator: the user would not need to specify the rd.luks parameter in their Ignition config, and we would ensure that the cmdline parameter is not yet present during the first boot.

About the timing; generators may run multiple times. We could also have a look at how the systemd gpt auto generator manages to "wait" for the device.

We could also have a look at how the systemd gpt auto generator manages to "wait" for the device.
I might be reading the code wrong, but I think the systemd-gpt-auto-generator does not itself wait for device files to appear. Instead, it generates a cryptsetup service that waits for a device file to appear, which is later created by udev (see here). We could adapt that strategy by adding a systemd service that decrypts the ROOT partition once it's available, but the whole idea was to just piggy-back on the cryptsetup-generator rather than writing our own decryption service.

About the timing; generators may run multiple times.
What did you have in mind here? One option that might work (not sure whether that's what you thought of) is to trigger re-running of generators by adding a service that runs systemctl daemon-reload. If we do this after udev, the generator I wrote would probably work because all the device files are present. However, I'm not sure running systemctl daemon-reload is a good idea, since this is a global operation and might have unintended side-effects.

Thanks, thinking about this I guess we can try to define a helper service that runs the logic to invoke the generator. Since we know what service the generator will write out, we can explicitly start this generated service with systemctl start. It's best to avoid the daemon-reload and we can do so by calling the generator with /etc/systemd/system/ as directory to write to.

Thanks, thinking about this I guess we can try to define a helper service that runs the logic to invoke the generator. Since we know what service the generator will write out, we can explicitly start this generated service with systemctl start. It's best to avoid the daemon-reload and we can do so by calling the generator with /etc/systemd/system/ as directory to write to.

Sounds good :+1: I implemented this approach. Let me know what you think.

I added the service under 31decrypt-root/ but I'm uncertain whether 31 is the most appropriate number here because I'm uncertain about what the logic behind the names of the various dracut modules in dracut/ is. Let me know if a different number prefix would be more fitting.

Hello @simoncampion, the only usable systemd command for this scenario is systemd-analyze plot. I used it succesfully during my kmod static nodes PR, to be able to see properly the systemd execution tree during initrd and after switch root.

@ader1990 Thanks for the hint! That's a neat command.


updating tests to reflect changes in image and album models

Coverage decreased (-0.78%) to 72.0% when pulling 23016f2293e66ec7f1b360aacf4459178085ea5c on updating_tests into 80ce5e480b0684142300a9d2a2fd81ae2cebdcfb on development.


[Source] Remove empty parenthesis in installing gem message

@homu r+

:pushpin: Commit d4cb1b3 has been approved by segiddins

:hourglass: Testing commit d4cb1b3 with merge a33112c...

:sunny: Test successful - status


Basic JaxPEAKVI class

Partially addresses #1578

Latent space from JPVI, matching the one from the PeakVI tutorial


HLSL: Implement geometry shaders

Just curious, has there been an progress toward this functionality?  Or does it have the same blockers as tessellation and mesh shaders?

It's still a blocker as of now


More CXF tests and cleanup

fix #4203

@llowinge I have added some fixes for MTOM in native mode, most notably https://github.com/apache/camel-quarkus/pull/4207/files#diff-7c81397449fd40a8260c63c1f182320f8aab02b7e56879589ffce1d46b36df6fR35-R38 and https://github.com/apache/camel-quarkus/pull/4207/files#diff-a6565d3563e7493df31d39fe68eba12b8d5892af636f96541f78d22e385cac8fR72-R74 but it still fails.
So I filed https://github.com/apache/camel-quarkus/issues/4208 and disabled the test in native mode https://github.com/apache/camel-quarkus/pull/4207/files#diff-e9e596e55a8e034e9d04f6b83ba0a06cc87694d14824540234ece03404e73082R37
Feel free to try to fix it.
I think it would be worth to try to see an MTOM test with java.awt.Image working on plain quarkus-cxf. If it won't work there, then we know that it is an AWT problem rather than CXF problem. Otherwise it will point in direction CQ. Note that a DataHandler MTOM test is already there in quarkus-cxf: https://github.com/quarkiverse/quarkus-cxf/tree/main/integration-tests/mtom

I've fixed it with not using the awt.* classes in the test app (it is needed for the test scenario anyway). I was unable to work on that with macOS because of https://github.com/oracle/graal/issues/4124. Fix is in commit https://github.com/llowinge/camel-quarkus/commit/2fae5ab2019b7278bc84ffdab8e6ccfdf7332953.


OLS-567: catalog refers production image + version specific dockerfile for catalog

Description
The catalog refers production image, the same as bunlde image does.
Add 4.15 and 4.16 version specific dockerfile for catalog images.
The script update_bundle_catalog.sh is updated to generate the corret catalog index.
The 2 dockerfile allow us to build FBC component for 4.15 and 4.16 catalog.
The old catalog will be removed afterwards.
Konflux component onboard PRs to merge after this PR:

ols-fbc-v4-16 https://github.com/openshift/lightspeed-operator/pull/285
ols-fbc-v4-15 https://github.com/openshift/lightspeed-operator/pull/284

Catalog files are generated by
#!/bin/bash

OPERATOR_IMAGE="registry.redhat.io/openshift-lightspeed-beta/lightspeed-rhel9-operator@sha256:049a1a398ed87e4f35c99b36304055c7f75d0188a4d8c1726df59b5f400561e5"
BUNDLE_IMAGE="registry.redhat.io/openshift-lightspeed-beta/lightspeed-operator-bundle@sha256:e46e337502a00282473e083c16a64e6201df3677905e4b056b7f48ef0b8f6e4b"
CONSOLE_IMAGE="registry.redhat.io/openshift-lightspeed-beta/lightspeed-console-plugin-rhel9@sha256:4f45c9ba068cf92e592bb3a502764ce6bc93cd154d081fa49d05cb040885155b"
SERVICE_IMAGE="registry.redhat.io/openshift-lightspeed-beta/lightspeed-service-api-rhel9@sha256:fedfaaba10cf2dd952cf735167e9898c034ecec433a22b38293aeccaeed6e3bf"
RELATED_IMAGES=$(
  cat <<-EOF
[
  {
    "name": "lightspeed-service-api",
    "image": "${SERVICE_IMAGE}"
  },
  {
    "name": "lightspeed-console-plugin",
    "image": "${CONSOLE_IMAGE}"
  },
  {
    "name": "lightspeed-operator",
    "image": "${OPERATOR_IMAGE}"
  }
]
EOF
)

find config/ -type f -name "*.yaml" | while read -r manifest; do
  echo "Updating ${manifest}"
  sed -i 's|quay\.io/redhat-user-workloads/crt-nshift-lightspeed-tenant/ols/lightspeed-operator@sha256:[0-9a-f]\{64\}|'"${OPERATOR_IMAGE}"'|g' ${manifest}
  sed -i 's|quay\.io/redhat-user-workloads/crt-nshift-lightspeed-tenant/ols/lightspeed-console@sha256:[a-z0-9]\{64\}|'"${CONSOLE_IMAGE}"'|g' ${manifest}
  sed -i 's|quay\.io/redhat-user-workloads/crt-nshift-lightspeed-tenant/ols/lightspeed-service@sha256:[0-9a-f]\{64\}|'"${SERVICE_IMAGE}"'|g' ${manifest}
done

# Put BUNDLE_IMAGE=$BUNDLE_IMAGE when the bunlde is built and pushed to the staging registry
OPERATOR_IMAGE=${OPERATOR_IMAGE} RELATED_IMAGES=${RELATED_IMAGES} BUNDLE_IMAGE=${BUNDLE_IMAGE} CATALOG_FILE=lightspeed-catalog-4.15/index.yaml ./hack/update_bundle_catalog.sh
OPERATOR_IMAGE=${OPERATOR_IMAGE} RELATED_IMAGES=${RELATED_IMAGES} BUNDLE_IMAGE=${BUNDLE_IMAGE} CATALOG_FILE=lightspeed-catalog-4.16/index.yaml ./hack/update_bundle_catalog.sh
OPERATOR_IMAGE=${OPERATOR_IMAGE} RELATED_IMAGES=${RELATED_IMAGES} BUNDLE_IMAGE=${BUNDLE_IMAGE} CATALOG_FILE=lightspeed-catalog/index.yaml ./hack/update_bundle_catalog.sh

Type of change

[ ] Refactor
[ ] New feature
[ ] Bug fix
[ ] CVE fix
[ ] Optimization
[ ] Documentation Update
[x] Configuration Update
[ ] Bump-up dependent library

Related Tickets & Documents

Related Issue # OLS-567
Closes #

/override "Red Hat Konflux / lightspeed-operator-fbc-on-pull-request"


defaultOptions on Module vs defaultOptions on input config

I'm submitting a ...  (check one with "x")
[ X ] bug report => search github for a similar issue or PR before submitting
[ X ] feature request
[ ] support request

Current behavior
It's possible when we configure the fields on a component, to add a templateOptions descriptionWrapper to disable the display after the input of the error when an input is in error.
templateOptions: {          descriptionWrapper: false,         label: 'Address',         required: true,         classLabel: 'col-md-10 col-sm-10 col-xs-24 control-label',         classDivInput: 'col-sm-8 col-xs-24',         type: 'text',       },
Expected behavior
Sometimes we have a custom input define in our module of our application.
Instead of making false the descriptionWrapper in every each field in our component, it could be nice to add this configuration on a custom input direct inside the declaration in the module like this
types: [         {            name: 'custom',            component: FormlyFieldCustomInput,            wrappers: ['default'],           defaultOptions: {             templateOptions: {               descriptionWrapper: false,             }           }         },       ],
What is the motivation / use case for changing the behavior?
Remove boilerplate when the case arrive often

Angular version: 6.0.3

already supported with the same way that you've suggested, have you tried it?

description and validation wrappers are deprecated since 4.7 and removed in v5.0, normally a warning should be displayed in console I'll take a look later to see what's wrong with it.
which version you use ?

in v5 either create a custom wrapper with a name form-field to override the default one
or just ensure not passing description in templateOption or pass a false value.
but for validation I don't think so, is there any reasonable reason to not showing it ?

I have in my project a custom wrapper with no description and no validation
So for description, it's working in version 4.7.2
But for validation, it adds automatically the validation message even if I have already a place where validation message should display
`

		{{ to.label }}

			<formly-validation-message [field]="field"></formly-validation-message>

		<ng-template #fieldComponent></ng-template>

`
On my module I have :
FormlyModule.forRoot({
wrappers: [
{ name: 'default', component: DefaultWrapperComponent },
],
types: [
{
name: 'custom',
component: FormlyFieldCustomInput,
wrappers: ['default'],
defaultOptions: {
templateOptions: {
descriptionWrapper: false,
}
}
},
],
}),
where I define the custom wrapper and a custom input
With descriptionWrapper inside the module, nothing happen even for validationWrapper

since you've have used a custom wrapper normally it should work in v5, I'll try to find what's wrong there.

Thank you!
I will try to take a look!
So you said it should work either on 4.7 and 5?

I think there no issue here  descriptionWrapper and validationWrapper are not required in v5, so if you don't use them inside you custom wrapper you can remove them.
https://stackblitz.com/edit/angular-6d8efg

But without that, I have the custom validation before the input and after the input..
This is what I don't understand..

I found the issue, we used descriptionWrapper instead of validationWrapper in https://github.com/formly-js/ngx-formly/blob/v4/src/bootstrap/src/lib/run/validation.ts#L6

Ok cool!
So is it possible to show me a example with a custom wrapper where every input which derived from this wrapper will get validation wrapper to false so the validation will not display by default?

the easy way to get rid of those wrappers is to rename your default wrapper to 'form-field':
https://stackblitz.com/edit/angular-6d8efg

Ok I will take a look tomorrow!
But it could be a great option to be able to do it to a lot of wrapper and  not with rename to form-field only..
Thank you for the quick answer!