fetching all stocks

hi @Chavithra per your request issue instead of discord.
I am trying out the library and am confused by the following. I would like to get a list of all tradable stocks from USA by exchange.
I am trying the following loop:
# get all USA stocks in green list
index = 0
products = []
while True:
    print(index)
    r = ProductSearch.RequestStocks(
        stock_country_id=846, #USA
        index_id=index,
        offset=0,
        is_in_us_green_list=True,
        limit=100,
        require_total=True,
        sort_columns="name",
        sort_types="asc",
    )
    res = message_to_dict(trading_api.product_search(request=r, raw=False))
    if len(res['products']) == 0:
        break
    else:
        index += 1
        products += res['products']
        print(len(res['products']))

however this does not give me all stocks that I can see from the UI.
I am confused by the index_id and offset parameters. I tried both and for both had a confusing result that it didnt give me a lot of the stocks that I saw on the UI. Could you perhaps give details what the difference between those two is and how I can fix my code? Thanks in advance!

Hello there,
Pagination
In the ProductSearch endpoint, the pagination is handled with these two parameters :

offset
limit

This is how most Databases (SQL or NoSQL) handle the pagination : with parameters for the offset and limit.
Indices
The index parameter seems to be the stock index.
Examples of stock index :

CAC 40
Dow Jones Industrial Average
Nasdaq Composite
S&P 500

You can get all the available indices for each country in the ProductsConfig table.
As mentioned in the documentation : This table contains useful parameters to filter products. .
Hope that helps

hi @Chavithra thanks for your answer
I thought as much and a I already tried that. For example I pulled the NASDAQ id from the productsconfig:
products_info = message_to_dict(trading_api.get_products_config())

then pulled out nasdaq:
{'country': 846.0, 'id': 663.0, 'exchange': 663.0, 'postfix': 'NASDAQ'},
and have the index 663
and then tried this code:
index = 0
products = []
while True:
    print(index)
    r = ProductSearch.RequestStocks(
        stock_country_id=846, #USA
        index_id=663,
        offset=index*100,
        is_in_us_green_list=True,
        limit=100,
        require_total=True,
        sort_columns="name",
        sort_types="asc",
    )
    res = message_to_dict(trading_api.product_search(request=r, raw=False))
    if len(res['products']) == 0:
        break
    else:
        index += 1
        products += res['products']
        print(len(res['products']))

but then I always get zero products?

extra remark, I see that the column 'productBitTypes' does not always contain US_RAS_GREEN_LIST even though I put "is_in_us_green_list=True" in there. does that mean that filter is broken? or does that column mean something else?

Congrats
It's good that you understood for the offset : there is progress indeed.

With your feedback and by doing some tests I found out that the ProductSearch endpoint uses two parameters to filter on indices (what is called indices on the website).
Here are this two parameters :

index_id
exchange_id

So according to what you are looking for your need to set the right parameter in the ProductSearch.RequestStock.
New release
A new version of this library was released : you can know use exchange_id inside ProductSearch.RequestStock.
ProductsConfig
To found out which index_id and exchange_id are available according to a country you can use the ProductsConfig table.
Example for the US :
{
    ...
    "countries": [
        ...
        {
            "id": 846,
            "name": "US",
            "region": 2,
            "translation": "list.country.846"
        },
        ...
    ],
    ...
    "stockCountries": [
        ...
        {
            "country": 846,
            "exchanges": [
                650,
                1004,
                663
            ],
            "id": 846,
            "indices": [
                12,
                122001,
                13,
                14
            ]
        },
        ...
    ],
    ...
    "exchanges": [
        ...
        {
            "city": "New York",
            "code": "XNAS",
            "country": "US",
            "hiqAbbr": "NDQ",
            "id": 663,
            "micCode": "XNAS",
            "name": "NASDAQ"
        },
        ...
    ],
    ...
    "indices": [
        {
            "id": 122001,
            "name": "NASDAQ 100",
            "productId": 12153880
        },
        ...
    ],
    ...
}

extra remark, I see that the column 'productBitTypes' does not always contain US_RAS_GREEN_LIST even though I put "is_in_us_green_list=True" in there. does that mean that filter is broken? or does that column mean something else?

Do you have a specific example ?

hi @Chavithra Thank you for the quick new release. However the new version throws an error for me. I have looked up the ids in product config like so:
# find the stock exchange
[x for x in product_info['exchanges'] if x['name'] == 'NASDAQ'][0]
# this is for specific subsets of an index
[x for x in product_info['indices'] if x['name'] == 'NASDAQ 100'][0]
# find the country id
[x for x in product_info['countries'] if x['name'] == 'US'][0]

which returns the following three dictionaries:
{'country': 'US', 'city': 'New York', 'name': 'NASDAQ', 'id': 663.0, 'code': 'XNAS', 'micCode': 'XNAS', 'hiqAbbr': 'NDQ'}
{'productId': 12153880.0, 'name': 'NASDAQ 100', 'id': 122001.0}
{'name': 'US', 'region': 2.0, 'id': 846.0, 'translation': 'list.country.846'}

then I use the follow code to fetch the stocks:
r = ProductSearch.RequestStocks(
    stock_country_id=846, #USA
    index_id=663, #NASDAQ
    exchange_id=122001, #NASDAQ 100
    offset=0,
    #search_text = "activision",
    is_in_us_green_list=True,
    limit=100,
    require_total=True,
    sort_columns="name",
    sort_types="asc",
)
res = message_to_dict(trading_api.product_search(request=r, raw=False))

but this throws the following error:
<Response [400]>
400 Client Error: Bad Request for url: https://trader.degiro.nl/product_search/secure/v5/stocks?isInUsGreenList=True&indexId=663&exchangeId=122001&stockCountryId=846&limit=100&requireTotal=True&sortColumns=name&sortTypes=asc&intAccount=<snip>&sessionId=<snip>BC3FF02A3E564EE06E31AFAD.prod_a_168_5

AttributeError                            Traceback (most recent call last)
/tmp/ipykernel_12571/2263861331.py in <module>
     11     sort_types="asc",
     12 )
---> 13 res = message_to_dict(trading_api.product_search(request=r, raw=False))

~/pycharmprojects/finance/venv/lib/python3.9/site-packages/degiro_connector/core/helpers/pb_handler.py in message_to_dict(message)
     10
     11 def message_to_dict(message: Message) -> dict:
---> 12     return json_format.MessageToDict(
     13         message=message,
     14         including_default_value_fields=True,

~/pycharmprojects/finance/venv/lib/python3.9/site-packages/google/protobuf/json_format.py in MessageToDict(message, including_default_value_fields, preserving_proto_field_name, use_integers_for_enums, descriptor_pool, float_precision)
    163       float_precision=float_precision)
    164   # pylint: disable=protected-access
--> 165   return printer._MessageToJsonObject(message)
    166
    167

~/pycharmprojects/finance/venv/lib/python3.9/site-packages/google/protobuf/json_format.py in _MessageToJsonObject(self, message)
    197   def _MessageToJsonObject(self, message):
    198     """Converts message to an object according to Proto3 JSON Specification."""
--> 199     message_descriptor = message.DESCRIPTOR
    200     full_name = message_descriptor.full_name
    201     if _IsWrapperMessage(message_descriptor):

AttributeError: 'NoneType' object has no attribute 'DESCRIPTOR'

extra remark, I see that the column 'productBitTypes' does not always contain US_RAS_GREEN_LIST even though I put "is_in_us_green_list=True" in there. does that mean that filter is broken? or does that column mean something else?

Do you have a specific example ?

the same code as above. but I think its better to first focus on the error that I get now with the new version

You haved used this code to get the exchange_id :
# find the stock exchange
[x for x in product_info['exchanges'] if x['name'] == 'NASDAQ'][0]

Result:
{'country': 'US', 'city': 'New York', 'name': 'NASDAQ', 'id': 663.0, 'code': 'XNAS', 'micCode': 'XNAS', 'hiqAbbr': 'NDQ'}

So :
exchange_id = 663

But in your request you have put :
exchange_id=122001

hi @Chavithra thank you that was indeed my mistake I swapped the index and exchange ids, oops.
my code for only nasdaq100 is now:
r = ProductSearch.RequestStocks(
    stock_country_id=846, #USA
    exchange_id=663, #NASDAQ
    index_id=122001, #NASDAQ 100
    is_in_us_green_list=True,
    limit=100,
    require_total=True,
    sort_columns="name",
    sort_types="asc",
)
res = message_to_dict(trading_api.product_search(request=r, raw=False))

my code to fetch ALL nasdaq stocks is:
index = 0
products = []
while True:
    print(index)
    r = ProductSearch.RequestStocks(
        stock_country_id=846, #USA
        exchange_id=663, #NASDAQ
        offset=index*100,
        is_in_us_green_list=True,
        limit=100,
        require_total=True,
        sort_columns="name",
        sort_types="asc",
    )
    res = message_to_dict(trading_api.product_search(request=r, raw=False))
    if len(res['products']) == 0:
        break
    else:
        index += 1
        products += res['products']

some observations:

using raw=True like you say results in a crash with message_to_dict, I guess that makes sense. I use that function since the raw dictionary has many superfluous keys.
offset is ignored when getting a result that is less then 100 results. That makes kinda sense but if you dont know the number of results on beforehand and you just keep increasing the offset you will keep getting the same stocks over and over again. something to keep in mind

an error I still see:
when fetching all stocks with is_in_us_green_list=True I still see many stocks that have property productBitTypes as an empty list instead of [US_RAS_GREEN_LIST]. to me that seems this property is ignored?

Total
If you look at the result of product_search method, you can see that the first parameter the API returns is the total :
{
    'total': 84,
    'offset': 0,
    'products': [{'id': '332109'...}]
}

productBitTypes
No idea what this productBitTypes attribute is, you will have to find it by yourself.

okay thanks for the help!


There seems to be something wrong with the alembic

I copied alembic ⚗ to the clipboard and then pasted it to the console (iterm2) and here is what I see:

That does not happen with, for instance, lock icon:

Also, it looks ok when I paste the alembic to another medium, such as here: ⚗
One more thing: I have noticed that with lock, there is a space between the icon and to in the message Copied 🔒 to the clipboard, while with alembic it does not exist.

Thanks for reporting this.
That's strange indeed. gimoji is just taking the UTF-8 string/character verbatim from the emoji db provided by gitmoji project. One thing you can do to check if the issue is in the DB, is C&P from the db directly (file ~/.cache/gimoji/emojis.json, you may need to adjust the path if Mac uses a different cache dir). If it is, the issue should be reported to gitmoji then.
Down the line, we may provide our own db.

@rimbi I know you went down a rabbit hole with this one but don't forget the original intention of quickly finding out the cache location. :wink:

First of all, the entity value for the alembic in the cache and thus in the gitmoji url is wrong but it is irrelevant to pur issue.

Could you please report that to gitmoji?

When pasted, the terminal uses the first code ⚗ to display the alembic and prints the second code  as fe0f

So gimoji copies two characters into the clipboard? :thinking:

Yes and I have created an issue on gitmoji.
Btw, I have used another similar tool moji and I see the same behaviour.
I think the problem is related to the terminal's handling of the pasted character.
The following experiment seems to support my hypothesis:

echoing the pasted character results in the correct emoji.
I believe we can close this issue.

@rimbi Ah ok, thanks so much for investigating this. :pray:
On a related note, I've decided to make the emoji db part of the binary itself and keep a clone of the gitmoji's json in the source. The emoji list is small enough (and unlikely to grow a lot) so it's ok even if our fork had to diverge. Each time emojis are update (rarely), we'll roll out a release. So instead of gimoji -u, people will just do cargo install -f gimoji (although I'm hoping that we can package for most distros and Mac soon so the update is handled by package managers.


locally ?

clear && python3.10 -m flask run
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://<IP_ADDRESS>:5000 (Press CTRL+C to quit)
<IP_ADDRESS> - - [14/May/2022 03:46:29] "GET / HTTP/1.1" 404 -

Calamity locally ? (wsgi ?)
make serve doesn't speak to model (but is loaded), means no generation
(base) macos@iPro-von-MacOS calamity-main % make serve
oak src/main.oak
Calamity running at :7575
GET: /
GET: /css/main.css
GET: /js/torus.min.js
GET: /js/bundle.js
POST: /gen

So... this is where I have to apologize that the setup is kind of convoluted. As I note in the readme:

There are two independent web apps: the model and API server, and the web app that wraps and calls the API and serves the client user interface. It's designed this way so that the API itself can be reused across other apps in my personal infrastructure, some of which aren't public yet.

But this does make it weird to run outside of my infra. To run it locally, you need to run both client and server apps locally and tell the client where the server is running.
When you start the client server (make serve) run it with the env var telling it where the backend is running with AI_INFERENCE_URL="http://localhost:5000". Since you're running it locally you don't need to mess with auth stuff, just set the URL. Then if you go to localhost:7575, you should see the "Generate" call make a POST to the Flask server, which means the two are talking to each other.
Hope that makes sense! Let me know if you run into more troubles.

I get response but no connection I tray get some flask logs
(AI) macos@iPro-von-MacOS service % python3.10 -m flask run
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://<IP_ADDRESS>:5000 (Press CTRL+C to quit)
<IP_ADDRESS> - - [14/May/2022 20:23:34] "POST / HTTP/1.1" 404 -
<IP_ADDRESS> - - [14/May/2022 20:25:13] "POST / HTTP/1.1" 404 -
<IP_ADDRESS> - - [14/May/2022 20:25:17] "POST / HTTP/1.1" 404 -

Ah, wait, sorry — I think I misspoke earlier because it had been a while since I set it up. AI_INFERENCE_URL needs to be the full path to the API endpoint which is http://localhost:5000/generate/.

Using AI_INFERENCE_URL="http://localhost:5000/generate/" ./oak src/main.oak
Model Kicks up but not to normal usage and now again  no response to flask

I'm not exactly sure what you mean by that last sentence, but if you're running the full gpt-j-6B model, it's quite a large model, and may take a long time for you to generate text on a slower computer. (So you may not see the Flask server respond for a long time). I run it on a server with 8 CPUs and 64GB of RAM, and it takes 15-30sec to generate 100 tokens/words. If your machine has less than 32GB of RAM, you won't be able to run that size model at all, and will need to use a smaller model from Huggingface Transformers.

i just mean that model runs but strange maybe huggingface was do update of pytorch_model.bin
but in [calamity](url) i see not outputs

I see that the Flask server is returning a 200 OK response to the POST /generate/ request. That indicates to me that the model is generating responses correctly and Flask returns a correct response, and it is just not being returned to the browser. If you open the Network pane in your browser's developer tools and see the request and response for the generate request, what do you see?
As a side note - if you're running things locally, nothing in calamity.service, including the log stuff, matters. That's only for deployment using systemd.

but oak build --entry src/app.js.oak --output static/js/bundle.js --web without errors

I see that in your screenshot, at the bottom there's an error that tells us the request to /gen is failing with the 403 Forbidden error code. This should mean that in the Networks panel in your DevTools ("Netzwerk" above in your screenshot) there should be something showing more data about that failed request.
I'm not sure where that "Forbidden" response is coming from, because I don't think any code in this repository ever generates a 403 response. To my understanding, if you're running these two commands at the same time:

flask run
AI_INFERENCE_URL="http://localhost:5000/generate/" ./oak src/main.oak

And open localhost:7575, everything should just work. Your screenshot from this comment makes me think that your backend (Flask) is receiving the right request, but your most recent screenshot tells me that it actually isn't receiving the request.... I'm not sure what to make of that discrepancy :/


python3Packages.afdko: 3.9.0 -> 3.9.2

Description of changes
Update python packages

afdko 3.9.0 -> 3.9.2 https://github.com/adobe-type-tools/afdko/blob/develop/NEWS.md#392-released-2023-01-10
lxml 4.9.1 -> 4.9.2 https://github.com/lxml/lxml/compare/lxml-4.9.1...lxml-4.9.2
fonttools 4.34.4 -> 4.38.0 https://github.com/fonttools/fonttools/releases

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

@ofborg build noto-fonts-emoji

@sternenseemann Thank you, done :)

@ofborg build python3Packages.afdko.tests noto-fonts-emoji python3Packages.pygments


Why the performance slowly by double 1080ti than single 1080ti?

I write a code for double 1080ti , I find the speed not reduce than single 1080ti, but I check the information by nvidia-smi the double 1080ti is working, I don't understand why the double 1080ti almost same speed than single 1080ti ?
I use the tensorboard to watch the profile:

It is looks input waste lots of time , how to fix this problem?
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
The full code:
Double 1080ti:

#!/usr/bin/env python
# coding: utf-8

# In[1]:

import sys
import os
from datetime import datetime
#from packaging import version
os.environ["CUDA_VISIBLE_DEVICES"]="0,1"
batch_size = 128
epochs = 500
IMG_HEIGHT = 224
IMG_WIDTH = 224
NUM_WORKERS = 4

PATH = os.path.abspath(os.path.dirname(__file__))

train_dir = os.path.join(PATH, 'train')
validation_dir = os.path.join(PATH, 'valid')

train_close_dir = os.path.join(train_dir, 'close')  # directory with our training cat pictures
train_open_dir = os.path.join(train_dir, 'open')  # directory with our training dog pictures
validation_close_dir = os.path.join(validation_dir, 'close')  # directory with our validation cat pictures
validation_open_dir = os.path.join(validation_dir, 'open')  # directory with our validation dog pictures

import tensorflow as tf

print (tf.__version__)

# In[2]:

print("Version: ", tf.__version__)
print("Eager mode: ", tf.executing_eagerly())
print("GPU is", "available" if tf.config.experimental.list_physical_devices("GPU") else "NOT AVAILABLE")

gpus = tf.config.experimental.list_physical_devices('GPU')
for g in gpus:
    tf.config.experimental.set_virtual_device_configuration(g, [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=7000)])

# In[3]:

from tensorflow import keras
from tensorflow.keras.preprocessing.image import ImageDataGenerator
import numpy as np
import random

import matplotlib.pyplot as plt

# In[4]:
#mirrored_strategy = tf.distribute.experimental.MultiWorkerMirroredStrategy()
mirrored_strategy = tf.distribute.MirroredStrategy()

num_close_tr = len(os.listdir(train_close_dir))
num_open_tr = len(os.listdir(train_open_dir))

num_close_val = len(os.listdir(validation_close_dir))
num_open_val = len(os.listdir(validation_open_dir))

total_train = num_close_tr + num_open_tr
total_val = num_close_val + num_open_val

print('total training close images:', num_close_tr)
print('total training open images:', num_open_tr)
print('total validation close images:', num_close_val)
print('total validation open images:', num_open_val)
print("--")
print("Total training images:", total_train)
print("Total validation images:", total_val)

if len(sys.argv) > 1:
    batch_size = int(sys.argv[1])

if len(sys.argv) > 2:
    epochs = int(sys.argv[2])

def preprocess_image(image):
    image = tf.image.decode_jpeg(image, channels=3)
    image = tf.image.resize(image, [image_size, image_size])
    image /= 255.0
    return image

def load_and_preprocess_image(path):
    image = tf.io.read_file(path)
    return preprocess_image(image)

train_image_generator = ImageDataGenerator(rescale=1./255, horizontal_flip=True, zoom_range=0.1, rotation_range=45) # Generator for our training data
validation_image_generator = ImageDataGenerator(rescale=1./255) # Generator for our validation data

train_data_gen = train_image_generator.flow_from_directory(batch_size=batch_size,
                                                           directory=train_dir,
                                                           shuffle=True,
                                                           target_size=(IMG_HEIGHT, IMG_WIDTH),
                                                           class_mode='binary')

val_data_gen = validation_image_generator.flow_from_directory(batch_size=batch_size,
                                                              directory=validation_dir,
                                                              target_size=(IMG_HEIGHT, IMG_WIDTH),
                                                              class_mode='binary')

sample_training_images, _ = next(train_data_gen)

# In[5]:

def plotImages(images_arr):
    fig, axes = plt.subplots(1, 5, figsize=(20, 20))
    axes = axes.flatten()
    for img, ax in zip( images_arr, axes):
        ax.imshow(img)
        ax.axis('off')
    plt.tight_layout()
    plt.show()

plotImages(sample_training_images[:5])

# In[6]:

# In[7]:

with mirrored_strategy.scope():
    tinydarknet = keras.Sequential([
        keras.layers.Conv2D(16, (3, 3), strides=[1, 1], padding="same", input_shape=(IMG_HEIGHT,IMG_WIDTH, 3)),
        keras.layers.BatchNormalization(),
        keras.layers.LeakyReLU(alpha=0.1),
        keras.layers.MaxPooling2D((2, 2), strides=(2, 2)),
        keras.layers.Conv2D(32, (3, 3), strides=[1, 1], padding="same"),
        keras.layers.BatchNormalization(),
        keras.layers.LeakyReLU(alpha=0.1),
        keras.layers.MaxPooling2D((2, 2), strides=(2, 2)),
        keras.layers.Conv2D(16, (1, 1), strides=[1, 1], padding="same"),
        keras.layers.BatchNormalization(),
        keras.layers.LeakyReLU(alpha=0.1),
        keras.layers.Conv2D(128, (3, 3), strides=[1, 1], padding="same"),
        keras.layers.BatchNormalization(),
        keras.layers.LeakyReLU(alpha=0.1),
        keras.layers.Conv2D(16, (1, 1), strides=[1, 1], padding="same"),
        keras.layers.BatchNormalization(),
        keras.layers.LeakyReLU(alpha=0.1),
        keras.layers.Conv2D(128, (3, 3), strides=[1, 1], padding="same"),
        keras.layers.BatchNormalization(),
        keras.layers.LeakyReLU(alpha=0.1),
        keras.layers.MaxPooling2D((2, 2), strides=(2, 2)),
        keras.layers.Conv2D(32, (1, 1), strides=[1, 1], padding="same"),
        keras.layers.BatchNormalization(),
        keras.layers.LeakyReLU(alpha=0.1),
        keras.layers.Conv2D(256, (3, 3), strides=[1, 1], padding="same"),
        keras.layers.BatchNormalization(),
        keras.layers.LeakyReLU(alpha=0.1),
        keras.layers.Conv2D(32, (1, 1), strides=[1, 1], padding="same"),
        keras.layers.BatchNormalization(),
        keras.layers.LeakyReLU(alpha=0.1),
        keras.layers.Conv2D(256, (3, 3), strides=[1, 1], padding="same"),
        keras.layers.BatchNormalization(),
        keras.layers.LeakyReLU(alpha=0.1),
        keras.layers.MaxPooling2D((2, 2), strides=(2, 2)),
        keras.layers.Conv2D(64, (1, 1), strides=[1, 1], padding="same"),
        keras.layers.BatchNormalization(),
        keras.layers.LeakyReLU(alpha=0.1),
        keras.layers.Conv2D(512, (3, 3), strides=[1, 1], padding="same"),
        keras.layers.BatchNormalization(),
        keras.layers.LeakyReLU(alpha=0.1),
        keras.layers.Conv2D(64, (1, 1), strides=[1, 1], padding="same"),
        keras.layers.BatchNormalization(),
        keras.layers.LeakyReLU(alpha=0.1),
        keras.layers.Conv2D(512, (3, 3), strides=[1, 1], padding="same"),
        keras.layers.BatchNormalization(),
        keras.layers.LeakyReLU(alpha=0.1),
        keras.layers.Conv2D(128, (1, 1), strides=[1, 1], padding="same"),
        keras.layers.BatchNormalization(),
        keras.layers.LeakyReLU(alpha=0.1),
        keras.layers.Conv2D(1000, (1, 1)),
        keras.layers.BatchNormalization(),
        keras.layers.AveragePooling2D(),
        keras.layers.Flatten(),
        keras.layers.Dense(1)
    ])

    # In[8]:

    tinydarknet.compile(optimizer="adam",
                 loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
                 metrics=["accuracy"])

# In[9]:
logs = "logs/" + datetime.now().strftime("%Y%m%d-%H%M%S")
tboard_callback = tf.keras.callbacks.TensorBoard(log_dir = logs,
                                                 histogram_freq = 1,
                                                 profile_batch = '500,520')

history = tinydarknet.fit(#_generator(
    train_data_gen,
    steps_per_epoch=total_train // batch_size,
    epochs=epochs,
    validation_data=val_data_gen,
    validation_steps=total_val // batch_size,
    workers=NUM_WORKERS,
    callbacks = [tboard_callback]
)

# In[10]:

acc = history.history['accuracy']
val_acc = history.history['val_accuracy']

loss=history.history['loss']
val_loss=history.history['val_loss']

epochs_range = range(epochs)

tinydarknet.save("keras_model")

converter = tf.lite.TFLiteConverter.from_keras_model(model)
tflite_model = converter.convert()

with open("tinydarknet.tflite", "w+b") as fp:
    fp.write(tflite_model)
    fp.flush()

The single 1080ti code:
#!/usr/bin/env python
# coding: utf-8

# In[1]:

import sys
import os

batch_size = 128
epochs = 500
IMG_HEIGHT = 224
IMG_WIDTH = 224
NUM_WORKERS = 4

PATH = os.path.abspath(os.path.dirname(__file__))

train_dir = os.path.join(PATH, 'train')
validation_dir = os.path.join(PATH, 'valid')

train_close_dir = os.path.join(train_dir, 'close')  # directory with our training cat pictures
train_open_dir = os.path.join(train_dir, 'open')  # directory with our training dog pictures
validation_close_dir = os.path.join(validation_dir, 'close')  # directory with our validation cat pictures
validation_open_dir = os.path.join(validation_dir, 'open')  # directory with our validation dog pictures

import tensorflow as tf

print (tf.__version__)

# In[2]:

print("Version: ", tf.__version__)
print("Eager mode: ", tf.executing_eagerly())
print("GPU is", "available" if tf.config.experimental.list_physical_devices("GPU") else "NOT AVAILABLE")

gpus = tf.config.experimental.list_physical_devices('GPU')
tf.config.experimental.set_virtual_device_configuration(gpus[0], [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=6000)])

# In[3]:

from tensorflow import keras
from tensorflow.keras.preprocessing.image import ImageDataGenerator
import numpy as np
import random

import matplotlib.pyplot as plt

# In[4]:

num_close_tr = len(os.listdir(train_close_dir))
num_open_tr = len(os.listdir(train_open_dir))

num_close_val = len(os.listdir(validation_close_dir))
num_open_val = len(os.listdir(validation_open_dir))

total_train = num_close_tr + num_open_tr
total_val = num_close_val + num_open_val

print('total training close images:', num_close_tr)
print('total training open images:', num_open_tr)
print('total validation close images:', num_close_val)
print('total validation open images:', num_open_val)
print("--")
print("Total training images:", total_train)
print("Total validation images:", total_val)

if len(sys.argv) > 1:
    batch_size = int(sys.argv[1])

if len(sys.argv) > 2:
    epochs = int(sys.argv[2])

def preprocess_image(image):
    image = tf.image.decode_jpeg(image, channels=3)
    image = tf.image.resize(image, [image_size, image_size])
    image /= 255.0
    return image

def load_and_preprocess_image(path):
    image = tf.io.read_file(path)
    return preprocess_image(image)

train_image_generator = ImageDataGenerator(rescale=1./255, horizontal_flip=True, zoom_range=0.1, rotation_range=45) # Generator for our training data
validation_image_generator = ImageDataGenerator(rescale=1./255) # Generator for our validation data

train_data_gen = train_image_generator.flow_from_directory(batch_size=batch_size,
                                                           directory=train_dir,
                                                           shuffle=True,
                                                           target_size=(IMG_HEIGHT, IMG_WIDTH),
                                                           class_mode='binary')

val_data_gen = validation_image_generator.flow_from_directory(batch_size=batch_size,
                                                              directory=validation_dir,
                                                              target_size=(IMG_HEIGHT, IMG_WIDTH),
                                                              class_mode='binary')

sample_training_images, _ = next(train_data_gen)

# In[5]:

def plotImages(images_arr):
    fig, axes = plt.subplots(1, 5, figsize=(20, 20))
    axes = axes.flatten()
    for img, ax in zip( images_arr, axes):
        ax.imshow(img)
        ax.axis('off')
    plt.tight_layout()
    plt.show()

plotImages(sample_training_images[:5])

# In[6]:

# In[7]:

tinydarknet = keras.Sequential([
    keras.layers.Conv2D(16, (3, 3), strides=[1, 1], padding="same", input_shape=(IMG_HEIGHT,IMG_WIDTH, 3)),
    keras.layers.BatchNormalization(),
    keras.layers.LeakyReLU(alpha=0.1),
    keras.layers.MaxPooling2D((2, 2), strides=(2, 2)),
    keras.layers.Conv2D(32, (3, 3), strides=[1, 1], padding="same"),
    keras.layers.BatchNormalization(),
    keras.layers.LeakyReLU(alpha=0.1),
    keras.layers.MaxPooling2D((2, 2), strides=(2, 2)),
    keras.layers.Conv2D(16, (1, 1), strides=[1, 1], padding="same"),
    keras.layers.BatchNormalization(),
    keras.layers.LeakyReLU(alpha=0.1),
    keras.layers.Conv2D(128, (3, 3), strides=[1, 1], padding="same"),
    keras.layers.BatchNormalization(),
    keras.layers.LeakyReLU(alpha=0.1),
    keras.layers.Conv2D(16, (1, 1), strides=[1, 1], padding="same"),
    keras.layers.BatchNormalization(),
    keras.layers.LeakyReLU(alpha=0.1),
    keras.layers.Conv2D(128, (3, 3), strides=[1, 1], padding="same"),
    keras.layers.BatchNormalization(),
    keras.layers.LeakyReLU(alpha=0.1),
    keras.layers.MaxPooling2D((2, 2), strides=(2, 2)),
    keras.layers.Conv2D(32, (1, 1), strides=[1, 1], padding="same"),
    keras.layers.BatchNormalization(),
    keras.layers.LeakyReLU(alpha=0.1),
    keras.layers.Conv2D(256, (3, 3), strides=[1, 1], padding="same"),
    keras.layers.BatchNormalization(),
    keras.layers.LeakyReLU(alpha=0.1),
    keras.layers.Conv2D(32, (1, 1), strides=[1, 1], padding="same"),
    keras.layers.BatchNormalization(),
    keras.layers.LeakyReLU(alpha=0.1),
    keras.layers.Conv2D(256, (3, 3), strides=[1, 1], padding="same"),
    keras.layers.BatchNormalization(),
    keras.layers.LeakyReLU(alpha=0.1),
    keras.layers.MaxPooling2D((2, 2), strides=(2, 2)),
    keras.layers.Conv2D(64, (1, 1), strides=[1, 1], padding="same"),
    keras.layers.BatchNormalization(),
    keras.layers.LeakyReLU(alpha=0.1),
    keras.layers.Conv2D(512, (3, 3), strides=[1, 1], padding="same"),
    keras.layers.BatchNormalization(),
    keras.layers.LeakyReLU(alpha=0.1),
    keras.layers.Conv2D(64, (1, 1), strides=[1, 1], padding="same"),
    keras.layers.BatchNormalization(),
    keras.layers.LeakyReLU(alpha=0.1),
    keras.layers.Conv2D(512, (3, 3), strides=[1, 1], padding="same"),
    keras.layers.BatchNormalization(),
    keras.layers.LeakyReLU(alpha=0.1),
    keras.layers.Conv2D(128, (1, 1), strides=[1, 1], padding="same"),
    keras.layers.BatchNormalization(),
    keras.layers.LeakyReLU(alpha=0.1),
    keras.layers.Conv2D(1000, (1, 1)),
    keras.layers.BatchNormalization(),
    keras.layers.AveragePooling2D(),
    keras.layers.Flatten(),
    keras.layers.Dense(1)
])

# In[8]:

tinydarknet.compile(optimizer="adam",
             loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
             metrics=["accuracy"])

# In[9]:

history = tinydarknet.fit(#_generator(
    train_data_gen,
    steps_per_epoch=total_train // batch_size,
    epochs=epochs,
    validation_data=val_data_gen,
    validation_steps=total_val // batch_size,
    workers=NUM_WORKERS
)

# In[10]:

acc = history.history['accuracy']
val_acc = history.history['val_accuracy']

loss=history.history['loss']
val_loss=history.history['val_loss']

epochs_range = range(epochs)

tinydarknet.save("keras_model")

converter = tf.lite.TFLiteConverter.from_keras_model(tinydarknet)
tflite_model = converter.convert()

with open("tinydarknet.tflite", "w+b") as fp:
    fp.write(tflite_model)
    fp.flush()

# In[ ]:

- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 18.04
- TensorFlow installed from (source or binary):binary by pip
- TensorFlow version (use command below): 2.3.0
- Python version:Python 3.7.7
- CUDA/cuDNN version:10.2
- GPU model and memory: 1080ti

Hi @smallworld-network-wupeng, can you start by sharing profiling information for your single GPU model? When trying to debug performance issues it's often helpful to start with the simplest case and make sure that you aren't input bound with one GPU as well. That could indicate either a bottleneck from the transformation or extraction of your data.
I also noticed that in your multi-GPU case you were using callbacks to log histograms on each epoch, but there are no callbacks for the single-GPU case. It's worth comparing the results with the same callbacks and frequency, since logging this information can also slow down the training process. These ops are executed eagerly and don’t have as good a performance as the ops inside a TF graph.
Lastly, what did you use as your batch size? Typically, you want to increase your batch size as you add more accelerators so as to make effective use of the extra computing power.
If single GPU case you have BATCH_SIZE = 8
For the multi GPU case you'll want to set your batch size to be BATCH_SIZE = 8 * mirrored_strategy.num_replicas_in_sync'

Hi @smallworld-network-wupeng, can you start by sharing profiling information for your single GPU model? When trying to debug performance issues it's often helpful to start with the simplest case and make sure that you aren't input bound with one GPU as well. That could indicate either a bottleneck from the transformation or extraction of your data.
I also noticed that in your multi-GPU case you were using callbacks to log histograms on each epoch, but there are no callbacks for the single-GPU case. It's worth comparing the results with the same callbacks and frequency, since logging this information can also slow down the training process. These ops are executed eagerly and don’t have as good a performance as the ops inside a TF graph.
Lastly, what did you use as your batch size? Typically, you want to increase your batch size as you add more accelerators so as to make effective use of the extra computing power.
If single GPU case you have BATCH_SIZE = 8
For the multi GPU case you'll want to set your batch size to be BATCH_SIZE = 8 * mirrored_strategy.num_replicas_in_sync'

The single gpu profile:

Looks like your program is still input bound in the single GPU case. You'll want to first work on that first before trying to scale to two GPUs, as adding more GPUs will likely only exacerbate the bottleneck.
Firstly, you should try tuning the batch size in the single GPU case. If your batch size is too small then your GPU might be waiting around for the CPU to process and prepare the data for the next step of training.
The TF guides have a lot of suggestions on how to improve your input pipeline when using tf.data.Dataset which is recommended when doing distributed training. I think you should be able to wrap your ImageDataGenerator in the from_generator method and get the benefits of using tf.data. There is an example of this here in the Consuming Python generators section of the tf.data guide.
Please see the performance optimization guide for tf.data and this Tensorboard Profiler tutorial for more suggestions on how to debug bottlenecks in your input data pipeline. I think this video is also very helpful in understanding the different ways your input pipeline can be a bottleneck.
Hope these resources help you make your program less input bound!

Yes, it seems that the ImageDataGenerator is what is causing your input bottleneck. Did you try wrapping it in the from_generator method as I mentioned above? (see example in docs here). Another option (although more work) is to use tf.data and the tf.image library to do the augmentation. This tutorial might help, specifically the Using tf.image section.


BMW F22 2015 not working correctly in Hass

I am not getting locks,light, correct location etc in Hass with this component. Running hass 0.77.1
I have attached the status and vehicles files from running fingerprint
I get the following errors in the hass log.
2018-08-31 08:39:32 ERROR (SyncWorker_4) [bimmer_connected.state] No data available! 2018-08-31 08:39:32 ERROR (SyncWorker_0) [bimmer_connected.state] No data available! 2018-08-31 08:39:32 ERROR (SyncWorker_1) [bimmer_connected.state] No data available! 2018-08-31 08:39:32 ERROR (MainThread) [bimmer_connected.state] No data available! 2018-08-31 08:39:32 ERROR (MainThread) [bimmer_connected.state] No data available! 2018-08-31 08:39:32 ERROR (MainThread) [homeassistant.core] Error doing job: Task exception was never retrieved Traceback (most recent call last):   File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/helpers/entity_platform.py", line 344, in _async_add_entity     await entity.async_update_ha_state()   File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/helpers/entity.py", line 243, in async_update_ha_state     device_attr = self.device_state_attributes   File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/components/lock/bmw_connected_drive.py", line 70, in device_state_attributes     'door_lock_state': vehicle_state.door_lock_state.value AttributeError: 'NoneType' object has no attribute 'value' 2018-08-31 08:39:32 ERROR (SyncWorker_2) [bimmer_connected.state] No data available! 2018-08-31 08:39:32 ERROR (SyncWorker_9) [bimmer_connected.state] No data available! 2018-08-31 08:39:32 ERROR (SyncWorker_3) [bimmer_connected.state] No data available! 2018-08-31 08:39:32 ERROR (MainThread) [bimmer_connected.state] No data available! 2018-08-31 08:39:32 ERROR (MainThread) [homeassistant.components.binary_sensor] bmw_connected_drive: Error on device update! Traceback (most recent call last):   File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/helpers/entity_platform.py", line 251, in _async_add_entity     await entity.async_device_update(warning=False)   File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/helpers/entity.py", line 353, in async_device_update     yield from self.hass.async_add_job(self.update)   File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run     result = self.fn(*self.args, **self.kwargs)   File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/components/binary_sensor/bmw_connected_drive.py", line 164, in update     self._state = vehicle_state.has_check_control_messages   File "/srv/homeassistant/lib/python3.6/site-packages/bimmer_connected/state.py", line 71, in _func_wrapper     return func(self, *args, **kwargs)   File "/srv/homeassistant/lib/python3.6/site-packages/bimmer_connected/state.py", line 323, in has_check_control_messages     return len(self.check_control_messages) > 0 TypeError: object of type 'NoneType' has no len() 2018-08-31 08:39:32 ERROR (SyncWorker_5) [bimmer_connected.state] No data available! 2018-08-31 08:39:32 ERROR (MainThread) [homeassistant.components.binary_sensor] bmw_connected_drive: Error on device update! Traceback (most recent call last):   File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/helpers/entity_platform.py", line 251, in _async_add_entity     await entity.async_device_update(warning=False)   File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/helpers/entity.py", line 353, in async_device_update     yield from self.hass.async_add_job(self.update)   File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run     result = self.fn(*self.args, **self.kwargs)   File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/components/binary_sensor/bmw_connected_drive.py", line 162, in update     self._state = not vehicle_state.are_all_cbs_ok   File "/srv/homeassistant/lib/python3.6/site-packages/bimmer_connected/state.py", line 224, in are_all_cbs_ok     for cbs in self.condition_based_services: TypeError: 'NoneType' object is not iterable 2018-08-31 08:39:32 ERROR (MainThread) [homeassistant.core] Error doing job: Task exception was never retrieved Traceback (most recent call last):   File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/helpers/entity_platform.py", line 344, in _async_add_entity     await entity.async_update_ha_state()   File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/helpers/entity.py", line 243, in async_update_ha_state     device_attr = self.device_state_attributes   File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/components/binary_sensor/bmw_connected_drive.py", line 118, in device_state_attributes     result['lights_parking'] = vehicle_state.parking_lights.value AttributeError: 'NoneType' object has no attribute 'value' 2018-08-31 08:39:32 ERROR (MainThread) [homeassistant.core] Error doing job: Task exception was never retrieved Traceback (most recent call last):   File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/helpers/entity_platform.py", line 344, in _async_add_entity     await entity.async_update_ha_state()   File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/helpers/entity.py", line 243, in async_update_ha_state     device_attr = self.device_state_attributes   File "/srv/homeassistant/lib/python3.6/site-packages/homeassistant/components/binary_sensor/bmw_connected_drive.py", line 115, in device_state_attributes     result['door_lock_state'] = vehicle_state.door_lock_state.value AttributeError: 'NoneType' object has no attribute 'value'
vehicles_0.txt
status_0.txt

@sevenofnine-jni the server does not return any data (see your status_0.txt), so there is no way to display these attributes for your vehicle. I guess this is related to your hmiVersion = ID4.
Concerning the error messages this is something we should fix, see #83.


Missions: Fetch Misssions

Fetch data from Mission API

Implement fetch functionality
Add a function to filter response data from API to get info to add to the store
Import fetch function and passed it to Missions component
Dispatch fetch Data in Missions component

Thank you @Ginohmk, I have made the required changes


Add /api  to base URL

Copied from GitLab:

This line assumes that baseUrl contains a trailing slash. We need some means of injecting a slash if its missing either here or in the sdk (whereby the SDK appends /api itself, rather than the user having to provide it.
This may actually be the better way of doing it so we have full control over the URLs - in case we change /api to another name or something... 🤷

:tada: This issue has been resolved in version 1.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Problem of Quick Start in Readme.md

I tried to run the code in the QuickStart and I found some problems.
federated_data = fedjax.FederatedData() can not be executed because it is an abstract class. So I replaced it as
client_a_data = {
        'x': np.array([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]),
        'y': np.array([7, 8])
    }
client_b_data = {'x': np.array([[9.0, 10.0, 11.0]]), 'y': np.array([12])}
client_to_data_mapping = {'a': client_a_data, 'b': client_b_data}
federated_data = fedjax.InMemoryFederatedData(client_to_data_mapping)

The other things are same as the QuickStart, but i got an error
for client_id, client_output, _ in func(shared_input, clients):
for client_id, client_batches, client_input in clients:
ValueError: not enough values to unpack (expected 3, got 2)

It seems that client_batches is missing and we need to batch the dataset, but there is no example which fits this situation.

I would recommend looking at https://github.com/google/fedjax/blob/main/examples/emnist_fed_avg.py instead for a real working example.

Thanks for brining this to our attention!
I just submitted an update to our README so the code snippet is a working example #240
This code snippet is meant to just be a quick "hello world" example to make sure FedJAX is installed correctly.
As mentioned already, https://github.com/google/fedjax/blob/main/examples is the better place to look for real working examples.


Aerial on High Sierra

I have Aerial running successfully on an iMac and a MBP 2015. However, it is not working on the new MBP with TouchBar. Do you intend to make this compatible with High Sierra. I love this app and am spoiled!

Duplicate of #381

Duplicate of #377


Add new cop Lint/OrAssignmentToConstant

This cop checks unintended or-assignment to constant.
Usually, you don't need to consider whether A is already assigned elsewhere.
# bad
CONST ||= 1

# good
CONST = 1

Before submitting the PR make sure the following are checked:

[x] The PR relates to only one subject with a clear title and description in grammatically correct, complete sentences.
[x] Wrote good commit messages.
[ ] ~Commit message starts with [Fix #issue-number] (if the related issue exists).~
[x] Feature branch is up-to-date with master (if not - rebase it).
[x] Squashed related commits together.
[x] Added tests.
[x] Ran bundle exec rake default. It executes all tests and runs RuboCop on its own code.
[x] Added an entry (file) to the changelog folder named {change_type}_{change_description}.md if the new code introduces user-observable changes. See changelog entry format for details.

Thank you for review.
I fixed the three places you pointed out, resolved conflicts, and squashed them.

Thank you for review.
I fixed the three places you pointed out, resolved conflicts, and squashed them.

ci/circleci: cc-upload-coverage keeps failing on this PR, anyone have any ideas why?

It's possible that Code Climate's API wasn't working properly.  I think this failure can be ignored.
#!/bin/bash -eo pipefail
./tmp/cc-test-reporter sum-coverage tmp/codeclimate.*.json --parts 6 --output tmp/codeclimate.total.json
./tmp/cc-test-reporter upload-coverage --input tmp/codeclimate.total.json
Error: response from https://api.codeclimate.com/v1/test_reports.
HTTP 401: You are not authorized for this action
Usage:
  cc-test-reporter upload-coverage [flags]

Flags:
  -s, --batch-size int    batch size for source files (default 500)
  -e, --endpoint string   endpoint to upload coverage information to (default "https://api.codeclimate.com/v1/test_reports")
  -r, --id string         reporter identifier (default "a11b66bfbb1acdf220d5cb317b2e945a986fd85adebe29a76d411ad6d74ec31f")
  -i, --input string      input path (default "coverage/codeclimate.json")
      --insecure          send coverage insecurely (without HTTPS)

Global Flags:
  -d, --debug   run in debug mode

Exited with code exit status 255
CircleCI received exit code 255

ci/circleci: cc-upload-coverage keeps failing on this PR, anyone have any ideas why?

It's possible that Code Climate's API wasn't working properly.  I think this failure can be ignored.
#!/bin/bash -eo pipefail
./tmp/cc-test-reporter sum-coverage tmp/codeclimate.*.json --parts 6 --output tmp/codeclimate.total.json
./tmp/cc-test-reporter upload-coverage --input tmp/codeclimate.total.json
Error: response from https://api.codeclimate.com/v1/test_reports.
HTTP 401: You are not authorized for this action
Usage:
  cc-test-reporter upload-coverage [flags]

Flags:
  -s, --batch-size int    batch size for source files (default 500)
  -e, --endpoint string   endpoint to upload coverage information to (default "https://api.codeclimate.com/v1/test_reports")
  -r, --id string         reporter identifier (default "a11b66bfbb1acdf220d5cb317b2e945a986fd85adebe29a76d411ad6d74ec31f")
  -i, --input string      input path (default "coverage/codeclimate.json")
      --insecure          send coverage insecurely (without HTTPS)

Global Flags:
  -d, --debug   run in debug mode

Exited with code exit status 255
CircleCI received exit code 255

@koic yeah that’s what I figured but I wanted to double check, thanks.

@koic yeah that’s what I figured but I wanted to double check, thanks.


Create a Git usage guide

As I have run or participated in Hackathons for allReady, one of the consistent issues that I have run into is people not understanding how to use Git.    Many times we have had participants that it is their first time using Git or Github.  It would be nice to have a guide that walks them through the Git process for forking, cloning, commiting, pushing, submitting PR, and syncing fork with HTBox/allReady.
Just a basic guide that walks them through what they need to know to be a successful contributor.

I can take this one

For reference:
https://www.stevejgordon.co.uk/contributing-to-allready
http://www.davepaquette.com/archive/2016/01/24/Submitting-Your-First-Pull-request.aspx
have been our most successful guides to date
Thanks for jumping on this

thank you for the additional links.  I will take a look at them.

I have a draft blog post covering some Git related tips and tricks / GitHub etiquette (Gitiquette) as well which I will eventually be publishing!

Along with @stevejgordon's upcoming "Gititquette" blog post, this is a great link to how write a well formatted git commit message: http://chris.beams.io/posts/git-commit/.


file not found './Job_Descriptions', also not after adding new job description in the folder

When trying to run the 'screen' function, it gives the error that it cannot locate the file. This is probably because the os.chdir redirecting it to the wrong folder (./Job_Descriptions when it is still in Original_Resumes). Even after resetting the os.chdir to '../', errors keep coming in. Please help.

Same I am facing the same issue.

I have created a PR, pretty bad tho. I'll debug this when I get time. Apologizes!
Change line 124 in screen.py to -
f = open(os.getcwd() + jobfile[17:] , 'r', encoding="utf8", errors='ignore')

That should work!

It's actually:
    f = open(os.getcwd() + '/../Job_Description' + jobfile[17:] , 'r', encoding="utf8", errors='ignore')


[design doc] Support multiple cnis

PROBLEM
k3s only supports automatic deployments with flannel CNI plugin. It is possible to deploy with other CNI plugins but it requires manual intervention. We would like to change this and be able to automatically deploy other CNI plugins.
USE CASE
As as user, I'd like to have a config parameter (--cni) that I can set to "cilium", "calico", "multus,cilium" or "multus,calico". When that happens:
1 - Flannel will not get deployed (flannel-backend=none)
2 - Network policy controller will be disabled (disable-network-policy=true)
3 - The set CNI plugin will get installed
IMPLEMENTATION
Flannel remains embedded in k3s binary but the rest of the CNI plugins will not. They will be consumed using helm in a similar way as we do in rke2. However, to avoid k3s depending on rke2, k3s will consume charts from a different chart repo.
One manifest for each new CNI plugin will be placed in the manifests directory. These manifests will contain a HelmChart object pointing to the chart tarball using spec.chart. For example:
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: calico
  namespace: kube-system
spec:
  chart: $url_to_calico_tarball.tgz
  set:
    global.systemDefaultRegistry: "%{SYSTEM_DEFAULT_REGISTRY_RAW}%"

In func run of pkg/cli/server/server.go, there will be a switch based on the cni value and if case "calico", cilium and multus will be part of serverConfig.ControlConfig.Skips, serverConfig.ControlConfig.FlannelBackend = "none", etc.
QUESTIONS
1 - Should flannel be treated as a different CNI plugin? I.e. we don't change how we are currently deploying flannel
2 - How to maintain the k3s charts? Should we create some sort of k3s-charts similar to rke2-charts?

This design doc looks great! Is there any activity going on?
Is it simpler if we make multus as a helmchart like how upstream works, once multus is installed, it will be the primary CNI.

We decided against supporting multiple selectable components serving the same function in K3s, in favor of a single opinionated choice for each component.

k3s will drop multus support?

K3s has never supported Multus; all we ship is Flannel. Are you thinking of RKE2?

Closing per comment about choosing to be opinionated.


Allow Project admins to force set password of Project-scoped Users

Currently, Force Set Password is only available to the server superadmin.  For a User scoped to a specific Project (i.e. with User.projectId set), their Project admins should be able to force (re)set their password

This refers to the feature that allows admin users to Forcibly set passwords from the Medplum App


Support ApplicationConfiguration controller

Description of your changes

This PR adds two relatively unrelated pieces of functionality:

Getters and setters for the unstructured content of a fieldpath.Paved This helps convert a *fieldpath.Paved to and from an *unstructured.Unstructured. The API matches that of *unstructured.Unstructured
An Apply function that either creates or patches a resource. We've implemented this in at least two controllers, and I've found myself doing it again.

The only theme linking these commits is "stuff I need to implement the OAM ApplicationConfiguration controller". :)
Checklist

I have:

[x] Run make reviewable to ensure this PR is ready for review.
[x] Ensured this PR contains a neat, self documenting set of commits.
[ ] Updated any relevant documentation, examples, or release notes.
[ ] Updated the RBAC permissions in clusterrole.yaml to include any new types.

What do you think?

@muvaf made a solid case for only supporting the "constant patch" option. This is effectively what I've done, by implementing the Patch interface in a fashion that returns a constant patch based on the original object.

@negz Thanks for writing modified version of ConstantPatch where the input is directly the object instead of []byte to bury that conversion in the Data function. I'll update templating-controller, too, to use this function once merged.

I'll update templating-controller, too, to use this function once merged.

@muvaf I'll hopefully merge today; I haven't actually run this code yet so I want to do that first. :)

@muvaf @hasheddan I added two commits just now. They’re both fairly small updates that relate to the two existing approved commits, so I’m going to cheekily hit merge but I’d appreciate it if you could take a retroactive look when you come online.


Change search background

This pr adds the possibility to change the background
of the search area.
Signed-off-by: Gergely Csatari<EMAIL_ADDRESS>

Is this the same as #404?

Assuming there won't be many of these style overrides, I like this approach more than the one in #404.
Two things I'm not certain on:

Do we need to prefix this option with something like style_ so that we can group future style overrides, or do we think we will not have any more?
This allows changing of the CSS background-color property. Would CSS background be a better choice so one can do background images or gradients as well?

Assuming there won't be many of these style overrides, I like this approach more than the one in #404.
Two things I'm not certain on:

Do we need to prefix this option with something like style_ so that we can group future style overrides, or do we think we will not have any more?
This allows changing of the CSS background-color property. Would CSS background be a better choice so one can do background images or gradients as well?

Yes I think we should prefix this.
I think so, it gives more freedom.

Thanks for the suggestions @jessetan and @Blendify I've applied both of them.

Code looks OK, just needs some documentation here: https://github.com/rtfd/sphinx_rtd_theme/blob/master/docs/configuring.rst#html-theme-options

@jessetan documentation is added.

@ericholscher , @agjohnson can you please take a look please?

This feels oddly specific. Is there a reason we'd only support this one setting for a background color? It feels like having the theme be more overridable in general is a good goal, but this is a half measure. The comments on #404 touch on this, but having a full set of colors that can be overridden feels like the proper solution.

On second thought, this is a good approach for now, but we should do a larger rethink in #404.

Thanks @ericholscher . We can remove this approach once #404 is merged and use a more generic one.


fix(package.json): updated razzle and razzle ts plugin, added mapper for tests

Big Sur try to starts tests then throw exception:
EMFILE: too many open files, watch at FSEvent.FSWatcher._handle.onchange (internal/fs/watchers.js:178:28)
Mapper added for resolve #16

@cinic Thank you!


fix: upgrade to version 3.2.60

from: https://github.com/jenkins-x/jx

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To complete the pull request process, please assign abayer
You can assign the PR to them by writing /assign @abayer in a comment when ready.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@jenkins-x-bot-test: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

gsm-kapp
91e4147b85b5bcde35846df305a0b97d4b3ef589
link
/test gsm-kapp

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the jenkins-x/lighthouse repository. I understand the commands that are listed here.

/retest


removes cli verison in engine

see https://github.com/TryGhost/Ghost/issues/11844

This would break our CLI tool


feat: migrate code base to TypeScript

Codecov Report
Attention: 9 lines in your changes are missing coverage. Please review.

Comparison is base (0ae06f1) 85.03% compared to head (c6b53ee) 88.60%.

Files
Patch %
Lines

src/processFiles.ts
78.94%
4 Missing :warning:

src/index.ts
87.50%
2 Missing :warning:

src/scms/git.ts
89.47%
2 Missing :warning:

src/scms/hg.ts
95.65%
1 Missing :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #175      +/-   ##
==========================================
+ Coverage   85.03%   88.60%   +3.56%
==========================================
  Files           8        8
  Lines         127      158      +31
  Branches       48       46       -2
==========================================
+ Hits          108      140      +32
  Misses         15       15
+ Partials        4        3       -1

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Encapsulate the hex module error types

We do not need to expose the hex error types in our public API, doing so is just asking for a headache later on.
Fully encapsulate all hex module errors.
Ouch, there is some complication to this because hashes currently exposes HexToArrayError as well. See duplicate code in hashes/src/error.rs and bitcoin/src/error.rs.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

-11 of 17 (35.29%)  changed or added relevant lines in 3 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.02%) to 84.126%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

bitcoin/src/crypto/key.rs
5
7
71.43%

bitcoin/src/crypto/ecdsa.rs
1
4
25.0%

bitcoin/src/error.rs
0
6
0.0%

Totals

Change from base Build<PHONE_NUMBER>:
-0.02%

Covered Lines:
19327

Relevant Lines:
22974

💛  - Coveralls

@apoelstra can you please merge #2335 so I can undraft this PR, and then use it in #2444.

Interesting use of pub(crate) on the internal error, to make map_err(error::HexToArrayError). It feels a little magical but I guess it's ok.

hmmm, after playing with hex yesterday and finding errors unusable in rust-bitcoin we may be better off providing constructors and using map_err(FooError::new)? Makes sense if we are going to provide getters.

I think FooError::new wouldn't work, in general, because we'd want it to be polymorphic.

FWIW I just realized that this error is different to other errors, it exists solely to hide the hex crate. We aren't likely to want add anything to it. Also it makes no sense for a user to have a hex error and want to turn it into a bitcoin::error hex error because they don't know the version of hex we are using.

Oh boy, this PR because really hard to do if we introduce all the generics in https://github.com/rust-bitcoin/hex-conservative/pull/67

To remove the generics I used this horrible code
/// Hex decoding error.
// Intentionally opaque so as to hide `hex` from the public API - do not make the inner error pub.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct FromHexInvalidCharError(pub(crate) hex::FromHexError<hex::InvalidCharError>);

impl fmt::Display for FromHexInvalidCharError {
    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { write_err!(f, "from hex"; self.0) }
}

#[cfg(feature = "std")]
impl std::error::Error for FromHexInvalidCharError {
    fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { Some(&self.0) }
}

/// Hex decoding error.
// Intentionally opaque so as to hide `hex` from the public API - do not make the inner error pub.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct FromHexHexToArrayError(pub(crate) hex::FromHexError<hex::InvalidCharError>);

impl fmt::Display for FromHexHexToArrayError {
    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { write_err!(f, "from hex"; self.0) }
}

#[cfg(feature = "std")]
impl std::error::Error for FromHexHexToArrayError {
    fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { Some(&self.0) }
}

I've just realized that this is pointless because hash_newtype! impls FromStr returning HexToArrayError. So either we have to modify the macro to allow wrapping errors (benefit: we can say "failed to parse transaction ID" in the error message), remove FromStr (please no) or stabilize hex-conservative.

Or we could wrap HexToArrayError once in bitcoin_hashes and change hash_newtype! to use that. We can even stick a &'static str into the wrapped error to give a description of the hash so that we can say "failed to parse transaction ID".

Yes, and only stabilize hashes. I was hoping we wouldn't need to stabilize hashes, just the output of hash_newtype macro but maybe it's not really possible/easy enough.

Draft till the dust settles on the hex crate error work.

We aren't doing this anymore.


Roll Engine from bae6f4490209 to d2fc4e48c4a1 (1 revision)

https://github.com/flutter/engine/compare/bae6f4490209...d2fc4e48c4a1
2021-01-04<EMAIL_ADDRESS>Roll Skia from fc914f43cdee to 2a92f13579fb (1 revision) (flutter/engine#23405)
If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-engine-flutter-autoroll
Please CC<EMAIL_ADDRESS>on the revert to ensure that a human
is aware of the problem.
To report a problem with the AutoRoller itself, please file a bug:
https://bugs.chromium.org/p/skia/issues/entry?template=Autoroller+Bug
Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/master/autoroll/README.md

Commit queue failed; closing this roll.

Commit queue failed; closing this roll.


see window around the Gnome-Pie graph

On archlinux i didn't see this window container around the gnome-pie graph.
On Fedora-34, i do install it from github clone then compile, and i see the window container.
What i did wrong and how to fix this please ?

This behavior strongly depends on the window manager and there seems to be no real ubiquitous solution. Which window manager are you using? Gnome-Pie actually requests a not-decorated window:
https://github.com/Schneegans/Gnome-Pie/blob/02e26da4ee56c3874df20dc49336ede07acce107/src/renderers/pieWindow.vala#L128
But this is ignored by some. There is also the type hint, which makes some window managers remove decorations:
https://github.com/Schneegans/Gnome-Pie/blob/02e26da4ee56c3874df20dc49336ede07acce107/src/renderers/pieWindow.vala#L127
You could try another type hint there. Here's a list: https://valadoc.org/gdk-3.0/Gdk.WindowTypeHint.html
For example, this.set_type_hint(Gdk.WindowTypeHint.SPLASHSCREEN); could work.

I'm using Awesome tiling Window Manager.
When i use it under Archlinux with Gnome-Pie; these window container doesn't appear, and under Fedora-34 with awesome, it does appear.
So i conclude that it should be possible to do the same (no window container to appear) on the same Window manager (Awesome) but with different operating system... so the question is not "Can i do it", but much more: "How to do it" ?
I thing that if i can be able to understand "WHY" this does append, then i should be able also to find the solution...

Did you compile it on Arch from source as well? Maybe the package maintainers modified this part?

No, from AUR repo named "gnome-pie-git"

Maybe they took the develop branch of Gnome-Pie. There it's set to DIALOG.


mktemp: illegal option -- -

On MacOS I'm seeing the following error when trying to start a debug session:

It turns out that mktemp on MacOS doesn't support the "--dry-run" option flag. It needs to be changed to "-u" on line 75 https://github.com/rogalmic/vscode-bash-debug/blob/6714b2f7e190030ce1e37d2d3336cd3983206bf5/src/bashDebug.ts
This change worked for me but to be honest I'm not exactly sure the flags are 100% equivilant. The man page for MacOS mktemp says:

-u  Operate in 'unsafe' mode.  The temp file will be unlinked before mktemp exits.  This is slightly better than mktemp(3) but still introduces a race condition.  Use of this option is not encouraged.

Yes, I think it is safe to change that:
" -u, --dry-run       do not create anything; merely print a name (unsafe)"

Usage of mktemp was removed, better to handle that internally in typescipt code...


vs code, invalid syntax error on preview with plugin due to tildes

Describe the bug
Using vscode plugin to show preview of mermaid code.  It kept giving a "syntax error in graph mermaid version 9.4.0".
To Reproduce
Steps to reproduce the behavior:
use mermaid code below and generate preview.  Error should occur.  Remove the 3 rows with the ~~~ and generate preview, error doesn't happen.  It's just not working with the tildes.
flowchart TB
%%B --> sefact
%%B --> F
subgraph Color_Key
direction TB  %% <-- here
mtbl[middle table]:::class1
~~~stbl[source table]
~~~strm[stream]
end

**Expected behavior**
a preview of the mermaid diagram

**Desktop (please complete the following information):**
- OS: Ventura 13.4.1(c)
- VSCode version 1.80.2 (Universal)
- Extension version 0.18.1

Thank you for your report! I think this is caused by the old mermaid version. I published v0.19.1 with the latest mermaid.js. Could you try it?


Unable to create a new calendar

I am trying to create new calendar, but when I call $Service->calendars->insert($Calendar)
I get the following error
Call to undefined method Google_Service_Calendar_Calendar::toSimpleObject()

Full Code
$Service	= new Google_Service_Calendar($this->Client);
$Calendar = new Google_Service_Calendar_Calendar();
$Calendar->setSummary('My Events');
$Calendar->setTimeZone('America/Los_Angeles');
$createdCalendar = $Service->calendars->insert($Calendar);

Which version of the client library are you using? That is a fairly fundamental method its missing.

Looks like I was overloading Google_Model with my own model which have the same name.
After figuring this out, everything is working fine.


Get IRenderedFragment/IRenderedComponent from DOM element (IElement)

In certain scenarios (#152), it can make it easier to create test when it is possible to get a component from a DOM element, due to the good DOM query (Find/FindAll) API.
Thus, it should be possible to go from an node to the component that rendered it.
The API could have two methods, GetComponent and GetComponent<TComponent>, and they should return a IRenderedFragment or IRenderedComponent<MyComp>, which is related to the nearest component in the render tree, that rendered the element. E.g.:
var elm = cut.Find("p");
IRenderedFragment componentThatRenderedElm = elm.GetComponent();
IRenderedComponent<MyComp> componentThatRenderedElm = elm.GetComponent<MyComp>();

It should be possible by adding meta information to the DOM tree created by the Htmlizer.cs type, and creating extension methods on INode that reads up the DOM tree until it finds some meta information about the source component. That extension method can then get the renderer and use its methods to find the component instance.
Update
With the current render pipeline, where we generate markup with Htmlizer and then convert to AngleSharp nodes does seems to contain too many compromises when info about the source component should be embedded in the nodes in the dom tree, at least for the variants I've tried, so I will move this item over to the AngleSharpRenderer project, where it will be a completely different store.
I tried adding custom <!--!X--> comments to markup in Htmlizer, but doing so had too much impact on other parts of the system. Another variant added a custom attribute to the first element in a component, the reasoning being that elements are what users query from using Find/FindAll. However, if a component simply contained a markup string, and not calls to "OpenElement", then the components would not get the attribute.

I am not sure about the name of the method. The method should return the component that rendered the element, but calling it GetComponent my lead some to think it will get a component which is a child of the element.
GetOwningComponent, GetSourceComponent, GetRenderingComponent are some other options.

As DOM elwlents are owned by components, elm.GetOwningComponent()

Does anyone know how to go from IRenderedFragment to IElement? Trying to assert css styles and ComputeCurrentStyle only exists on IElement

Would be a nice feature!


bug in background slider Project

Describe the bug
The images are not showing in background slider image .

@Ayush7614 Is this issue available under gssoc'22


Failing ESLint react/prop-types in Filter of columns

I use React-table v.6.10.0 and I try to custom filter to select drop down
but I got warning from react/prop-types told me missing in props validation
It's the same issue as https://github.com/tannerlinsley/react-table/issues/1207 but in Filter of columns
Filter: ({ filter, onChange }) => (
          <select onChange={event => onChange(event.target.value)} style={styles.filter} value={filter ? filter.value : ''}>
           <option value="all">ALL</option>
           <option value="yes">YES</option>
           <option value="no">NO</option>
            ))}
          </select>
        ),

Hi there! It looks as though you've submitted an issue or feature request for v6 of React Table, which I am no longer maintaining or supporting. I invite you to try the latest alpha release of v7 and help me push towards a v7 release soon. Thankyou!
If you still wish to discuss v6, please use https://spectrum.chat/react-table.


Suggestion: Common configuration function with validation

The providers are very repetitive atm, they don't validate the config, and attempt to get the env vars for id/secret even if those values are passed in the given additional config props, meaning that a consume of the lib must allow envs.
  return new OAuth2Client({
    clientId: Deno.env.get("GOOGLE_CLIENT_ID")!,
    clientSecret: Deno.env.get("GOOGLE_CLIENT_SECRET")!,
    authorizationEndpointUri: "https://accounts.google.com/o/oauth2/v2/auth",
    tokenUri: "https://oauth2.googleapis.com/token",
    ...additionalOAuth2ClientConfig,
  });

a simply function could standardize this, providing validation that all values are set as required and provide a place to improve/fix issues should they crop up (eg. fallback on env vars rather than requiring them).
a simple example of usage of such a fn:
const config = createOAuthConfig({
  name: "Google",
  authorizationEndpointUri: "https://accounts.google.com/o/oauth2/v2/auth",
  tokenUri: "https://oauth2.googleapis.com/token",
  ...additionalOAuth2ClientConfig,
});

This might be best as an internal function. Are you able to open a separate PR so we can explore this further? 🙂


feat: add auto-tls service

Adds an optional service that requests a Let's Encrypt-style TLS certificate when publicly dialable addresses are detected.
This will allow transports such as WebSockets to upgrade themselves to be the secure version.
Change checklist

[x] I have performed a self-review of my own code
[x] I have made corresponding changes to the documentation if necessary (this includes comments as well)
[x] I have added tests that prove my fix is effective or that my feature works

Merging to make landing #2800 easier


Grape::Exceptions::UnknownValidator

I have the following files:
# app/controllers/api/v2/base.rb
module Api
  module V2
    class Base < Grape::API
      mount V2::Jobs
    end
  end
end

# app/controllers/api/v2/jobs.rb
module Api
  module V2
    class Jobs < Grape::API
      resources :jobs do
        params do
          requires :id, type: String, uuid: true
        end
        get ':id' do
        end
      end
    end
  end
end

# app/controllers/api/v2/validators/uuid.rb
module Api
  module V2
    module Validators
      class Uuid < ::Grape::Validations::Base

        REGEX = %r{[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}}

        def validate_param!(attr_name, params)
          unless (params[attr_name] =~ REGEX).zero?
            fail ::Grape::Exceptions::Validation, params: [@scope.full_name(attr_name)], message: 'must be a valid UUID'
          end
        end

      end
    end
  end
end

If I want the above to work, I have to manually require the validator class in base.rb file, like so:
require_relative './validators/uuid'

which result in throwing warnings:
app/controllers/api/v2/validators/uuid.rb:6: warning: already initialized constant Api::V2::Validators::Uuid::REGEX
app/controllers/api/v2/validators/uuid.rb:6: warning: previous definition of REGEX was here

since rails will automatically require the validator as well. Any idea why grape is not able to pick up the custom validator? Looks like a require order issue or something. Any clue?
Thanks

Yes, It seems like a require order issue, how are you loading grape in rails?

nothing fancy, i have grape in the gemfile and have this in my route:
mount Api::V2::Base => '/'

Any idea why grape is not able to pick up the custom validator?

grape doesn't have an autoloading mechanism for custom validators (PR are welcomed).
In your case you can use [require-dependency] (http://guides.rubyonrails.org/autoloading_and_reloading_constants.html#require-dependency)
# app/controllers/api/v2/base.rb
require_dependency 'api/v2/validators/uuid'

But I should explain why this happens.
First, how does grape load buit-in validators?

Validator files are explicitly required in grape.rb
After the code was loaded and processed, inherited callback is invoked and starts the "registration" process.
All registered validators are saved in Grape::Validations.validators hash, where the keys are validator short names (ex. :uiid). When params is processed and the short name is not found in this hash then an UnknownValidator exception is raised.

In short, grape validators are "auto-registered" only if the validator file is explicity loaded.
Second, Rails autoloading relies on #const_missing to load constants. Obviously, the constant will be autoloaded only if we have its reference in the code.
In your case:
Api::V2::Base const is specified in routes.rb => loads base.rb
V2::Jobs in base.rb => loads jobs.rb
Api::V2::Validators::Uuid is not specified in jobs.rb, only the short name and this is why rails doesn't load your custom validator.
P.S. as you can see from the explanation some :scream: things can be done:
# app/controllers/api/v2/base.rb

Api::V2::Validators::Uuid # instead of require_dependency 'api/v2/validators/uuid'

module Api
  module V2

Thanks for the great explanation, makes perfect sense now. I'm guessing we could improve things on grape by looking up for the constant when we hit the :uuid value in validator parameters?

@gottfrois this sounds reasonable, but I think we should register all available validators(bullt-in/custom) before a hitting anything(validators_paths helper would be helpfull to register a bunch of validators from different directories).
Also I'm personally don't like the implementation using inherited callback(the validator code must be eager loaded for this).
ok, the 3th solution :)
    class Base < Grape::API
      Grape::Validations.register_validator("uuid", Api::V2::Validators::Uuid) # by design it will be called twice
    end

@dblock any thoughts?)

Since grape params are declarative, we could simply add a constant lookup strategy in place that will be trigger when the ruby file is loaded. Am I missing anything? We don't have to look for the constant when someone hits the endpoint.

Also I'm personally don't like the implementation using inherited callback(the validator code must be eager loaded for this).

Yeah sounds like an easy solution. Ideally you would look/require only used constants.

Since grape params are declarative, we could simply add a constant lookup strategy in place that will be trigger when the ruby file is loaded. Am I missing anything?

You should be able to resolve the constant nesting in that place (from :uuid to Api::V2::Validators::Uuid).

@gottfrois take a look on the referenced issue, I've implemented some part

What is the status of this? Has there been any progress on this validator issue (specifically with :uuid) in the last 2 years?

@kevinelliott Doesn't look like it, feel free to PR.

😢 Run into this bug(?) using grape in rails.
Still not fixed.

I have an issue related to this. I had one private api where I loaded all my api validators at the root of the grape api file:
Dir[File.dirname(__FILE__) + '/validators/*.rb'].each {|file| require file }

Now I have to create a second public api on the same rails app. I did the same to load all my public api validators on the root of the public api file.
The issue is that some validators have the same name and conflicting, for example category_id_validation below will raise either a MyApi::V1::Exceptions or a MyApiPublic::V1::Exceptions
ex:
class CategoryIdValidation < Grape::Validations::Base
  include MyApi::V1::Helpers::Authentication

  def validate(request)
    if request.params[:category_id].present?
      unless current_user(request).categories.map(&:id).include?(request.params[:category_id])
        raise MyApi::V1::Exceptions::ParamsIdNotExist, "category #{request.params[:category_id]} does not exist"
      end
    end
  end
end

It would be good to be able to load manually validators for each API so that we can have scoped validators and don't have name conflicts

It would be good to be able to load manually validators for each API so that we can have scoped validators and don't have name conflicts.

agreed

Has anyone figured out how to use this with Rails 6/zeitwerk?


KeyboardAwareScrollView stopping event from propagating on android only.

I have a minor problem with not being able to focus a TextInput or a button, when a TextInput is already focused. I believe the view is taking the event meant for the other components and stopping propagation.
Behaviour on iOS:
Click TextInput -> keyboard shows -> Click another TextInput -> keyboard stays and the other Text input is now focused.
This is the behaviour i want, but on Android it is:
Click TextInput -> keyboard shows -> Click another TextInput -> keyboard hides TextInput loses focus -> click another TextInput again -> keyboard shows and the TextInput is focused.
Does anyone have a fix for this, or is this just the way it is on android?

I think you should play with the ScrollView props, this is due to either softInputMode or http://facebook.github.io/react-native/releases/next/docs/scrollview.html#keyboardshouldpersisttaps

Thanks! I fixed it with the keyboardShouldPersistTaps='handled'


Localization CurrentExpenses.js issue#500

This is a partial implementation.  Please approve and merge if what is there looks good and then I will do the rest of it.

@ethanbb : Were these snippets always nested?

I'm not sure. Looks like the only snippet change for expenses was header  to title though?

@ethanbb : Yeah, I was just wondering about what this looked like in the past. I thought you were the one who went over the snippet PRs. My bad.
I think I'll go ahead and take care of any conflicts that come up with the extra changes and we'll just go ahead and merge this for now.


Notebook: cant seem to get focus into output

Testing #98985
Notice the Cmd+Shift+Down seems to just put focus into the entire Notebook's webview, and once focus is there Tabbing no longer does anything at all. (Also ref #99186 for tabbing inserting a literal tab even though "Tab moves focus" is enabled)

Try the ctrl+cmd+down again in tomorrow's build. If it still doesn't work, I might have to debug on your computer or something.

Now working.


V4 - Responsive Cards

The new Bootstrap 4 Cards are nice, but regarding the variants, like Decks and Columns, i see that they're completely non-responsive at the moment. Am i wrong?
I think many users would use them in some kind of responsive layout.

/CC @mdo I wanted to ping you about this yesterday too. One can see this by visiting the docs card page with a mobile phone.

I'm trying to achieve the result in Decks being responsive. The main problem is that if you want equal heights, you have to apply 'card' styles on the flexbox item, then inside of it build the card. Applying the style to the box causes a collapse between elements due to the use of padding for spacing (codepen).
If you take a look at my website, i achieved something like that using a flex container and boxes styled as cards themselves. Inside of the box there's the wrapper for the card. For spacing i had to play with margins instead of paddings as Boostrap does by default.

Work started in 0d0edeb.


azqueue GetAccessPolicy returns 404

Bug Report

github.com/Azure/azure-sdk-for-go/sdk/storage/azqueue v0.1.0
go version go1.19.1 darwin/arm64

I build the queue URL using:
	queueURL := runtime.JoinPaths("https://"+*accName+".queue.core.windows.net", url.PathEscape(*queueName))

After creating the QueueClient, client.GetProperties() seems to work, but client.GetAccessPolicy() fails with:

GET https://the_account_name.queue.core.windows.net/the_queue_name\n--------------------------------------------------------------------------------\nRESPONSE 404: 404 The specified resource does not exist.\nERROR CODE: ResourceNotFound\n--------------------------------------------------------------------------------\n\ufeffResourceNotFoundThe specified resource does not exist.\nRequestId:a62c90e9-e003-0090-2393-779299000000\nTime:2023-04-25T16:32:34.0640516Z\n--------------------------------------------------------------------------------\n

I was expecting to get the access policy assigned to the queue. Documentation for the method is here.

Hi @disq I will attempt to repro this and get back to you

Hi @disq I will attempt to repro this and get back to you

@tasherif-msft Any luck? Let me know if you need any more info

@tasherif-msft Yes it's happening consistently with multiple queues but the az cli (az storage queue policy list --connection-string ... --queue-name ...) works... I tried overriding the API version to 2021-02-12 (which is what the CLI uses... azqueue seems to use 2018-03-28) with a custom transporter but it's still the same error.

@disq can you share which method you're using to create your queue client and how you're passing your credential? if you can share your full code snippet and just redact any sensitive information, that would help repro your issue.

@tasherif-msft The code is publicly here: https://github.com/cloudquery/cloudquery/pull/10323/files#diff-c9bb3a1a217756575eb01adad5eb0b927b27fa55022bdf66c5e46a7ed3a8751aR45-R55
I also tried generating a ServiceClient and then creating the QueueClient from that but it leads to the same http request with the same result (404) in the end.

@tasherif-msft and here's a bare-bones main.go gist that shows the issue: https://gist.github.com/disq/d8de7925b700660a397dd652bd6771b9
go run main.go
2023/05/02 10:39:46 Found subscription xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
2023/05/02 10:39:47 Found storage account cqdestinationazblob
2023/05/02 10:39:48 Found queue anothertestqueue
2023/05/02 10:39:49 Error:GET https://cqdestinationazblob.queue.core.windows.net/anothertestqueue
--------------------------------------------------------------------------------
RESPONSE 404: 404 The specified resource does not exist.
ERROR CODE: ResourceNotFound
--------------------------------------------------------------------------------
<?xml version="1.0" encoding="utf-8"?><Error>ResourceNotFound<Message>The specified resource does not exist.
RequestId:6954655e-d003-0040-77da-7cafcf000000
Time:2023-05-02T09:39:49.0573352Z</Message></Error>
--------------------------------------------------------------------------------
exit status 1

Hi @disq so I was looking at your request ID, and it seems it's a permissions issue. Although queues does support OAuth, please refer to https://learn.microsoft.com/en-us/rest/api/storageservices/get-queue-acl#authorization "Only the account owner may call this operation."
So the service principal being used does not have the proper permissions here

it's a permissions issue

@tasherif-msft az storage queue policy list works for me using the same credentials/account, so could it be something else?


App error Invalid type for path "table.primary_key". Expected array, but got string should be user error

https://keboola.slack.com/archives/docker-notifications/p1450371500000024

Fixed in @a77995f27e474ade6fb8357c84ba9c91cd18b27b


Adjust indentation and remove not valid comma

Hi @gregorwolf, thanks for your contribution! We'll take a look and will get back to you.
@MHoltbruegge

cc @melaniedisson

Thank you @gregorwolf for your contribution. The change is available with the next documentation release on SAP Help Portal.

My SAP Community User ID is 12545.


PLATUI-3321: update to scala 3 and play 3

I tried this out locally by creating a repo from the template, adding the extra bits needed by sbt-auto-build, and running all scalafmt, Test/scalafmt and scalafmtSbt to make sure nothing changed in formatting rules going to scala 3 all worked as expected
The changes to return types is needed because returning a path-dependent type from within a trait doesn't work in scala 3

I don't think I need to bump https://github.com/hmrc/api-testing-template-scalatest.g8/blob/main/build.sbt#L4 because we're not building an artefact that's used, I think build-and-deploy will just be grabbing main branch of this repo


[Concurrency] Build C-only libdispatch as part of Swift build

This is a requirement to properly link the _Concurrency library against libdispatch on non-Darwin platforms.

@swift-ci test

@swift-ci test

Build failed
Swift Test Linux Platform
Git Sha - d217ee397b68443bed7b54a1106163408ee68cdb

@swift-ci test

@swift-ci test

@swift-ci test windows

@swift-ci test

@swift-ci test

@swift-ci test

@swift-ci test

Both Androids broke again with these changes: https://ci-external.swift.org/job/oss-swift-RA-linux-ubuntu-16.04-android/ and https://ci-external.swift.org/job/oss-swift-RA-linux-ubuntu-16.04-android-arm64/.
Is it possible to revert and get a notice of when this is sent again so can be checked for correctness?

@drodriguez I have a PR open that hopefully fixes this: https://github.com/apple/swift/pull/35881
Would you mind giving that a try?


Modify nvm install behavior when 'nvm install' fails

Fixes https://github.com/travis-ci/travis-ci/issues/5333
With this change:

Install the latest version known to nvm if nodejs.org is reachable
If nodejs.org is not reachable or 'nvm install' fails for some reason,
use the locally available and applicable version
If there is no version available locally, fail the build immediately

Test: https://staging.travis-ci.org/BanzaiMan/travis_staging_test/builds/492969
Note that this modified /etc/hosts to simulate the situation where nodejs.org is unreachable. This was done for demonstration purposes only. This PR does not do such a thing.


dev branch "yarn build" still building with adityapk00 packages

Describe the bug
Dev branch rust build is still pointing to adityapk00 content
Compiling zcash_primitives v0.5.0 (https://github.com/adityapk00/librustzcash?rev=adeb3ec4ad15480482bc2962bc9fe453814db9ee#adeb3ec4)
   Compiling zcash_proofs v0.5.0 (https://github.com/adityapk00/librustzcash?rev=adeb3ec4ad15480482bc2962bc9fe453814db9ee#adeb3ec4)
Compiling zecwalletlitelib v0.1.0 (https://github.com/adityapk00/zecwallet-light-cli?rev=89be353b5ea600ee21d998254598fb19d8778417#89be353b)
   Compiling zcash_client_backend v0.5.0 (https://github.com/adityapk00/librustzcash?rev=adeb3ec4ad15480482bc2962bc9fe453814db9ee#adeb3ec4)
Compiling equihash v0.1.0 (https://github.com/adityapk00/librustzcash?rev=adeb3ec4ad15480482bc2962bc9fe453814db9ee#adeb3ec4)
To Reproduce
Steps to reproduce the behavior:

run yarn build after yarn install

Expected behavior
pointing at Zingolabs repositories, not adityapk00
Desktop (please complete the following information):

OS: OSX
Browser: Nill
Local compile

This was fixed by #24 - Thank you @Gunnar-Stunnar !


CODE OPTIMISATION: Enhance Code Organisation - TransactionForm Component

Description:
To streamline the code structure and improve maintainability, it is recommended to create a new component, TransactionForm.js, dedicated to handling the addition of transactions to the database. This refactoring effort aims to establish a clear separation of concerns, with a focused component for transaction form-related functionalities.
Steps:

Introduce a new component, TransactionForm.js, within the components folder.
Move the transaction form-related code, including input fields and form logic, from ExpenseTracker.js to TransactionForm.js.
Implement the necessary functionality for adding transactions to the database within TransactionForm.js.
Pass any required state variables or functions from ExpenseTracker.js to TransactionForm.js as props.

Expected Benefits:
Improved code modularity with a dedicated component for handling transaction forms.
Enhanced readability and maintainability of the ExpenseTracker component.
A clear separation of concerns, making it easier to understand and manage code related to transaction form interactions.

I can work on this.

@LakshyaSatpal sure go ahead


Update sources

This did not work locally but let's leave it here so we can discuss alternatives.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

Great finding! 👍


test_linalg_matrix_power fails due to tolerance issues when built against lapack 3.12.0

Reporting a bug

[x] I have tried using the latest released version of Numba (most recent is
visible in the release notes
(https://numba.readthedocs.io/en/stable/release-notes-overview.html).
[ ] I have included a self contained code sample to reproduce the problem.
i.e. it's possible to run as 'python bug.py'.

Description
This came up when trying to update packages for lapack on openSUSE to its latest version 3.12.0. I understand numba depends on Lapack indirectly via numpy, and we find that when running the test-suite:
~> python3.11 -m numba.runtests -v -b --exclude-tags=long_running -m 4 -- numba.tests

test_linalg_matrix_power fails with the following log. Previously we were building against lapack 3.9.0, with otherwise identical versions of packages/libraries,[^versions] and the tests all succeeded.
[ 2481s] FAIL: test_linalg_matrix_power (numba.tests.test_linalg.TestLinalgMatrixPower.test_linalg_matrix_power)
[ 2481s] ----------------------------------------------------------------------
[ 2481s] Traceback (most recent call last):
[ 2481s]   File "/usr/lib64/python3.12/site-packages/numba/tests/test_linalg.py", line 2440, in test_linalg_matrix_power
[ 2481s]     check(a, pwr)
[ 2481s]   File "/usr/lib64/python3.12/site-packages/numba/tests/test_linalg.py", line 2428, in check
[ 2481s]     np.testing.assert_allclose(got, expected, rtol=res, atol=res)
[ 2481s]   File "/usr/lib64/python3.12/site-packages/numpy/testing/_private/utils.py", line 1504, in assert_allclose
[ 2481s]     assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
[ 2481s]   File "/usr/lib64/python3.12/contextlib.py", line 81, in inner
[ 2481s]     return func(*args, **kwds)
[ 2481s]            ^^^^^^^^^^^^^^^^^^^
[ 2481s]   File "/usr/lib64/python3.12/site-packages/numpy/testing/_private/utils.py", line 797, in assert_array_compare
[ 2481s]     raise AssertionError(msg)
[ 2481s] AssertionError:
[ 2481s] Not equal to tolerance rtol=5e-15, atol=5e-15
[ 2481s]
[ 2481s] Mismatched elements: 1 / 25 (4%)
[ 2481s] Max absolute difference: 1.12132525e-14
[ 2481s] Max relative difference: 3.46189033e-14
[ 2481s]  x: array([[-0.644914,  0.433068,  0.244191,  0.299553,  0.497169],
[ 2481s]        [ 0.360293, -0.456952,  0.328252, -0.061924,  0.741485],
[ 2481s]        [ 0.241062, -0.141202,  0.180883,  0.9199  , -0.207403],...
[ 2481s]  y: array([[-0.644914,  0.433068,  0.244191,  0.299553,  0.497169],
[ 2481s]        [ 0.360293, -0.456952,  0.328252, -0.061924,  0.741485],
[ 2481s]        [ 0.241062, -0.141202,  0.180883,  0.9199  , -0.207403],...
[ 2481s]
[ 2481s] Stderr:
[ 2481s] /usr/lib64/python3.12/site-packages/numba/tests/test_linalg.py:399: NumbaPerformanceWarning: np.dot() is faster on contiguous arrays, called on (Array(float64, 2, 'A', False, aligned=True), Array(float64, 2, 'A', False, aligned=True))
[ 2481s]   return np.linalg.matrix_power(a, n)
[ 2481s] /usr/lib64/python3.12/site-packages/numba/tests/test_linalg.py:399: NumbaPerformanceWarning: np.dot() is faster on contiguous arrays, called on (Array(float64, 2, 'C', False, aligned=True), Array(float64, 2, 'A', False, aligned=True))
[ 2481s]   return np.linalg.matrix_power(a, n)

Thanks in advance for any suggestions and/or fixes.
[^versions]: Package versions used:
* GCC 13.2.1
* Lapack 3.12.0
* Numba: 0.59.1
* Numpy 1.26.4
* Python 3.10 / 3.11 / 3.12
* Scipy 1.13.0

Thanks for the report, and apologies for the delay in responding. It looks like there's some minor numerical difference the results being checked in the matrix_power tests, the Numba buildfarm has also picked up on it: https://github.com/numba/numba/pull/9602#issuecomment-2146213246. Perhaps subscribe to that issue for updates as it will be fixed as part of the upcoming Numba 0.60.0 release. Thanks again!


Use the same number of clbits for all subexperiments generated from a given subcircuit

Reasoning about the quasi-dist outputs from various Sampler implementations requires knowing the size of the two classical registers added during cutting -- observable_measurements and qpd_measurements. Making all of the subexperiments generated from a given subcircuit have the same size qpd_measurements creg makes post-processing more straightforward.
This change will be useful in the #363 refactor as well.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

6 of 6 (100.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.006%) to 91.027%

Totals

Change from base Build<PHONE_NUMBER>:
0.006%

Covered Lines:
2658

Relevant Lines:
2920

💛  - Coveralls

This is no longer necessary in our workflow, so I'm closing this PR

Working with Runtime team to pack what we need in Runtime SamplerResult metadata. We won't need this PR when that is provided.