Add Table Partitioning Option for PostgreSQL

Feature
Feature description
Table partitioning has existed since PostgreSQL 9, but in PostgreSQL 11 has been made significantly easier to use and more performant.
Who will this benefit?
Anyone who has solid Medium-Sized™ data that doesn't quite need the complexity or resources provided by BigQuery/RedShift/Snowflake.
For instance, in our current environment we have a table for all email messages sent to every user historically that is 200 million rows. While queryable, it could definitely benefit from using a range partition based on dates or even user groups.

Thanks for the request @sheyd!
I took a look at the docs you've sent over. Table partitioning on pg looks pretty slick but also very involved.
There are a couple of challenges for us to sort out here: in dbt, users don't specify table DDL. Instead, dbt creates tables with the schema described in create table as queries generated from model select statements. It doesn't appear to me that there's a way to specify table partitioning in a create table as statement - do you know if such a thing is possible?
One approach would be to create an empty table from the specified select statement, then create a table like that other table, copying over the schema and adding a partitioning clause. In practice, this might look like:
create table dbt_dbanin.tbl_partitioning_test__dbt_tmp as (

        -- model SELECT statement (with a limit 0)
	select
		1 as id,
		'drew'::text as name,
		'green'::text as favorite_color

	limit 0

);

-- create the partitioned table from the __tmp table definition
create table dbt_dbanin.tbl_partitioning_test
(
	like dbt_dbanin.tbl_partitioning_test__dbt_tmp
)
partition by HASH (favorite_color);

While this might work for creating the partitioned table, it's really only a tiny part of the overall implementation. The docs here make it look to me like dbt would need to create one table per partition, specifying the range of dates in that partition. Further, there's a bunch of examples of custom functions like
CREATE RULE measurement_insert_y2006m02 AS
ON INSERT TO measurement WHERE
    ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' )
DO INSTEAD
    INSERT INTO measurement_y2006m02 VALUES (NEW.*);

or
CREATE OR REPLACE FUNCTION measurement_insert_trigger()
RETURNS TRIGGER AS $$
BEGIN
    INSERT INTO measurement_y2008m01 VALUES (NEW.*);
    RETURN NULL;
END;
$$
LANGUAGE plpgsql;

There's also a discussion of triggers and this sort of wild line:

We must redefine the trigger function each month so that it always points to the current child table. The trigger definition does not need to be updated, however.

In all, this feels like a pretty cumbersome process, and one that's a little outside the realm of how dbt usually operates!
I am acutely interested in figuring out how dbt can better create date-partitioned datasets on data warehouses (like redshift / snowflake / bigquery), so I'm super happy to spend the time to explore some approaches on postgres too. As it stands though, I don't imagine this is something we'd implement in dbt-core. Maybe it's a good use for a Custom Materialization?
Let me know if I'm overlooking or overcomplicating anything, and thanks again for the suggestion!

Closing this as I don't think it's something we're going to prioritize in the next 6 months. Happy to re-open if anyone feels strongly!

@joeyfezster Thanks for the bump! Do you have a sense of whether the implementation required for Postgres partitions has become any simpler in the years since this issue was originally opened? A lot of the original concerns still seem relevant.
In the meantime, there's nothing blocking from someone implementing this as a custom materialization in user-space code (i.e., no fork of dbt necessary). I'd be interested to see the complexity required, before deciding whether this should live in the dbt-postgres adapter plugin and be available out of the box.

Hi,
I would also value this functionality. I think postgres' ability to function effectively as a data warehouse depends on partitioning.
It seems that the simplest way to deal with partitions is to use the pg_partman extension. AWS have a good tutorial on it. https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/PostgreSQL_Partitions.html
So, when creating the table, dbt would create the table as normal, but with a PARTITION BY clause. Then it could call partman.create_parent:
 p_control => 'created_at',
 p_type => 'native',
 p_interval=> 'daily',
 p_premake => 30);```

Also, during the dbt run, before inserting any data into the table, it should run partman.run_maintenance_proc, so partman can make sure all the required partitions are created.

Would you please consider reopening this issue?

Kind regards

Also - would it work if I created partitioned tables manually? Would dbt just use them as if they're normal tables?

Hi everyone,
I understand the difficulties mentioned by @drewbanin. And I agree with everyone else that this would be a very valuable feature for companies that rely on Postgres as an open source solution. I would appreciate it greatly if this could be supported by dbt-core.
@jtcohen6 Thank you for the following comment:

In the meantime, there's nothing blocking from someone implementing this as a custom materialization in user-space code
Could you be so kind to point me in the right direction on how to do this? In the dbt docs I did not find anything regarding custom materializations.

Thank you

Also - would it work if I created partitioned tables manually? Would dbt just use them as if they're normal tables?

I'm using AlloyDB with list partitioned tables and yes DBT can use partitioned tables on Postgres that are created prior to running DBT.  We use incremental loading. You can opt for incremental and load just new rows, or if you want to replace entire partitions or data you can choose incremental and set the unique identifier to be the key you are partitioning on.
The default DBT patterns are not very optimal for replacing entire partitions of data.  (large scale deletes and inserts can be slow) To combat this you may need to make adjustments to the incremental materialization to enable better partition pruning.  particularly when using a composite key for your unique key.
The default incremental materialization out of the box will probably work okay for most simple use cases but you're likely to run into a few performance issues if your workload is anything but simple.
I've been tinkering with a few components/ideas that could eventually go into a new materialisation to enable hot-swapping of entire partitions for Postgres but don't really have time to build out the materialisation. Happy to share my thoughts if anyone is looking at tackling this,

Thanks for the request @sheyd!
I took a look at the docs you've sent over. Table partitioning on pg looks pretty slick but also very involved.
There are a couple of challenges for us to sort out here: in dbt, users don't specify table DDL. Instead, dbt creates tables with the schema described in create table as queries generated from model select statements. It doesn't appear to me that there's a way to specify table partitioning in a create table as statement - do you know if such a thing is possible?
One approach would be to create an empty table from the specified select statement, then create a table like that other table, copying over the schema and adding a partitioning clause. In practice, this might look like:
create table dbt_dbanin.tbl_partitioning_test__dbt_tmp as (

        -- model SELECT statement (with a limit 0)
	select
		1 as id,
		'drew'::text as name,
		'green'::text as favorite_color

	limit 0

);

-- create the partitioned table from the __tmp table definition
create table dbt_dbanin.tbl_partitioning_test
(
	like dbt_dbanin.tbl_partitioning_test__dbt_tmp
)
partition by HASH (favorite_color);

While this might work for creating the partitioned table, it's really only a tiny part of the overall implementation. The docs here make it look to me like dbt would need to create one table per partition, specifying the range of dates in that partition. Further, there's a bunch of examples of custom functions like
CREATE RULE measurement_insert_y2006m02 AS
ON INSERT TO measurement WHERE
    ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' )
DO INSTEAD
    INSERT INTO measurement_y2006m02 VALUES (NEW.*);

or
CREATE OR REPLACE FUNCTION measurement_insert_trigger()
RETURNS TRIGGER AS $$
BEGIN
    INSERT INTO measurement_y2008m01 VALUES (NEW.*);
    RETURN NULL;
END;
$$
LANGUAGE plpgsql;

There's also a discussion of triggers and this sort of wild line:

We must redefine the trigger function each month so that it always points to the current child table. The trigger definition does not need to be updated, however.

In all, this feels like a pretty cumbersome process, and one that's a little outside the realm of how dbt usually operates!
I am acutely interested in figuring out how dbt can better create date-partitioned datasets on data warehouses (like redshift / snowflake / bigquery), so I'm super happy to spend the time to explore some approaches on postgres too. As it stands though, I don't imagine this is something we'd implement in dbt-core. Maybe it's a good use for a Custom Materialization?
Let me know if I'm overlooking or overcomplicating anything, and thanks again for the suggestion!

Thanks for the request @sheyd!
I took a look at the docs you've sent over. Table partitioning on pg looks pretty slick but also very involved.
There are a couple of challenges for us to sort out here: in dbt, users don't specify table DDL. Instead, dbt creates tables with the schema described in create table as queries generated from model select statements. It doesn't appear to me that there's a way to specify table partitioning in a create table as statement - do you know if such a thing is possible?
One approach would be to create an empty table from the specified select statement, then create a table like that other table, copying over the schema and adding a partitioning clause. In practice, this might look like:
create table dbt_dbanin.tbl_partitioning_test__dbt_tmp as (

        -- model SELECT statement (with a limit 0)
	select
		1 as id,
		'drew'::text as name,
		'green'::text as favorite_color

	limit 0

);

-- create the partitioned table from the __tmp table definition
create table dbt_dbanin.tbl_partitioning_test
(
	like dbt_dbanin.tbl_partitioning_test__dbt_tmp
)
partition by HASH (favorite_color);

While this might work for creating the partitioned table, it's really only a tiny part of the overall implementation. The docs here make it look to me like dbt would need to create one table per partition, specifying the range of dates in that partition. Further, there's a bunch of examples of custom functions like
CREATE RULE measurement_insert_y2006m02 AS
ON INSERT TO measurement WHERE
    ( logdate >= DATE '2006-02-01' AND logdate < DATE '2006-03-01' )
DO INSTEAD
    INSERT INTO measurement_y2006m02 VALUES (NEW.*);

or
CREATE OR REPLACE FUNCTION measurement_insert_trigger()
RETURNS TRIGGER AS $$
BEGIN
    INSERT INTO measurement_y2008m01 VALUES (NEW.*);
    RETURN NULL;
END;
$$
LANGUAGE plpgsql;

There's also a discussion of triggers and this sort of wild line:

We must redefine the trigger function each month so that it always points to the current child table. The trigger definition does not need to be updated, however.

In all, this feels like a pretty cumbersome process, and one that's a little outside the realm of how dbt usually operates!
I am acutely interested in figuring out how dbt can better create date-partitioned datasets on data warehouses (like redshift / snowflake / bigquery), so I'm super happy to spend the time to explore some approaches on postgres too. As it stands though, I don't imagine this is something we'd implement in dbt-core. Maybe it's a good use for a Custom Materialization?
Let me know if I'm overlooking or overcomplicating anything, and thanks again for the suggestion!

The way I was looking at approaching this is to have dbt create the table as it normally would. however, this would just be a temporary table.  The operation can be pretty fast as long as you add in a 1=2 kind of filter in the outer select so that the product is an empty table.  Once this is created you can query the system catalogue and use the materialisation metadata to create a new table appending the partitioned by clause with the field you want. (assume partition by list)
I was then thinking about hot-swapping partitions, which involved creating a new partition with a dummy partition value (can't create two partitions with the same partition value ) You can then detach the partition and then load the table as normal. once loaded, you would need to detach the previously active partition and attach the new partition.
This is a simplification of the details required to implement this as you would also normally want the materialization to be safe in the event of a failure/disconnect which would involve a decent amount of thought around how to run in a transaction or if that isn't feasible with the mix of operations then ensuring that the job is recoverable. Its been a few months since i really looked a this but i think i was also coming across the issue of having to ensure constraint names/indexes were not duplicated.

I'm interested in this feature too, but I don't have the knowledge to implement it myself. I think there are two usecases here:
1- Incremental by hot swapping partitions. Example: partition by month. Dbt creates new partitions and swaps the affected ones. This would be the most sophisticated case as described by @IS-Josh.
2- Incremental by delete+write as it's now, just with underlying partitioned tables.
The second case should be way simpler to implement, at the expense of more end-user care. What does the simple case bring to the table? Basically, I could use a different storage (Citus Columnar) on non-changing partitions. Also, partitioning by date should reduce the need for some indexes.

Just in case it helps anyone, I have a very rough implementation of partitioning by overriding a Postgres macro:
{% macro postgres__create_table_as(temporary, relation, sql) -%}
  {%- set unlogged = config.get('unlogged', default=false) -%}
  {%- set sql_header = config.get('sql_header', none) -%}

  {{ sql_header if sql_header is not none }}

  {% if config.get('partition_by') != None %}
    {# Use partitioning #}

    {%- set partition_by_field = config.get('partition_by')['field'] -%}
    {%- set partition_by_granularity = config.get('partition_by')['granularity'] -%}

    -- We cannot create a partitioned table with "create as select".
    -- Create a dummy temporary table just to be used as a template for the real one
    create temporary table "{{ this.identifier }}__tt" as
    select *
    from ({{ sql }}) model_subq
    where 1 = 2;

    -- Create partitioned table as a copy of the template
    create {% if temporary -%}
        temporary
      {%- elif unlogged -%}
        unlogged
      {%- endif %} table {{ relation }} (like "{{ this.identifier }}__tt")
      partition by range ({{ partition_by_field }});

    {% set queryRequiredPartitions %}
      -- We need to create the partitions before the parent table is able to handle inserts
      with partitions as (
          -- Get the begin and end dates by granularity.
          -- Generate a suffix to append to every partition. This is to avoid collisions as the children tables are not renamed at the end of the process
          -- Old children will just be deleted when dbt drops its parent
          select
              generate_series(
                  date_trunc('{{ partition_by_granularity }}', min({{ partition_by_field }})),
                  date_trunc('{{ partition_by_granularity }}', max({{ partition_by_field }})),
                  '1 {{ partition_by_granularity }}'::interval
              ) as begin_date,
              (select floor(random() * 100 + 1)::int) as rand -- Random number to avoid name colissions
          from ({{ sql }}) model_subq
      )
      select
          -- Generate the final data
          begin_date,
          begin_date + '1 {{ partition_by_granularity }}'::interval as end_date,
          to_char(begin_date, 'yyyymmdd') || '_' || rand as partition_suffix
      from partitions;
    {% endset %}
    {% set requiredPartitions = run_query(queryRequiredPartitions) %}

    -- Create the required partitions
    {% for requiredPartition in requiredPartitions.rows %}
      create table {{ make_intermediate_relation(relation, requiredPartition['partition_suffix']) }}
      partition of {{ relation }} for values from ('{{ requiredPartition["begin_date"] }}'::timestamp) to ('{{ requiredPartition["end_date"] }}'::timestamp);
    {% endfor %}

    -- Insert into the parent table
    insert into {{ relation }}
    {{ sql }};
  {% else %}
    {# Standard procedure, copied from: https://github.com/dbt-labs/dbt-postgres/blob/cf100560e7712dcc387e286057ffeeccb23a944e/dbt/include/postgres/macros/adapters.sql#L1 #}
    {# TODO: delegate to the standard implementation #}
     create {% if temporary -%}
        temporary
      {%- elif unlogged -%}
        unlogged
      {%- endif %} table {{ relation }}
      {% set contract_config = config.get('contract') %}
      {% if contract_config.enforced %}
        {{ get_assert_columns_equivalent(sql) }}
      {% endif -%}
      {% if contract_config.enforced and (not temporary) -%}
          {{ get_table_columns_and_constraints() }} ;
        insert into {{ relation }} (
          {{ adapter.dispatch('get_column_names', 'dbt')() }}
        )
        {%- set sql = get_select_subquery(sql) %}
      {% else %}
        as
      {% endif %}
      (
        {{ sql }}
      );
  {% endif %}
{%- endmacro %}

{% macro postgres__rename_relation(from_relation, to_relation) %}
  {% set target_name = adapter.quote_as_configured(to_relation.identifier, 'identifier') %}
  {% call statement('rename_relation') -%}
    alter table {{ from_relation }} rename to {{ target_name }};

    {# TODO: check if this is a partitioned table and rename its children #}
  {%- endcall %}
{% endmacro %}

To use it:
{{
config(
    materialized = 'table',
    partition_by={
      "field": "created_at",
      "granularity": "month"
    }
) }}

select generate_series(
        current_date - interval '1000 day',
        current_date, '1 day'::interval)::date as created_at, 'hello' as dummy_text

I plan to create a pull request for this to the dbt-postgres repo once I fully test it. Next steps would be to create an incremental materialization that drops and recreates whole partitions instead of merging.

For completeness, a very rough implementation of a custom incremental partitioning strategy:
The strategy macro:
{% macro get_incremental_partition_sql(arg_dict) %}

    {%- set target = arg_dict["target_relation"] -%}
    {%- set source = arg_dict["temp_relation"] -%}
    {%- set dest_columns = arg_dict["dest_columns"] -%}
    {%- set incremental_predicates = arg_dict["incremental_predicates"] -%}

    {% if incremental_predicates is none %}
      {% do exceptions.raise_compiler_error('Config "incremental_predicates" is required!') %}
    {% endif %}

    -- Get the to-be-dropped partitions
    {% set queryToDropPartitions %}
        select distinct ns.nspname as schema_name, c.relname as partition_name
        from {{ target }} lt
        left join pg_class c on lt.tableoid = c.oid
        left join pg_namespace ns on c.relnamespace = ns.oid
        where 1=1
        {% for predicate in incremental_predicates %}
            and {{ predicate }}
        {% endfor %}
    {% endset %}

    {% set toDropPartitions = run_query(queryToDropPartitions) %}
    {% for toDropPartition in toDropPartitions.rows %}
      drop table "{{ toDropPartition.schema_name }}"."{{ toDropPartition.partition_name }}";
    {% endfor %}

    {%- set partition_by_field = config.get('partition_by')['field'] -%}
    {%- set partition_by_granularity = config.get('partition_by')['granularity'] -%}

    -- Get the partitions that need to be created
    {% set queryRequiredPartitions %}
      -- We need to create the partitions before the parent table is able to handle inserts
      with partitions as (
          -- Get the begin and end dates by granularity.
          -- Generate a suffix to append to every partition. This is to avoid collisions as the children tables are not renamed at the end of the process
          -- Old children will just be deleted when dbt drops its parent
          select
              generate_series(
                  date_trunc('{{ partition_by_granularity }}', min({{ partition_by_field }})),
                  date_trunc('{{ partition_by_granularity }}', max({{ partition_by_field }})),
                  '1 {{ partition_by_granularity }}'::interval
              ) as begin_date,
              (select floor(random() * 100 + 1)::int) as rand -- Random number to avoid name colissions
          from {{ source }} model_subq
      )
      select
          -- Generate the final data
          begin_date,
          begin_date + '1 {{ partition_by_granularity }}'::interval as end_date,
          to_char(begin_date, 'yyyymmdd') || '_' || rand as partition_suffix
      from partitions;
    {% endset %}
    {% set requiredPartitions = run_query(queryRequiredPartitions) %}

    -- Create the required partitions
    {% for requiredPartition in requiredPartitions.rows %}
      create
      {% if temporary -%}
        temporary
      {%- elif unlogged -%}
        unlogged
      {%- endif %}
      table
      "{{ schema }}".{{ make_intermediate_relation(source, requiredPartition['partition_suffix']) }}
      partition of {{ target }} for values from ('{{ requiredPartition["begin_date"] }}'::timestamp) to ('{{ requiredPartition["end_date"] }}'::timestamp);
    {% endfor %}

    {%- set dest_cols_csv = get_quoted_csv(dest_columns | map(attribute="name")) -%}
    -- Move the data to the partitioned table
    insert into {{ target }} ({{ dest_cols_csv }})
    (
        select {{ dest_cols_csv }}
        from {{ source }}
    )

{% endmacro %}

The model:
{{ config(
    materialized = 'incremental',
    partition_by = {
      "field": "created_at",
      "granularity": "month"
    },
    incremental_strategy = 'partition',
    incremental_predicates = ["created_at >= date_trunc('month', current_date - 90)"]
)
}}

with raw_data as (
  select generate_series(
          current_date - interval '2000 day',
          current_date, '1 day'::interval)::date as created_at, 'hello' as dummy_text
)
select *
from raw_data
{% if is_incremental() %}
    where created_at >= date_trunc('month', current_date - 90)
{% endif %}

Opened a PR on https://github.com/dbt-labs/dbt-postgres/pull/78. If you are interested in this feature, please show your support and feedback!


Finetune MPT models with  local dataset

Hello Team,
Can you please guide on how to finetune on local datasets, the instructions given in scripts/train are not so clear. Below yaml file was given as sample example:
train_loader:
name: finetuning
dataset:
hf_name: my-local-dataset
hf_kwargs:
data_files:
train: /path/to/train.jsonl
preprocessing_fn: my.import.path:my_preprocessing_fn
split: train
So do we need to create a loading_script to load local dataset using huggingface datasets or is there a way we can directly use jsonl file paths instead of converting it into huggingface dataset & for that what changes I need to make in yaml file

Sorry for the lack of clarity here, we'll update the docs.
Going off of @arpitkk 's example, you would want to use train_loader.dataset.hf_name: json and keep the rest the same.
Just to clarify, that config will eventually influence the behavior of this dataset-building method here:https://github.com/mosaicml/llm-foundry/blob/6c16a6e9b31abe179dda7d71e9601f866721d6fa/llmfoundry/data/finetuning/tasks.py#L223
So, by setting your config to
train_loader:
   name: finetuning
   dataset:
      hf_name: json
      hf_kwargs:
         data_files:
            train: /path/to/train.jsonl
      preprocessing_fn: my.import.path:my_preprocessing_fn
      split: train

The build_from_hf method will effectively execute the following code:
import datasets
from my.import.path import my_preprocessing_fn

dataset = datasets.load_dataset('json', split='train',  data_files={'train': '/path/to/train.jsonl'})

def dataset_mapper(example: Dict):
    example = my_preprocessing_fn(example)
    return _tokenize_formatted_example(example, tokenizer)

columns_to_remove = list(dataset[0].keys())
tokenized_dataset = dataset.map(
    dataset_mapper,
    batched=False,
    remove_columns=columns_to_remove,
)

Does that help to clear things up? If so, I can work a similar explanation into the README. If not, please let me know what still feels unclear!

There might be an issue then. Here is the config I am using:
train_loader:
  name: finetuning

  dataset:
    hf_name: json
    hf_kwargs:
       data_files:
          train: /mnt/training/mylocaldataset/train.jsonl
    preprocessing_fn: mylocaldataset.utils:prep_fn
    split: train
    max_seq_len: ${max_seq_len}
    allow_pad_trimming: false
    decoder_only_format: true
    shuffle: true

I am running with: composer llm-foundry/scripts/train/train.py   mpt.yml   save_folder=mpt-tuned at the path /mnt/training.
wc -l /mnt/training/mylocaldataset/train.jsonl: 1212821
I will debug this today and let you know if I make some progress :)

Issue found!
It seems there are a couple of bugs in the HuggingFace library. First, due to a regex problem the system is mixing the train.jsonl and the split train.
The detailed should not be named train.jsonl. Name it for instance prompts.jsonl.
Also, I now use data_dir instead and it works fine this way.
dataset:
    hf_name: json
    hf_kwargs:
      keep_in_memory: true
      data_dir: /mnt/training/mylocaldataset
    preprocessing_fn: mylocaldataset.utils:prep_fn

@baptistejamin, do you have to specify the splits? so data_dir is only the dir? how would it find prompt.jsonl then?

i do not know why, but i keep getting error "FileNotFoundError: Unable to find '/workspace/scripts/train/train' at /workspace/scripts/train"
at dataset = datasets.load_dataset(dataset_name, split=split, **kwargs)
my yaml is # Dataloaders train_loader:   name: finetuning   dataset:     hf_name: json     hf_kwargs:       data_files:         train: data/train.json     preprocessing_fn: preprocess_investopedia:preprocess_investopedia
the data folder is inside the current working dir.

Yes, keep the spit. I strongly recommend using data_dir rather than  data_files. Keep the same config, but replace data_files with data_dir

Thanks for helping to surface these issues!!
@baptistejamin @arpitkk Did the explanation I posted above provide a useful intuition for how to set up the YAML? I want to make sure our README instructions are clear. I'll use your feedback to update them.
I'll also aim to include some of the gotchas that you have caught, e.g., data_dir vs data_files, relative pathing.

You should make a receive that is as easy as possible. Something that can be repeated by newbies. For instance a config with a specific type of GPU with the right batch sized, the right preprocessing function and an example of jsonl datasetFor instance I am familiar with deepspeed, and already fine tuned a dozen of models with it, but it took me some time to fine tune with llm foundrySent from my iPhoneOn 11 May 2023, at 18:57, Alex Trott @.***> wrote:
Thanks for helping to surface these issues!!
@baptistejamin @arpitkk Did the explanation I posted above provide a useful intuition for how to set up the YAML? I want to make sure our README instructions are clear. I'll use your feedback to update them.
I'll also aim to include some of the gotchas that you have caught, e.g., data_dir vs data_files, relative pathing.
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

Thanks for helping to surface these issues!!
@baptistejamin @arpitkk Did the explanation I posted above provide a useful intuition for how to set up the YAML? I want to make sure our README instructions are clear. I'll use your feedback to update them.
I'll also aim to include some of the gotchas that you have caught, e.g., data_dir vs data_files, relative pathing.

@alextrott16  Yes with the approach you mentioned now I am able to finetune local dataset. I have made the below changes in yaml file
dataset: &hf_dataset
hf_name: json
hf_kwargs:
data_files:
train: /home/MPT-7B/mpt_dataset/mpt-train.jsonl
test: /home/MPT-7B/mpt_dataset/mpt-test.jsonl
Dataloaders
train_loader: &train_loader
name: finetuning
dataset:
<<: *hf_dataset
split: train
max_seq_len: ${max_seq_len}
allow_pad_trimming: false
decoder_only_format: true
shuffle: true
# Use python llmfoundry/data/packing.py --yaml-path /path/to/this/yaml/ ... to profile
# this run's optimal packing_ratio
# packing_ratio:
drop_last: true
num_workers: 8
pin_memory: false
prefetch_factor: 2
persistent_workers: true
timeout: 0
eval_loader:
<<: *train_loader
dataset:
<<: *hf_dataset
split: test
max_seq_len: ${max_seq_len}
allow_pad_trimming: false
decoder_only_format: true
shuffle: false
And also made changes in tasks.py build_from_hf function to load the json files.
So when you are updating the readme please update both the yaml & build_from_hf so people has less confusion

Hi, After running few batches the code is failing with the below errors:
IndexError: Caught IndexError in DataLoader worker process 6.
Original Traceback (most recent call last):
File "/home/anaconda3/envs/mpt-train/lib/python3.10/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
data = fetcher.fetch(index)
File "/home/anaconda3/envs/mpt-train/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 61, in fetch
return self.collate_fn(data)
File "/home/llm-foundry/llmfoundry/data/finetuning/collator.py", line 116, in call
batch = self._process_and_batch_decoder_only(examples)
File "/home/llm-foundry/llmfoundry/data/finetuning/collator.py", line 222, in _process_and_batch_decoder_only
batch = self.tokenizer.pad(
File "/home/anaconda3/envs/mpt-train/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2949, in pad
if isinstance(encoded_inputs, (list, tuple)) and isinstance(encoded_inputs[0], Mapping):
IndexError: list index out of range
I am using the below yaml file
max_seq_len: 2048
global_seed: 17
Run Name
run_name: # If left blank, will be read from env var $COMPOSER_RUN_NAME
Model
These must match pretraining
model:
name: hf_causal_lm
device: cuda:0
pretrained: true
pretrained_model_name_or_path: mosaicml/mpt-7b
Tokenizer
tokenizer:
name: EleutherAI/gpt-neox-20b
kwargs:
model_max_length: ${max_seq_len}
dataset: &hf_dataset
hf_name: json
hf_kwargs:
data_files:
train: /home/MPT-7B/mpt_dataset/mpt-train.jsonl
test: /home/MPT-7B/mpt_dataset/mpt-test.jsonl
Dataloaders
train_loader: &train_loader
name: finetuning
dataset:
<<: *hf_dataset
split: train
max_seq_len: ${max_seq_len}
allow_pad_trimming: false
decoder_only_format: true
shuffle: true
# Use python llmfoundry/data/packing.py --yaml-path /path/to/this/yaml/ ... to profile
# this run's optimal packing_ratio
# packing_ratio:
drop_last: true
num_workers: 8
pin_memory: false
prefetch_factor: 2
persistent_workers: true
timeout: 0
eval_loader:
<<: *train_loader
dataset:
<<: *hf_dataset
split: test
max_seq_len: ${max_seq_len}
allow_pad_trimming: false
decoder_only_format: true
shuffle: false
Optimization
scheduler:
name: linear_decay_with_warmup # linear no warmup is HF default which dolly used
t_warmup: 0ba
alpha_f: 0
optimizer:
mimic HF defaults to replicate dolly
name: decoupled_adamw
lr: 1.0e-5
betas:

0.9
0.999
eps: 1.0e-8
weight_decay: 0

algorithms:
gradient_clipping:
clipping_type: norm
clipping_threshold: 1.0
max_duration: 1ep
eval_interval: 500ba
eval_first: false
eval_subset_num_batches: -1
global_train_batch_size: 2
System
seed: ${global_seed}
device_eval_batch_size: 1
device_train_microbatch_size: 1
device_train_microbatch_size: auto
precision: amp_bf16
FSDP
fsdp_config:
sharding_strategy: FULL_SHARD
mixed_precision: PURE
activation_checkpointing: false
activation_checkpointing_reentrant: false
activation_cpu_offload: false
limit_all_gathers: true
verbose: false
Logging
progress_bar: false
log_to_console: true
console_log_interval: 1ba
callbacks:
speed_monitor:
window_size: 10
lr_monitor: {}
memory_monitor: {}
runtime_estimator: {}
loggers:
wandb: {}
Checkpoint to local filesystem or remote object store
save_interval: 2000ba
save_num_checkpoints_to_keep: 1  # Important, this cleans up checkpoints saved to DISK
save_folder: ./llm_local_finetune/checkpoints
save_folder: s3://my-bucket/my-folder/{run_name}/checkpoints
Load from remote object store
REPLACE THE BELOW with you own checkpoint!
#load_path: oci://my-bucket/my-folder/mpt-7b/checkpoints/some_checkpoint.pt

I tried fine tuning mpt7b using dolly dataset. Using below command:
composer train.py yamls/finetune/mpt-7b_dolly_sft.yaml
Before strating training i am getting below error:
[Eval batch=321/321] Eval on eval data:
Eval metrics/eval/LanguageCrossEntropy: 9.1594
Eval metrics/eval/LanguagePerplexity: 9503.6523
/home/stsingha/LLM/llm-foundry/llmfoundry-venv/lib/python3.9/site-packages/torch/utils/data/dataloader.py:554: UserWarning: This DataLoader will create 4 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
warnings.warn(_create_warning_msg(
Traceback (most recent call last):
File "", line 21, in _bwd_kernel
KeyError: ('2-.-0-.-0-842f0fbd42a6607893f7134cdd9d16f2-2b0c5161c53c71b37ae20a9996ee4bb8-c1f92808b4e4644c1732e8338187ac87-f24b6aa9b101a518b6a4a6bddded372e-12f7ac1ca211e037f62a7c0c323d9990-5c5e32ff210f3b7f56c98ca29917c25e-06f0df2d61979d629033f4a22eff5198-0dd03b0bd512a184b3512b278d9dfa59-d35ab04ae841e2714a253c523530b071', (torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.bfloat16, torch.float32, torch.bfloat16, torch.bfloat16, torch.float32, torch.float32, 'fp32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32', 'i32'), ('vector', True, 128, False, True, True, True, 128, 128), (True, True, True, True, True, True, True, True, True, True, (False,), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False), (True, False)))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/stsingha/LLM/llm-foundry/scripts/train/train.py", line 254, in
main(cfg)
File "/home/stsingha/LLM/llm-foundry/scripts/train/train.py", line 243, in main
trainer.fit()
File "/home/stsingha/LLM/llm-foundry/llmfoundry-venv/lib/python3.9/site-packages/composer/trainer/trainer.py", line 1766, in fit
self._train_loop()
File "/home/stsingha/LLM/llm-foundry/llmfoundry-venv/lib/python3.9/site-packages/composer/trainer/trainer.py", line 1940, in _train_loop
total_loss_dict = self._train_batch(use_grad_scaling)
File "/home/stsingha/LLM/llm-foundry/llmfoundry-venv/lib/python3.9/site-packages/composer/trainer/trainer.py", line 2115, in _train_batch
optimizer.step(closure=lambda **kwargs: self._train_microbatches(
File "/home/stsingha/LLM/llm-foundry/llmfoundry-venv/lib/python3.9/site-packages/torch/optim/lr_scheduler.py", line 68, in wrapper
return wrapped(*args, **kwargs)
File "/home/stsingha/LLM/llm-foundry/llmfoundry-venv/lib/python3.9/site-packages/torch/optim/optimizer.py", line 140, in wrapper
out = func(*args, **kwargs)
File "/home/stsingha/LLM/llm-foundry/llmfoundry-venv/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/home/stsingha/LLM/llm-foundry/llmfoundry-venv/lib/python3.9/site-packages/composer/optim/decoupled_weight_decay.py", line 288, in step
loss = closure()
File "/home/stsingha/LLM/llm-foundry/llmfoundry-venv/lib/python3.9/site-packages/composer/trainer/trainer.py", line 2115, in
optimizer.step(closure=lambda **kwargs: self._train_microbatches(
File "/home/stsingha/LLM/llm-foundry/llmfoundry-venv/lib/python3.9/site-packages/composer/trainer/trainer.py", line 2213, in _train_microbatches
microbatch_loss_dict = self._train_microbatch(use_grad_scaling, current_batch_size, is_final_microbatch)
File "/home/stsingha/LLM/llm-foundry/llmfoundry-venv/lib/python3.9/site-packages/composer/trainer/trainer.py", line 2340, in _train_microbatch
microbatch_loss.backward(create_graph=self._backwards_create_graph)
File "/home/stsingha/LLM/llm-foundry/llmfoundry-venv/lib/python3.9/site-packages/torch/_tensor.py", line 487, in backward
torch.autograd.backward(
File "/home/stsingha/LLM/llm-foundry/llmfoundry-venv/lib/python3.9/site-packages/torch/autograd/init.py", line 197, in backward
Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
File "/home/stsingha/LLM/llm-foundry/llmfoundry-venv/lib/python3.9/site-packages/torch/autograd/function.py", line 267, in apply
return user_fn(self, *args)
File "/home/stsingha/LLM/llm-foundry/llmfoundry-venv/lib/python3.9/site-packages/flash_attn/flash_attn_triton.py", line 827, in backward
_flash_attn_backward(do, q, k, v, o, lse, dq, dk, dv,
File "/home/stsingha/LLM/llm-foundry/llmfoundry-venv/lib/python3.9/site-packages/flash_attn/flash_attn_triton.py", line 694, in _flash_attn_backward
_bwd_kernel[grid](
File "/home/stsingha/LLM/llm-foundry/llmfoundry-venv/lib/python3.9/site-packages/triton/runtime/jit.py", line 106, in launcher
return self.run(*args, grid=grid, **kwargs)
File "/home/stsingha/LLM/llm-foundry/llmfoundry-venv/lib/python3.9/site-packages/triton/runtime/autotuner.py", line 73, in run
timings = {config: self._bench(*args, config=config, **kwargs)
File "/home/stsingha/LLM/llm-foundry/llmfoundry-venv/lib/python3.9/site-packages/triton/runtime/autotuner.py", line 73, in
timings = {config: self._bench(*args, config=config, **kwargs)
File "/home/stsingha/LLM/llm-foundry/llmfoundry-venv/lib/python3.9/site-packages/triton/runtime/autotuner.py", line 63, in _bench
return do_bench(kernel_call)
File "/home/stsingha/LLM/llm-foundry/llmfoundry-venv/lib/python3.9/site-packages/triton/testing.py", line 140, in do_bench
fn()
File "/home/stsingha/LLM/llm-foundry/llmfoundry-venv/lib/python3.9/site-packages/triton/runtime/autotuner.py", line 62, in kernel_call
self.fn.run(*args, num_warps=config.num_warps, num_stages=config.num_stages, **current)
File "/home/stsingha/LLM/llm-foundry/llmfoundry-venv/lib/python3.9/site-packages/triton/runtime/autotuner.py", line 200, in run
return self.fn.run(*args, **kwargs)
File "", line 43, in _bwd_kernel
RuntimeError: Triton Error [CUDA]: invalid argument
Could you please help in this issue. @arpitkk @baptistejamin @alextrott16

see https://github.com/mosaicml/llm-foundry/issues/143#issuecomment-1553334904

Since this issue was originally made to address the lack of clarity around finetuning from a local dataset, I just want to let folks know that we just pushed a PR that includes a much more concrete example of this workflow.
In the scripts/train directory, you'll find finetune_example, which includes:

a detailed README
an example local training dataset
an implementation of a preprocessing function for that dataset
a YAML which puts it all together and can be run locally via train.py

To help us stay on top of other issues, I'll close this one. If things remain unclear, feel free to add another comment and I'll re-open the issue if necessary. Thank you!

Hi Team, I have identified the issue there was problem with batch size.. now its working fine.. thanks for support !!

@arpitkk can you explain what you changed to get this working? I am running into the same issue.

Hi Team, I have identified the issue there was problem with batch size.. now its working fine.. thanks for support !!

@arpitkk can you explain what you changed to get this working? I am running into the same issue.

I am facing the same issue as well. @arpitkk can you please share as to what you changed to get this working?


WIP - DEVEXP-419: Bootstrapping operator as Removed on BareMetal.

If we detect that the platform we are bootstrapping on is bare metal we
set management state to "Removed". We still report ourselves as "Available"
but with "Reason" set to "Removed".

/test e2e-aws-operator

/test e2e-aws-operator

/test e2e-aws

/assign @adambkaplan @dmage

/retest

/hold cancel
https://github.com/openshift/release/pull/5649 is in the merge pool


If patching a romhack...

If patching a romhack such as Sacred Gold and Storm Silver, should I edit the .nds file before or after patching it with the xdelta patch? Maybe it doesn't matter?

Hey, I'm not sure what would be recommended by the developer. But I
believe I've done it both ways with success. My recommendation would apply
your patch first, and test a couple times it may not work right away.
On Sun, Aug 20, 2023 at 3:49 PM NeonSpaceTurtle @.***>
wrote:

If patching a romhack such as Sacred Gold and Storm Silver, should I edit
the .nds file before or after patching it with the xdelta patch? Maybe it
doesn't matter?
—
Reply to this email directly, view it on GitHub
https://github.com/choogiesaur/hgss-shiny-modifier/issues/3, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AE4EZ7L254TDJRQB2WBZG6LXWKH7VANCNFSM6AAAAAA3XSCF2A
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

Yeah i noticed after this that xdelta would give me troubles patching if it isn't the exact rom with exact checksum unless I bypass, but if I bypass it corrupts it? Regardless of order though my 3DS won't play it :/ Twilight Menu++ just freezes on white screen.

After a good night of sleep and further testing, the game launches fine every step of the way without shiny patch (US rom on EU console>xdelta patched rom), but when I patch the shiny script onto it, the rom won't launch. I've tested both (xdelta patched rom+shiny script>unpatched US rom+shiny script) and it appears the problem is in both cases the shiny script itself.
Could it be because when I installed the script dependency, I got a warning message telling me that the python direction is not added to Windows' PATH? I don't know much about this stuff but I've also installed Stable Diffusion and it wouldn't work until I did a specific edit of a PATH directory so unless it is different, it should already be setup.
Here's some screenshots of the script compilations for both default rom and hackrom:
HeartGold: https://i.imgur.com/jaraZKy.png
SacredGold: https://i.imgur.com/aAqftEv.png
And for good measure here's my environment variables window:
https://i.imgur.com/3VrakCp.png
Hopefully that helps pinpoint the issue.

Update to say I managed to successfully hack the shiny rate manually by loading the rom into DSPRE and editing the arm9.bin hex value manually, then clicking on yes when saving the rom back in DSPRE with the warning about uncompressed arm9.bin file. So I know it is possible to get it working on my 3DS now, the issue is with the python script in itself. It seems it is a similar issue as with the other issue about patching a French rom someone else raised on this github repo.

Hi @NeonSpaceTurtle . So the way the script works, it goes to a predetermined offset where we know the shiny encounter rate is located in clean roms. So if you had patched your rom beforehand with a romhack, if the offset is no longer in the same place (quite likely given that romhacks change a lot of the contents), it wouldn't work.
You would have to figure out where the new shiny encounter offset is in your modded rom which people do with an emu/cheat code debugger or hex editor. See this
Also since this script uses NDSPY and the arm9 recompression does not generate an exact 1-to-1 copy of the original, it makes it unlikely that an xdelta patch could be applied as the resultant rom is different from a clean rom.
Sorry there isn't better news, I may try and look into solutions for this when I have time.

Hi @NeonSpaceTurtle . So the way the script works, it goes to a predetermined offset where we know the shiny encounter rate is located in clean roms. So if you had patched your rom beforehand with a romhack, if the offset is no longer in the same place (quite likely given that romhacks change a lot of the contents), it wouldn't work.
You would have to figure out where the new shiny encounter offset is in your modded rom which people do with an emu/cheat code debugger or hex editor. See this
Also since this script uses NDSPY and the arm9 recompression does not generate an exact 1-to-1 copy of the original, it makes it unlikely that an xdelta patch could be applied as the resultant rom is different from a clean rom.
Sorry there isn't better news, I may try and look into solutions for this when I have time.

That's the thing I have the same issue with a clean rom it doesn't come from the romhack replacing the offset. I was able to patch it manually through different tools as mentioned and it works. I think there is an issue with how arm9 is recompressed into the rom because I used the same offsets when doing it manually.

@NeonSpaceTurtle Oh yeah- the arm9 is actually not recompressed. It's decompressed, modified, then reinserted uncompressed.
This was done because at the time recompression was causing the game to crash randomly. Not sure if this is still the case with NDSPY. Again will have to look into it later. If anyone has ideas though, please comment them here.

I actually just now forked this repository to fix this very issue with a ROM hack, Pokemon Soothing Silver. The issue, from what I can tell, is that the arm9 is not recompressed but is marked as compressed. I fixed it by adding the following code after the shiny rate is changed:
# Mark the arm9 as uncompressed
decompressed_arm9[0xBB4] = 0x00
decompressed_arm9[0xBB5] = 0x00
decompressed_arm9[0xBB6] = 0x00
decompressed_arm9[0xBB7] = 0x00

Thanks @digikitty21 ! Would you mind making a pull request with this and your hex output change? I would like to give you credit :) If not and you don’t mind, I’ll manually add the changes to my repo.

Should be fixed by this pr


Cannot Recognize Speech Error: Error occurred while processing 'audio.wav'. Invalid WAV header in file, RIFF was not found

Hi,
I captured  one voice message from WhatsApp and it saved as wav file using node js. But I pass this wav file for speech translation using JavaScript sdk it not showing anything and also try this file into demo app also, in demo app I got error "Cannot Recognize Speech Error: Error occurred while processing 'audio.wav'. Invalid WAV header in file, RIFF was not found".

This is my audio media information

@nileenas
What platform are you trying this on, and do you have GStreamer installed?
To read Opus and other compressed formats, you need to decode the audio file.
The SDK needs RAW PCM under the covers.  You can either convert the audio ahead of time, or on some system you can install GStreamer and the speech extension will decode the audio for you.
General overview
C# example
Linux example

I am captured one voice message from WhatsApp its voices format is ogg and it is encoded by base64 and written into wav extension file. I am using speech to text JavaScript sdk  with Bot framework sdk with node js.

@nileenas The JavaScript SDK does not support compressed audio, you'll need to convert to WAV in the default format - some details can be found here.

Closed as answered, please open a new issue if more support is needed.


README: add Mikros Image

And thank you for your amazing tool :+1:

Thanks to you @cchampet!

Very cool! 🎉


Missing Catalonia Flag (CT?)

Maybe you can add catalonia flag (CT):
https://commons.wikimedia.org/wiki/File:Flag_of_Catalonia.svg

what's the best sources for the CT?

Hi
@lipis
Catalonia is not a country , it is an autonomy of a country

That's true @Suriv. 👍
According to the project description (a collection of all country flags in SVG), it should not be added.
I opened this issue because I use the SVG flags to represent languages in my web projects.

Then maybe we'll pass on this one as well :)


C2W

Hey Ankit,
I am Siddhant Bapna from Udaipur Rajasthan, I secured AIR-2 in C2W and it's nice to meet you.
Firstly, Congratulation for AIR-1 in C2W.
I want to talk to you, can we have a chat or a call if you are free ?

Yes, add me on Skype.
: https://join.skype.com/invite/v2sMXHl9K5T3


Extra actions after build.

I would like to run some extra fictions to zip files, create DMG e.t.c after the package.js script has built the various binaries. But I can't for the life of me work out how to add an additional .then to the promise chain after the build step. Anyone got any suggestions/guidance/examples they could share?
Thanks!

Electron builder makes this easier:
const builder = require("electron-builder")
const Platform = builder.Platform

// Promise is returned
builder.build({
  targets: Platform.MAC.createTarget(),
  devMetadata: {
    "//": "build and other properties, see https://goo.gl/5jVxoO"
  }
})
  .then(() => {
    // handle result
  })
  .catch((error) => {
    // handle error
  })

Interesting, though I am currently using electron-packager, have you modified the package.js file to use electron-builder instead? Could you share your modifications?
Thanks
Gareth

@gazzer82 you could try changing the package script:
 "package": "cross-env NODE_ENV=production node -r babel-register package.js && some_other_script.js",

Closing, due to no response


Adding settings path and container_opts

A couple changes required for running in production.

Can now set DOOZER_SETTINGS envvar (rundoozer only) with path to yaml config to us (so I can use the jenkins settings when running as podman as root)
For good measure, added ability to include arbitrary CLI options to the podman invocation. In case we need to mount another directory or something. Not sure if it'll be needed yet, but easy enough to add.

/lgtm


Reorder the tests so that nightly defines if the build fails, not stable or beta.

I would like to do this because some of my dependencies are currently nightly only, therefore my project wouldn't build on stable or beta rust.

If I understand correctly, you only want to test on nightly and not on stable/beta, right? Then simply removing these jobs from the test matrix should do that. Attention Remember to preserve the packages to install.
Other thing you could do is mark the stable/beta tests as allowed_failures.
Let me know if you actually wanted to do something else!

But the issue with putting stable and beta as allowed_failers it doesn't work.

@TheDoctorsLife Could you provide more details about how it doesn't work? i.e. what errors are you getting, what are you seeing vs what do you expect to see. Sharing your travis build log and .travis.yml would certainly help!

language: rust

matrix:
  fast_finish: true
# AFAICT There are a few ways to set up the build jobs. This one is not the DRYest but I feel is the
# easiest to reason about.
# TODO Feel free to remove the channels/targets you don't need
# NOTE Make *sure* you don't remove a reference (&foo) if you are going to dereference it (*foo)
matrix:
  include:
    # nightly channel
    - os: linux
      rust: nightly
      env: TARGET=aarch64-unknown-linux-gnu
      # need Trusty because the glibc in Precise is too old and doesn't support 64-bit arm
      dist: trusty
      sudo: required
      # Extra packages only for this job
      addons:
        apt:
          packages: &aarch64_unknown_linux_gnu
            # Transparent emulation
            - qemu-user-static
            - binfmt-support
            - libcurl4-openssl-dev
            - libelf-dev
            - libdw-dev
            - binutils-dev
            - cmake
            - gcc
            - binutils-dev
    - os: linux
      rust: nightly
      env: TARGET=armv7-unknown-linux-gnueabihf
      # sudo is needed for binfmt_misc, which is needed for transparent user qemu emulation
      sudo: required
      addons:
        apt:
          packages: &armv7_unknown_linux_gnueabihf
            # Cross compiler and cross compiled C libraries
            - gcc-arm-linux-gnueabihf
            - libc6-armhf-cross
            - libc6-dev-armhf-cross
            # Transparent emulation
            - qemu-user-static
            - binfmt-support
            - libcurl4-openssl-dev
            - libelf-dev
            - libdw-dev
            - binutils-dev
            - cmake
            - gcc
            - binutils-dev
    - os: osx
      rust: nightly
      env: TARGET=i686-apple-darwin
    - os: linux
      rust: nightly
      env: TARGET=i686-unknown-linux-gnu
      addons:
        apt:
          packages: &i686_unknown_linux_gnu
            # Cross compiler and cross compiled C libraries
            - gcc-multilib
            - libcurl4-openssl-dev
            - libelf-dev
            - libdw-dev
            - binutils-dev
            - cmake
            - gcc
            - binutils-dev
    - os: linux
      rust: nightly
      env: TARGET=i686-unknown-linux-musl
    - os: osx
      rust: nightly
      env: TARGET=x86_64-apple-darwin
    - os: linux
      rust: nightly
      env: TARGET=x86_64-unknown-linux-gnu
    - os: linux
      rust: nightly
      env: TARGET=x86_64-unknown-linux-musl
  allow_failures:
    # Beta channel
    - os: linux
      rust: beta
      env: TARGET=aarch64-unknown-linux-gnu
      dist: trusty
      sudo: required
      addons:
        apt:
          packages: *aarch64_unknown_linux_gnu
    - os: linux
      rust: beta
      env: TARGET=armv7-unknown-linux-gnueabihf
      sudo: required
      addons:
        apt:
          # Use the same packages the stable version uses
          packages: *armv7_unknown_linux_gnueabihf
    - os: osx
      rust: beta
      env: TARGET=i686-apple-darwin
    - os: linux
      rust: beta
      env: TARGET=i686-unknown-linux-gnu
      addons:
        apt:
          packages: *i686_unknown_linux_gnu
    - os: linux
      rust: beta
      env: TARGET=i686-unknown-linux-musl
    - os: osx
      rust: beta
      env: TARGET=x86_64-apple-darwin
    - os: linux
      rust: beta
      env: TARGET=x86_64-unknown-linux-gnu
    - os: linux
      rust: beta
      env: TARGET=x86_64-unknown-linux-musl
    # stable channel
    - os: linux
      rust: stable
      env: TARGET=aarch64-unknown-linux-gnu
      dist: trusty
      sudo: required
      addons:
        apt:
          packages: *aarch64_unknown_linux_gnu
    - os: linux
      rust: stable
      env: TARGET=armv7-unknown-linux-gnueabihf
      sudo: required
      addons:
        apt:
          # Use the same packages the stable version uses
          packages: *armv7_unknown_linux_gnueabihf
    - os: osx
      rust: stable
      env: TARGET=i686-apple-darwin
    - os: linux
      rust: stable
      env: TARGET=i686-unknown-linux-gnu
      addons:
        apt:
          packages: *i686_unknown_linux_gnu
    - os: linux
      rust: stable
      env: TARGET=i686-unknown-linux-musl
    - os: osx
      rust: stable
      env: TARGET=x86_64-apple-darwin
    - os: linux
      rust: stable
      env: TARGET=x86_64-unknown-linux-gnu
    - os: linux
      rust: stable
      env: TARGET=x86_64-unknown-linux-musl
      # Target `i686-unknown-linux-musl` is currently only available on the nightly channel
    - os: linux
      rust: stable
      env: TARGET=i686-unknown-linux-musl
    - os: linux
      rust: beta
      env: TARGET=i686-unknown-linux-musl

cache:
  - cargo: true
  - apt: true
  - pip: true
  - directories:
    - $HOME/installed_libsodium
    - target/debug/deps
    - target/debug/build
    - target/release/deps
    - target/release/build

# load libsodium.
before_install:
  - |
    if [ ! -d "$HOME/installed_libsodium/lib" ]; then
      wget -O $HOME/libsodium-1.0.8.tar.gz https://github.com/jedisct1/libsodium/releases/download/1.0.8/libsodium-1.0.8.tar.gz
      cd $HOME
      tar xvfz $HOME/libsodium-1.0.8.tar.gz
      cd $HOME/libsodium-1.0.8
      ./configure --prefix=$HOME/installed_libsodium
      make -j2
      make install
      cd ..
      rm -r $HOME/libsodium-1.0.8
      rm $HOME/libsodium-1.0.8.tar.gz
    fi
  - |
    if [ ! "$HOME/.cargo/bin/clippy" ]; then
      cargo install clippy
    fi
  - pip install 'travis-cargo<0.2' --user
  - export PATH=$HOME/.local/bin:$PATH
  - export PKG_CONFIG_PATH=$HOME/installed_libsodium/lib/pkgconfig:$PKG_CONFIG_PATH
  - export LD_LIBRARY_PATH=$HOME/installed_libsodium/lib:$LD_LIBRARY_PATH

# The main build
install:
  - bash ci/install.sh

script:
  - bash ci/script.sh

before_deploy:
  - bash ci/before_deploy.sh

deploy:
  provider: releases
  # TODO Regenerate this api_key for your project, this one won't work for you. Here's how:
  # - Go to 'https://github.com/settings/tokens/new' and generate a Token with only the
  # `public_repo` scope enabled
  # - Call `travis encrypt $github_token` where $github_token is the token you got in the previous
  # step and `travis` is the official Travis CI gem (see https://rubygems.org/gems/travis/)
  # - Enter the "encrypted value" below
  api_key:
    secure: aLyVgmimaRMCvh2YF4ek0ZseWqKpqCSrvvYdf5rxhZUBnYK+KKlHBsPudtmWbHTh19PPC1C3wSHa8y94IObGl3iRRTICh5e/Oz9xzCiPYlQ0ZGmtM+MfRtUq2xFRc6aaAkrnfW0ur3uxk4a7ZBIOMg3D7kG7Ah4dg5dVf26OmpSBdEijo9bcvOLHXwZIc1bby+MJl8kxzdlwaV7UTrqHQFh+tmMsoO0GQcQpCP3y45f5/8aTkHRmnC1IKCiHKDZpRFDWhdoz02NGWcCnoH43iSwOobe1jacHicmZ7dNHKT/e1tfUIPpbO81fvQ7FHifvBRoO64Wvl5l/IaoDcngv6o11JlWRxIuZnr01oDv+DW8kv9POLYeh2xzMaQkZ8NkPZl82hqr8t0q0OtXZjm/Hysdcvr0T26hbwqs1sOkAAaeRdR0zl/Log53hNqM4HaZZ0CnKGU8dSatC+NOgHEfz68fHArf8DUdKIr3By73tD7T7kJjT1h6F+4E2nYVbHrl20/PV+XO97+9LUNOy6IhbQN0hQjoHVra5oWcZy9WiWgX8IV74yw6vEAQh6XDUWtU9XR7Tl1TjaO/mR8MgOJuQkLt58QWPs50BgUD9XSvDVIurPilZ0bQzvM7js6nTupmd3oaZiYSZCxOTj0LpV34fdEaFX0qSlrSFQw6YYMAawks=
  file_glob: true
  file: ${PROJECT_NAME}-${TRAVIS_TAG}-${TARGET}.*
  # don't delete the artifacts from previous phases
  skip_cleanup: true
  # deploy when a new tag is pushed
  on:
    # channel to use to produce the release artifacts
    # NOTE make sure you only release *once* per target
    # TODO you may want to pick a different channel
    condition: $TRAVIS_RUST_VERSION = nightly
    tags: true

branches:
  only:
    # Pushes and PR to the master branch
    - master
    # IMPORTANT Ruby regex to match tags. Required, or travis won't trigger deploys when a new tag
    # is pushed. This regex matches semantic versions like v1.2.3-rc4+2016.02.22
    - /^v\d+\.\d+\.\d+.*$/

# Upload Coveralls
after_success:
- travis-cargo --only nightly bench
- |
  wget https://github.com/SimonKagstrom/kcov/archive/master.tar.gz &&
  tar xzf master.tar.gz &&
  mkdir kcov-master/build &&
  cd kcov-master/build &&
  cmake -DCMAKE_INSTALL_PREFIX:PATH=$LOCAL .. &&
  make &&
  make install &&
  cd ../.. &&
  rm -rf kcov-master &&
  for file in $HOME/build/TheDoctorsLife/discord-bot/target/debug/discord_bot-*; do mkdir -p "target/cov/$(basename $file)"; kcov --exclude-pattern=/.cargo,/usr/lib --verify "target/cov/$(basename $file)" "$file"; done &&
  bash <(curl -s https://codecov.io/bash) &&
  echo "Uploaded code coverage"

env:
  global:
    # override the default `--features unstable` used for the nightly branch (optional)
    - TRAVIS_CARGO_NIGHTLY_FEATURE=""
    - LOCAL="~/.local"
    - PATH=$LOCAL/bin:$PATH
    # encrypted github token for doc upload
    - secure: "lCN6rBfwhIckFMGFTXhYLXxr3EAHfgG3zag/NgrnGQxd2jacBIf7kUTMbXYwO2ovst97q6MxbOCEX5ebMjoCKKCywWvG2u7dYM5D7uH2W1/zCFS6YvARz3h00pFNJh1gWCX2POtFH4i231HFjWHe5FQD5TL+/q/VktLBuXwrNd+qILiPYuZMjdkGbvMjbmiT/KgI8IzA4Qd48xRih3I5uH9eDYmjZ59we98rtER4XmYq2IPBmRlkj835zEdZNAcV1kJHSuhrRVh/ijGeV703qSEB97NAM7jE0UaCWc2o/fakP77dX3bsSU7FrH3HLPstn4t1XTw1Oo9eAGa0Up6lMiVtkuzCDnzROkTmLKiQtBJIGrUrqsnFmKPFfcOhK1/kCe4xccSLg6WoQqlZaOlwjOcu2rOMCtO0phyjqBb9G/ZTcHfGShHEZV7yWVdEHp66AyFytywdr1EZsPpcllZNNZ4tODfUCv5i3BKtUopyhKYwh8wBuqT4q9orH3+h35Q7Htmi54AhqJf/UGe+5ssYUaQ9o0XyhcTiy2M88oZfndfml86jEWF7QXxraEOODgKB9nMBc6TlIVgCA03mQL+ofNlgJk5DDvyhhEymeBVOCTiPRAC+AmyQUepwTdIMyWviDPIU3GD80ocmZ7oJg/BC1fVwT0OrBs5HZV2Os2EFt+I="
    # This will be part of the release tarball
    # TODO change the project name
    - PROJECT_NAME=discord_bot
    # TODO comment out this variable if you don't want to build .deb packages on all the targets.
    - MAKE_DEB=yes
    # TODO update these two variables. They are part of the .deb package metadata
    - DEB_MAINTAINER="Benjamin Hoffmeyer<EMAIL_ADDRESS>    - DEB_DESCRIPTION="A discord-bot, written in Rust"

notifications:
  email:
    recipients:
      secure: ZmaMhBI3vU0tGSq1aFBL+yr8TZVWBygPoiihbwXyHm9yFOSJ0Xh3J2xX6/FusPO0U9r+RcCh2QSiyctFylCBtvc7FCzV5UJ7arkMgosnGb5EsOkW/NhZ9IQRxTD/27ju6TXtwjStyyDX13IEaMpG4GB43yzqGpqA0JS+CUEKJn8bLtxMeBJTBiA6rx3A0/dre65cyS5b0iL/RSWnLHgJipVCszJtYyr/Cd8XBLp7FyjeueGvd4u9AnLzYRQjuafG/zyKfn98VVJ/koEy3w5oJ8pJzqW2KVmQeyLyDre6krIP7gK9flIJJQC/PcEvs11q7IpTEtKcdDUuNLsRjfP25z9UUu8fO2vDOYJmg5iFZ40T4n+dIooT9vQvg563xn3xhwkmn+z9AWF5Ay2EHh99mKrMCrLL/68wgpzAHLDYaid/SpgGSCBH43uhfbgT4jSGsGdETmGQZGC0hrPXVDiyKHh54XZtlJCt4ZCtW+0sDyi03X6Y+ISpHV2TZtctDs4WeemAqScrnFB5LocUf0B23/UwTUEQMIBgXHZ5RbehA5cmcWzKoUGFUdcEsaA2z7QShkYaewAzFZ84v1/+b7huaL4w+Btpwk3Wu+o8c3AFe73sDr+AjD1I3gRa8lzsq7mzJ7SeHkoLZ41bTuW+NkvRAfiC2oddeK83MZLL/mMRVNM=
    on_success: change
    on_failure: change
  irc:
    channels:
      secure: Obp76KDrFueDGDfTDcyHqx2RvZ8qwUQ+Hy2hx/PSOs7VgEisOWI1jprSmElvKRaDqZVwvYxG+OG/YjCo/8+DGGtMxcOAE+1iu9EZbZSofuGjU9TQWIk3aRCgHOlAmYyT5RyiXKVBB1jJOssYq6dBrD+XiSWk5Fvjb7RqGaKoyhrs8AKcXZ997zWQsynMm1DhBid+sw7YRoFhXTK6vgzFQFoZF19DRbXD4Gj9Jc2VTPzeHkX6b2o2r0SujWA/1GM+732ty61aM1iPWcHJPgDZEhNKvBpSNQBpOBADdUfPojrDSdlsLQPEzjh/PWLEheymN2LRHUZe0JpVFS3CAtFhEOy2GlEH8WCmcTEZmjcX+rMhbF3VzLd+Iyp/o19HClNGJCW+jvGY1HTn5gn9cD5/SAQg+QVTSAJNWKNt77/WEakAqKboAmO3nHLKgPg/sL8U88WXj0b//9MdyPbOuYXMZLwDGmuGdkj3qVoX6kOg1uTxufS5+2Vph+EMYPmhe4rc3rWzt8h+jovbAXWrk2M3LDvABcbWHkDWegByMmoSjW0BaYXA1UjwUu7tuRyRzRreX8dvYMq+E6j/MltiPP1jxGWszzzjms7ILw0o42gYmFO5xHbpbfOKEgNlCujEj3UgYlXjs3q1bA4EiXTnALjgjnk3wVvOFMykL1I+sIb+cSo=
    on_success: always
    on_failure: always
    on_error: always
    on_start: true
    use_notice: true
    skip_join: true
    template:
    - "%{repository_slug}#%{build_number} (%{branch} - %{commit} : %{author}) : %{message} "
    - "Result: %{result}"
    - 'Build details: %{build_url}'
    - 'Compare commits: %{compare_url}'

You only need os, rust and env on each allowed_failures job. So change jobs like this one:
  allow_failures:
    - os: linux
      rust: beta
      env: TARGET=aarch64-unknown-linux-gnu
      dist: trusty
      sudo: required
      addons:
        apt:
          packages: *aarch64_unknown_linux_gnu

to this:
  allow_failures:
    - os: linux
      rust: beta
      env: TARGET=aarch64-unknown-linux-gnu

I'm going to close this because I've already explained how to use the allow_failures field (and there's an example in HEAD's .travis.yml. @TheDoctorsLife if you are still having problems, let me know by commenting here or opening a new issue!


fix(sasjs-run): Change access token mechanism, store log to file

Issue
Fixes https://github.com/sasjs/cli/issues/128.
Intent
Improve the sasjs run command to address the issues described in https://github.com/sasjs/cli/issues/128.
Implementation

Migrated to the access token mechanism used by the other commands - fetch it from the .env file.
Saved execution log to a sasjs-run-{timestamp}.log file.

Checks

[x] Code is formatted correctly (npm run lint:fix).
[x] Any new functionality has been unit tested.
[x] All unit tests are passing (npm test).
[x] All CI checks are green.
[x] JSDoc comments have been added or updated.

Nice.  I wonder if we should have an additional checkbox (Automated tests have been added or updated)

By automated tests, do you mean the command tests that run as part of the build? The issue with that, is that we can't run commands that need to make requests to the server yet.

Understood.  I saw you did make a unit test in this PR.  There was a discussion last week that every (non trivial) PR should contain some kind of update to a test (even if it was an unrelated test) to ensure we keep the focus on testing.  Off topic for this thread but maybe it should be in our PR template.

I thought that between 'All unit tests are passing' and 'Any new functionality has been unit tested', we've covered all scenarios. Haven't we?

:tada: This PR is included in version 1.13.5 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Cannot reference a C++ windows runtime project from a .net 5 (preview 6) C# project

After updating to .net 5 preview 6, it is no longer possible to reference a C++ windows runtime project from a C# project in Visual Studio (version 16.7.0 Preview 4.0). The build failes with this error:
error NETSDK1130: Referencing a Windows Metadata component directly when targeting .NETCoreApp,Version=v5.0 is not supported. Use the C#/WinRT projection tool (https://aka.ms/cswinrt) or a provided projection for this target.
The generation of the projections should be handled automatically in this scenario.

@lhak - the tooling to autogenerate the projection must come from CSWinRT and the component author must explicitly create a projection project.

Ok, so if I have a solution with one C# project (app code) and one C++ project (library code) I now need to manually create a third project containing the projection code and reference that one from the app project? This seems overly complicated and a regression compared to the previous situation. Or is this just a temporary issue and will be fixed in future cswinrt or Visual Studio updates?

Not only that, you have to manually change the app-project to remove all winmd-references (direct and indirect through the projection library project).

@ihak - yes, to support .NET5 applications, you as the component developer need to build the .NET5 projection assembly. Right now this is a fairly manual process, however the desired experience is:

add a .NET5 dll project to your solution
add package reference to cswinrt
set a project property telling what namespaces to include in the projection
build
update your nuget package for .NET5 target
from your .net5 app, update your reference to the projection project or use the updated nuget package

@AliveDevil - yes we are working on the props/targets for cswinrt, so when its added, it helps remove all those winmd references. Additionally we are working with the .NET team to make the error more informative and less blocking - see https://github.com/dotnet/sdk/issues/13268

@AdamBraden Thank you for the clarification. In my case, I do not want to use or publish the C++ project as a nuget package but use it to simplify calling certain Windows APIs. I would have hoped that in this cases, the creation of a dedicated projection project could be skipped and adding a project reference from the C# to the C++ project would be enough to automatically create  the projection code. Maybe this can be considered in the future?


Place library on jCenter

Can you please place this library on jCenter so it can easily be used as a dependency within build.gradle
Kind regards

I will upload it to jCenter when first release version.


Revamp index

Thanks to Mary and Stephen hard work, we have a solid basis for the documentation on top of which we can now iterate.
This first attempt at enhancing things is rather modest, and focused on a couple of key items:

shortening things: making the message easier to digest by removing redundant information, avoidable expansions, etc
clearer message: with more focus on what matters from a product POV

Let me know if you feel this is going in the right direction and if we want to go further.

This looks much more clear and concise as a getting started point :+1:
We are trying to do an overhaul of the Swarm docs as well. Just like the Registry, a lot has changed since the documentation was initially written and an update is overdue.
LGTM

LGTM!

@spf13 @moxiegirl PTAL

LGTM after nits

@NathanMcCauley nits picked.

A small nit: 'registy' should have a capital 'r' in all places, since it's a proper name.  esp. in the second heading.

I like it, very succinct.  LGTM!

@RichardScothern nits picked.

@thaJeztah nits are picked.
All: is this ready to merge?

There are issue here with capitalization.  Docker Registry is capitalized and Registry alone is not.  Seems like a nit I know but it is point in usability of the content...it allows a reader  to reader "to configure your registry" without pausing to think is this Registry or registry.  So, if by Registry you mean the Docker Registry those all need to be corrected here.

@thaJeztah Thanks for the comments on the 80 characters. Having that  limit and enforcing that particularly with documentation is problematic.  Any company growing at our pace is going to have content issues, there are multiple reasons for that.  It seems like everyone is aware of that. We are working on mechanisms to make internal writers and external contributors more effective.  I'll bring up your comments with the OSS; please go ahead and make a proposal issue (if you don't already have one), that will raise visibility.

@moxiegirl thanks.
So: the Registry (cap) is short for the Docker Registry (cap).
My Registry, your Registry, your Swarm - cap as well - correct?
An instance of the Registry - cap.
Do we agree on this?
Is there any case where we want to speak about "registry"? (and what would that be?)
Either way, this can be fixed later on. Merging this is probably more important.

Docker Registry is a proper noun used to refer to our product.  When you want to emphasize its capabilities or make it the focus of a paragraph. Refer to it as the Docker Registry.  So, you might start the paragraph off as:
The Docker Registry has this feature...and so forth. The registry was developed.
Readers understand by context that you are referring to the Docker Registry.  Similarly, if you are talking about a user's own registry implementation:
You can configure your registry. The registry recognizes its configuration.
You shouldn't use Registry upper case at all. It confuses our branding and our readers.

There is no such thing as a "user's own registry implementation".
Furthermore, "Docker Registry" is redundant. "Registry" is the product name - just like Docker Swarm is Swarm.

Users configure and deploy a registry on their own networks.  I believe they would say "our registry" or "my registry" just as if they bought an Audi A8 and added a custom stereo --- they don't say "Audi's car" when referring to their A8.
I checked upstream with the writers in the corporate communication area.  Apparently, we haven't yet protected our product names. Common words like Registry, Swarm, etc. are not typically subject to being trademarked/protected so people tend to put the company name in front of it to protect it.

LGTM!
If there are required fixes, please PR them separately.


Offline

Offline: Offline
at R in src/vs/base/parts/request/browser/request.ts:13:9
at y.request in out/vs/workbench/workbench.desktop.main.js:642:61396
at X.B in src/vs/platform/extensionManagement/common/extensionGalleryService.ts:958:40
at async X.z in src/vs/platform/extensionManagement/common/extensionGalleryService.ts:828:71
at async re in src/vs/platform/extensionManagement/common/extensionGalleryService.ts:784:34
at async X.query in src/vs/platform/extensionManagement/common/extensionGalleryService.ts:788:33
at async Vo.queryGallery in src/vs/workbench/contrib/extensions/browser/extensionsWorkbenchService.ts:986:17
at async O.R in src/vs/workbench/contrib/extensions/browser/fileBasedRecommendations.ts:336:17

Go to Errors Site

Would it make sense to not log this at all.

/duplicate
https://github.com/microsoft/vscode/issues/172787

Thanks for creating this issue! We figured it's covering the same as another one we already have. Thus, we closed this one as a duplicate. You can search for similar existing issues. See also our issue reporting guidelines.
Happy Coding!


Add typing annotations to models/quantization

Following up on #2025, this PR adds missing typing annotations in models/quantization.
Any feedback is welcome!

Here are the errors and what are being suppressed.
torchvision/models/quantization/utils.py:31: error: Incompatible types in assignment (expression has type "QConfig", variable has type "Union[Tensor, Module]")  [assignment]
            model.qconfig = torch.quantization.QConfig(
                            ^

torchvision/models/quantization/utils.py:35: error: "Tensor" not callable  [operator]
model.fuse_model()
        ^

The cat cat errors
torchvision/models/quantization/shufflenetv2.py:33: error: Argument 1 to "cat" of "FloatFunctional" has incompatible type "Tuple[Tensor, Any]"; expected "List[Tensor]"  [arg-type]
                out = self.cat.cat((x1, self.branch2(x2)), dim=1)
                                    ^
torchvision/models/quantization/shufflenetv2.py:35: error: Argument 1 to "cat" of "FloatFunctional" has incompatible type "Tuple[Any, Any]"; expected "List[Tensor]"  [arg-type]
                out = self.cat.cat((self.branch1(x), self.branch2(x)), dim=1)
                                    ^

The init errors, this probably has a nice workaround
torchvision/models/quantization/inception.py:111: error: "__init__" of "InceptionA" gets multiple values for keyword argument "conv_block"  [misc]
            super(QuantizableInceptionA, self).__init__(conv_block=QuantizableBasicConv2d, *args, **kwargs)
            ^

torchvision/models/quantization/inception.py:184: error: "__init__" of "InceptionAux" gets multiple values for keyword argument "conv_block"  [misc]
            super(QuantizableInceptionAux, self).__init__(conv_block=QuantizableBasicConv2d, *args, **kwargs)
            ^

Assigning a layer of model to None error
torchvision/models/quantization/googlenet.py:81: error: Incompatible types in assignment (expression has type "None", variable has type Module)  [assignment]
                model.aux1 = None
                             ^

For now I have skipped the errors (except init error) by specifying return type

Hi Phillip!
Thanks a lot for detailed analysis! Such work teaches a lot to a newbie developer like me. :pray:
For now I'm fixing up the annotations by adding quotes.
As you said there isn't a very neat workaround for the init errors, so probably we can take that in a new PR after some discussion.
Hence, I think that we can suppress the bunch of init errors we are getting in CI
Let me know further steps :smile:

As you said there isn't a very neat workaround for the init errors, so probably we can take that in a new PR after some discussion.
Hence, I think that we can suppress the bunch of init errors we are getting in CI

That is fair in the scope of this PR. Can you open an issue afterwards so we can track this? Maybe also add a quick # TODO comment in the code linking to my comment above.
The same (ignore + issue) holds for the fuse_model() part detailed above.

Let me know further steps :smile:

The cat error seems to be fixable without larger refactorings. Could you try and see if my suggestion passes CI? After that the PR is good to go!

Hi @pmeier and @datumbox  I think this is good to go! I will open the issue as per Phillip's comments once merged.

Just pinging for visibility @datumbox (Probably he was busy in other PRs :smiley:)

Hey @oke-aditya, Vasilis' review will be delayed until next week. Unfortunately, besides me no one has time to review now, so we'll have to wait. Sorry for that.

No problem Philip ! 🙂

Hmm unsure if it is appropriate. But just re-tagging @datumbox


if we should check aliveness in load-balancer

we choose the proxy in load-balance group based on hash algo's direct output, without checking the chosen proxy's aliveness
is it expected to do so? clash.meta will retry if the calculated proxy is not alive
this issue is related to the feature request of sticky-sessions strategy, if the aliveness is not checked, the behavior of consistent-hash and sticky-sessions are very much alike

Good point.
Nginx does provide two approaches regarding the availability of the upstream server https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/#method
There might be some concepts we can borrow from.


Compatibility issue with the Crazy Enchantment's plugin's Furnace enchantment

Crazy Enchantments  has a plugin called Furnace that automatically smelts mined ores. When this happens, the Miner job doesn't register the action.
I would expect it to give the same amount of xp and money you would get from mining an iron ore with an unenchanted pickaxe, but it doesn't give any at all. Mining it with a regular pickaxe works as intended.

If plugin is not firing actual smelt event with appropriate data which can be process by Jobs plugin, then i cant really help you out. Things like this should be handled by Furnace plugin it self.


add user filter by asignees

@rrebase Finally I managed to spend some time and work on filtering the assignees. I'm just confronting an issue with backend code. I need to use distinct()  for the search API. As far as I know, MySql doesn't support distinct functionality and I have to use Postgres. The problem is that I couldn't find any settings file for Django to support Postgres in the project and also when I run docker-compose up I get an error that '....env' is not a file. What do you suggest to do? Thanks
    def get_queryset(self):
        user = self.request.user
        qs = super().get_queryset().filter(members=user)
        assignees = self.request.query_params.get('assignees', None)
        if self.action == "retrieve":
            queryset = None
            if assignees:
                queryset = Task.objects.filter(
                    Q(assignees__in=[int(x) for x in assignees.split(',')]))
            return qs.prefetch_related(Prefetch('columns__tasks', queryset=queryset)).distinct('id')
        return qs

The Django ORM is database agnostic with some exceptions. SQLite is actually the easiest to use for local development as it's just a file. By default it should already use db.sqlite3 unless you set it to something else.
docker-compose.yml is currently meant for running the services in production configuration. Could also create a separate one for local development if there was a need for it.
Regarding filtering assignees - did you consider doing it only on the React side? Currently when opening a board all the data for that board is fetched - tasks, members, etc. So all the data required for filtering is already there and there's no need for extra API calls. If the filtering was across multiple boards then API endpoints like above are needed 🙂

I had some problems to get the project up and running using Postgres. I will leave the commands to run the project here.
virtualenv .venv --python=python3.7 #or latest version of python if postgres works with it
.venv/scripts/activate
pip install -r requirements/local.txt

#if failed to import from psycopg:
pip install psycopg2-binary

docker-compose -f services.yml up --d

#if postgres is installed locally, stop its service or uninstall it
python manage.py migrate --settings=config.settings.local
python manage.py createsuperuser --username admin --email<EMAIL_ADDRESS>--settings=config.settings.local
python manage.py loaddata avatars --settings=config.settings.local
python manage.py runserver --settings=config.settings.local

@rrebase the code for filtering board members is ready and for testing it Postgres needs to be running (I also pasted the commands for running it in the previous comment).
The unit tests fail because role "knboard" does not exist" in Postgres of the server. The code is also ready for review. I also need a second opinion to resolve codeclimate issues. Thanks

Should disable/configure the duplication check for codeclimate as it shows absurd errors.

I'd love to see tests as well. codeclimate should require a 90% coverage anyway 🙂

I'll try to disable/configure the duplication check for codeclimate as it shows absurd errors. 🤔
Marked the current ones as invalid.
Thanks. I also wanted to learn how you solved the problem for the role in the Postgres? did you add the exact role and password to the backend or is there a different solution for it?
And about the tests, will tests in MemberListDialog.test.tsx b enough? I'm not sure which tests should go into Cypress and which tests here.

I'll try to disable/configure the duplication check for codeclimate as it shows absurd errors. 🤔
Marked the current ones as invalid.

Thanks. I also wanted to learn how you solved the problem for the role in the Postgres? did you add the exact role and password to the backend or is there a different solution for it?

Hmm, I didn't have any problems with the Postgres role. The docker image should automatically create them and I used the same DB connection settings you added.

I'll try to disable/configure the duplication check for codeclimate as it shows absurd errors. 🤔
Marked the current ones as invalid.

Thanks. I also wanted to learn how you solved the problem for the role in the Postgres? did you add the exact role and password to the backend or is there a different solution for it?

Hmm, I didn't have any problems with the Postgres role. The docker image should automatically create them and I used the same DB connection settings you added. Maybe you already have a different database running locally on the same port?

Guess I was asking the wrong question. 😄

@rrebase I think it is ready to be merged. The backend tests are failing because it can not find knboard role again!
If it's all good I can move to adding reset password feature to the system 😄

It makes sense that it's failing in CI. .circleci/config.yml was previously using sqlite for tests and now local.py has a different DB configuration (which is used for tests).
.circleci/config.yml should now be updated to have a postgres instance running either through docker or other means before the pytest step.

It makes sense that it's failing in CI. .circleci/config.yml was previously using sqlite for tests and now local.py has a different DB configuration (which is used for tests).
.circleci/config.yml should now be updated to have a postgres instance running either through docker or other means before the pytest step 😁

@rrebase I first created a new image and waited for it to load, turns out all I needed was to change the user and password of Postgres that was trying to connect! 🤕
      - setup_remote_docker
      - run: docker-compose -f ../services.yml up -d --no-recreate
      - run:
          name: install dockerize
          command: wget https://github.com/jwilder/dockerize/releases/download/$DOCKERIZE_VERSION/dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz && sudo tar -C /usr/local/bin -xzvf dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz && rm dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz
          environment:
            DOCKERIZE_VERSION: v0.3.0
      - run:
          name: Wait for db
          command: dockerize -wait tcp://localhost:5432 -timeout 1m

I'm getting a new error ImportError: cannot import name 'ORDER_PATTERN' from 'django.db.models.sql.constants' ,  not sure how to resolve it!

It seems that django rest framework needed upgrading! That took some time to figure out!
Two of the tests are failing, I think test_str_methods is failing due to upgrade.

It seems that Django rest framework needed an upgrade! That took some time to figure out!
Two of the tests are failing, @rrebase Can you please help with these two issues?

I think test_str_methods is failing due to the upgrade. It results in making a call to make_password.
I'm not quite sure what the logic for test_order_column_status_code is, it usually fails during the first run but succeeds randomly afterwards.

test_str_methods seems to be failing due to an upgrade indeed. Using PostGenerationMethodCall like that doesn't seem to be supported anymore. Could just replace with a simple password = Faker('password') (UserFactory in
tests/factories.py) and remove assert user.check_password("jackpassword") from test_factories.py
test_order_column_status_code is sometimes failing since database id's are used in parameterized tests set and if another test runs before then the Column objects won't have ids 1 & 2 anymore. That's a mistake and one fix would be to set the ids e.g Column.objects.create(id=1, board=board, title="col1") or move the first ({"order": [1, 2]}, 200),  into a separate test that properly uses the actual ids from created objects (since it can't be done in @pytest.mark.parametrize if I'm not mistaken)

@rrebase Thanks, it worked. But unfortunately a new is happening (only on the server and not local) when running pytest.
ImportError while loading conftest '/home/circleci/project/backend/conftest.py'.
conftest.py:5: in <module>
    from boards.tests.factories import (
boards/tests/factories.py:8: in <module>
    class UserFactory(factory.DjangoModelFactory):
E   AttributeError: module 'factory' has no attribute 'DjangoModelFactory'

@rrebase Thanks, it worked. But unfortunately a new is happening (only on the server and not local) when running pytest.
ImportError while loading conftest '/home/circleci/project/backend/conftest.py'.
conftest.py:5: in <module>
    from boards.tests.factories import (
boards/tests/factories.py:8: in <module>
    class UserFactory(factory.DjangoModelFactory):
E   AttributeError: module 'factory' has no attribute 'DjangoModelFactory'

Noticed that as well. It seems to be a bug on factory_boy side, see https://github.com/FactoryBoy/factory_boy/blob/master/docs/changelog.rst (3.0.1). Also the import was changed in 3.0.0 to from factory.django import DjangoModelFactory

Noticed that as well. It seems to be a bug on factory_boy side, see https://github.com/FactoryBoy/factory_boy/blob/master/docs/changelog.rst (3.0.1). Also the import was changed in 3.0.0 to from factory.django import DjangoModelFactory

Finally all tests finally pass! 🎆 🥳  The pull request reminded me Hal fixing lightbulb!
Although I didn't understand why the tests were passing on local machine but failing on the server since both were using the same pytest-factoryboy version!?
Thanks for your the help and comments


#13157 open project scripts

We now can press SHIFT + F3 to open project scripts instead of only datasource one

We don't need new icon icon, we need to add checkbox in script selector.
Do not read scripts by datasource (it is super-slow in case of big number of connections)


How to get label from Picker?

I have gone through these following points

[X]   Check latest documentation: https://docs.nativebase.io/
[X]   Check for existing open/closed issues for a possible duplicate before creating a new issue: https://github.com/GeekyAnts/NativeBase/issues
[X]   Use the latest NativeBase release: https://github.com/GeekyAnts/NativeBase/releases
[X]   Check examples from NativeBase KitchenSink https://github.com/GeekyAnts/NativeBase-KitchenSink
[X]   For discussion purpose make use of NativeBase Slack: http://slack.nativebase.io/
[X]   For queries related to theme, check Theme Variables from Docs and live NativeBase Theme Editor http://nativebase.io/customizer/

Issue Description

node, npm, react-native, react and native-base version, expo version if used, xcode version
Node: v10.14.2
React-native: react-native: 9.57.1
Expo: 2.6.14
Expected behaviour
Trying to retrieve value AND index from a picker.
Actual behaviour
Value is returned, index is not.
Steps to reproduce
Picker is loaded (from Firestore) using following function;
snapshot.forEach(doc => { list.push( <Picker.Item value={doc.id} label={doc.data()["RegionName"] key={doc.id /> )
CODE TO DISPLAY PICKER:
`<Picker
selectedValue={this.regionID}
onValueChange={(value,index) => this.onRegionChange(value, index)}
CODE EXECUTED ON CHANGE:onRegionChanged(value, index) {
this.regionID = value;
console.log('Value: ', value)
console.log('Index: ', index)
}
`
** if I look at the console at this point, the value shows the correct value (from firestore), while the index shows as Undefined.
Is the bug present in both iOS and Android or in any one of them?
YES
Any other additional info which would help us debug the issue quicker.
Using Mobx and Mobx-react

@derekgould
See "react-native" docs about Picker...
https://facebook.github.io/react-native/docs/picker#onvaluechange
Don't think your issue is related to "native-base"...

Closing the issue due to no response for a while...Hope your issue has been resolved.You can reopen the issue if you still feel the issue is exclusive to "native-base"...


Azure DevOps Syntax Highlithing - Wrong Highliting of KeyWords like Separator in procedures parameters

1. Describe the bug
Create a procedure with ParameterName with Name Separator and view this file in Azure Devops
2. To Reproduce
Steps to reproduce the behavior:

Go to '...'

procedure CreateString(Separator: Text)  //Wrong Hightlithing in Azure DevOps
begin
end;

3. Expected behavior
KeyWords in parameter Names should be ignored similiar to VS-Code:

4. Versions:

AL Language Syntax Highlights for Microsoft Dynamics 365 Business Central: 12.1.886884 (Latest)

What are you using for highlighting within Azure DevOps?

What are you using for highlighting within Azure DevOps?

Probably https://marketplace.visualstudio.com/items?itemName=ms-dynamics-smb.allanghighlights

Hello, this difference is due to the presence of semantic highlighting in the Visual Studio Code extension. To illustrate:

We strive to make the Azure DevOps extension have parity with VS Code extension syntax highlighting, Unfortunately Azure DevOps doesn't support semantic highlighting, this the gap you are experiencing. Hope this helped.
Thank you for reporting the issue, but as we cannot solve it so I am closing as not planned. Have a nice day!


WIP: Change ConnectInfo to take a RemoteConfig.

We need the other attributes of RemoteConfig in order to connect
to the simplestreams and other 'static' hosts.
Signed-off-by: John Arbash Meinel<EMAIL_ADDRESS>I'm not 100% sure this is enough for us, I want to update the Juju client to match, so that Juju can use the known remotes to copy images for LXD provider instead of "lxd-images import". But I wanted to check that this is an ok step in that direction.

This pull request didn't trigger Jenkins as its author isn't in the whitelist.
An organization member must perform one of the following:

To have this branch tested by Jenkins, use the "ok to test" command.
To have a one time test done, use the "test this please" command.

Those commands are simple Github comments of the format: "jenkins: COMMAND"

+1 on this, I have a PR that changes the way juju imports images, so I'll try and update it to use this (and whatever else we need) as well. I talked with @jameinel about it, so we're on the same page (although I need to find some time to update my juju pr :)

jenkins: test this please

Please run "go fmt ./..."

jenkins: test this please

Looks good, keeping it blocked until we finish doing some jenkins optimizations though.


[fftw3]Changed library linkage to static.

Because many functions are not declared as exportable in fftw3, we can only build them as static libraries now.
The following are these functions(prefix with X in the declaration):

imax
mksolver_ct_hook
mksolver_hc2hc_hook
mksolver
mkplan_d
mkplan_dft
mkplan_rdft
mkplan_rdft2
mktensor_1d
mktensor_2d
mkproblem_dft
mkproblem_dft_d
mkproblem_rdft
mkproblem_rdft_d
mkproblem_rdft2
solvtab_exec
dft_solve
ops_zero
ops_add2
tensor_tornk1
plan_destroy_internal
choose_radix
ct_applicable
tensor_copy
tensor_destroy
solver_register
pickdim
rdft_solve
hc2hc_applicable
rdft2_solve
rdft2_inplace_strides
rdft2_strides
set_planner_hooks

Relate: #6056.

After this merge, I can no more link my C++ code to FFTW3 with msvc compilers (both using Visual Studio projects and CMake projects, in Windows 10). Errors like

error LNK2019: unresolved external symbol __imp_fftw_execute referenced in function [...]

arise.
I have to restore the previous version of file ports/fftw3/portfile.cmake in order to be able to produce my executables.


Color ramp for isochrones fails for more than 10 isochrones

When using a multi-point layer to calculate isochrones, the hard-coded color scheme incrementally adds another color from the ramp, instead of defaulting to color(0) for each new point.
Also, script fails when more than 10 isochrones are calculated in total during one operation.
@noerw

Finally fixed in qgis2.99-dev


[TS3][Client][Server]Setting up a Musicbot on Ubuntu 14.04 LTS

Hey, you're my last hope :)
I want to set up a TS³ client on my Ubuntu Server, however, everytime I want to start the client via my VNC software, Its the same..
First time I start, no problems..
If I restart the client I get this error:
Qt: couldn't retrieve an initial keyboard state
2015-02-14 18:05:23.250378|INFO    |              |   | TeamSpeak 3 Client 3.0.16 (2014-08-04 13:42:43)
2015-02-14 18:05:23.250459|INFO    |              |   | SystemInformation: Linux 3.13.0-042stab092.3 #1 SMP Sun Jul 20 13:27:24 MSK 2014 x86_64 Binary: 64bit
2015-02-14 18:05:23.250484|INFO    |              |   | Using hardware aes
2015-02-14 18:05:23.263864|DEBUG   |PulseAudio    |   | connected to pulse audio server
XIO:  fatal IO error 2 (No such file or directory) on X server ":1"
after 5 requests (5 known processed) with 0 events remaining.
Already looked everywhere, but I didn't find anything that helped me :(
My system: Ubuntu 14.0.4 LTS 64bit
installed: TS-Client 64bit xfce4 xfce4-goodies tightvncserver
Hope you can help me :)

I'm afraid this is not the sort of issue know how to resolve as it is no way related to LInux Game Server Managers. I recommend asking the question on http://askubuntu.com/ you will have much more luck there.

However it looks like some sort of issue with X windows system. Apart from that I don't know good luck
XIO: fatal IO error 2 (No such file or directory) on X server ":1"


Set the counter into an element by ID

Hi,
Any idea how I can put the counter in an existing element? I basically have (as per screenshot) an area where to show a counter which I want to use instead of having the counter floating around.
Any help appreciated.
Thanks.

Necrospasm, per the API docs .... the value for the placement property can be set as one of three things.
It can be a string, such as
placement: "top left"
It can be an object, such as
placement: {'top':'10px', 'left':30px'}
Or it can be a callback function such as
placement: function(a, b, c){
       // a is the element that it is binded to, in other words the object whose length you are checking
       // b is the element that is being created, in other words the character counter output
       // c contains info useful for positioning the counter, such as the offsets of the object. the width
       of the object, etc.
  }

Here is an example of how I used the function .....
 $('input[maxlength]').maxlength({
                     alwaysShow:true,
                     showOnReady:true,
                     appendToParent:true,
                     warningClass:'',
                     placement:function($cont, $obj, pos){
                            $cont.parents().next().html($obj);
                     }
});

... I have a bootstrap horizontal form.  The grid is divided as follows ... label is set to "col-sm-2" the input is wrapped in a div with "col-sm-9" and the last grid is set to "col-sm-1".  With the function above, for each of my inputs that have a max length defined, the object is appended to the sibling container (the last grid item).
The author did a really good job of providing hooks to make this utility very flexible.


GEODE-1558: Fixing failures due to rebalancing buckets during lucene …

…index updates
There are two failures we're fixing here:

Calling repository.create for create events was leaving duplicate events in the
index because when a bucket fails over, the event is dispatched again on the new primary.
Using the isPossibleDuplicate flag did not work because it was not consistently set to
true for duplicate events. Changed the code to call repository.update even for create events

The async event queue was repeatedly dispatching the same events event after
a bucket moved to another node. We changed the async event queue code to filter out
events for buckets that are no longer present on this dispatching member.

Cleaning up the rebalancing test and adding new tests to make these scenarios
more reproducable.
Signed-off-by: Aparna Dharmakkan<EMAIL_ADDRESS>

+1

+1


omni_createpayload_simplesend doesn't handle decimals well

Here are some examples
./bin/omnicore-cli -datadir=data "omni_createpayload_simplesend" 3 "1"
00000000000000030000000000000001

./bin/omnicore-cli -datadir=data "omni_createpayload_simplesend" 3 "1.01"
00000000000000030000000000000001

./bin/omnicore-cli -datadir=data "omni_createpayload_simplesend" 3 "0.1"
error code: -3
error message:
Invalid amount

Notice that everything after the decimal is ignored, and amounts less than 1 result in an error. Is this expected behavior? Should amounts of divisible tokens be provided in the lowest possible denomination (i.e. multiple the amount by 100,000,000)?

If you look this line of the code https://github.com/OmniLayer/omnicore/blob/master/src/omnicore/rpcpayload.cpp/#L41
you can see how the amount variable is created. This variable depends of the property (if this property is divisible or indivisible). I don't know if you are in testnet or in mainnet, but i can guess that the property 3 that you are using is indivisible, so if you write 1.01 , the system just take this as 1. If you write 0.1, the code take this as 0, so you can't send 0 amount (this happened in parseamoun function: https://github.com/OmniLayer/omnicore/blob/master/src/omnicore/rpcvalues.cpp/#L60). I hope it helps you @palimarrao. Best Regards!

ahh I see, thanks for the detailed explanation @santos177! This helped a lot 👍


Master branch miscalculates the expected argument count when a hash is passed to rest of keywords

Subject of the issue
Given an instance method on a class defined as
def test(arg, **options) ... end

when creating a verified partial double from an instance of that class that calling it
allow(some_object).to receive(:test)
some_object.test('something', {})
the error Wrong number of arguments. Expected 1, got 2 is returned which is incorrect since the empty hash is accepted by the rest of keywords **options

Fixed in #374