Hyper Dimensional Aggregates (HDA)

Migrated from GitLab: https://gitlab.com/meltano/meltano/-/issues/286
Originally created by @iroussos on 2019-01-15 13:53:38

This issue describes the problem of auto generating aggregates over joins and how you can very easily end up with the wrong results. We are using an example to showcase the various issues and construct a proper query using the most simple, non optimized approach possible. We then sketch an algorithm that can easily be implemented as a starting point, so that we can optimize it in future iterations.
Characteristics of the approach:

It is really really dumb. It handles all aggregates in the same way, without distinguishing the ones that can be computed without issues straight from the join.
It is agnostic: It doesn't need to know whether the relation between the various tables is one-to-one, one-to-many or whatever.
Due to (1), it is very simple and easily implemented. But it is also super non-optimal and should not scale above couple million rows of data (??? needs tests ???).
It is meant to follow simple rules: No real analyst would construct such a query.

The problem
Let's assume that we have a very simple schema for GitFlix, a streaming video service.
Users stream episodes from various TV series. For each user we have stored their name, age and their lifetime value to GitFlix (total subscriptions until today).
We follow a simple, denormalized, schema for episodes, with their number (e.g. '304' for episode 4 of season 3), title, the TV series they belong to (e.g. 'Star Trek TNG') and the rating the episode got in IMDb.
USERS (id, name, age, gender, clv)
  |
STREAMS (id, user_id, episode_id, minutes, day, month, year)
  |
EPISODES (id, no, title, tv_series, rating)

We want to calculate a pretty complicated analytics report. Someone asked us to generate for each TV series with an episode streamed on a specific date (e.g. 2019/01/11):

the total number of users that watched any episode of the series (total reach of the series)
their average age (to figure out what's the target group)
their total lifetime value (how much they worth to GitFlix? is there a correlation between some series and people subscribing?)
the total streams for the series
the total minutes streamed (you may have lots of streams with people not watching the full episode)
the total number of episodes watched (do people binge watch it?)
and the average rating those episodes have in IMDb (are those considered a good set of episodes or do they watch it for some other reason?)

And all of the above separately by gender, as there may be some type of a gender bias in tv series.
How the above instructions are translated?

Generate a join between the 3 tables
Filter (in the where clause) by date and only keep streams for 2019/01/11
We want to group by TV Series and User Gender
Somehow COUNT the total users, AVERAGE their age and SUM their CLV for this dataset.

As we'll see, just straight forward joining everything together and computing the aforementioned aggregates will lead to very wrong results for most of the requested stats.
Example data set
We create the following tables in Postgres and seed with some data so that we can see how various queries behave:
CREATE SCHEMA test_aggregates;

drop table streams;
drop table users;
drop table episodes;

CREATE TABLE users (
        id integer PRIMARY KEY,
        name varchar(40),
        age integer NOT NULL,
        gender varchar(10),
		clv float NOT NULL
);

CREATE TABLE episodes (
        id integer PRIMARY KEY,
        no varchar(40) NOT NULL,
        title varchar(40) NOT NULL,
        tv_series varchar(40),
        rating float NOT NULL
);

CREATE TABLE streams (
        id integer PRIMARY KEY,
        user_id integer REFERENCES users (id),
        episode_id integer REFERENCES episodes (id),
        minutes integer NOT NULL,
        day integer NOT NULL,
        month integer NOT NULL,
        year integer NOT NULL
);

INSERT INTO users VALUES (1, 'John', 23, 'male', 163.7);
INSERT INTO users VALUES (2, 'George', 42, 'male', 287.3);
INSERT INTO users VALUES (3, 'Mary', 19, 'female', 150.3);
INSERT INTO users VALUES (4, 'Kate', 52, 'female', 190.1);
INSERT INTO users VALUES (5, 'Bill', 35, 'male', 350.8);
INSERT INTO users VALUES (6, 'Fiona', 63, 'female', 278.5);

INSERT INTO episodes VALUES (1, '101', 'Pilot', 'Breaking Bad', 8.9);
INSERT INTO episodes VALUES (2, '102', 'Cat in the Bag...', 'Breaking Bad', 8.7);
INSERT INTO episodes VALUES (3, '202', 'Grilled', 'Breaking Bad', 9.2);
INSERT INTO episodes VALUES (4, '101', 'The National Anthem', 'Black Mirror', 7.9);
INSERT INTO episodes VALUES (5, '406', 'Black Museum', 'Black Mirror', 8.7);
INSERT INTO episodes VALUES (6, '104', 'Old Cases', 'The Wire', 8.3);
INSERT INTO episodes VALUES (7, '306', 'Homecoming', 'The Wire', 8.9);

INSERT INTO streams VALUES (1, 1, 1, 40, 10, 1, 2019);
INSERT INTO streams VALUES (2, 1, 2, 42, 10, 1, 2019);
INSERT INTO streams VALUES (3, 1, 3, 38, 11, 1, 2019);
INSERT INTO streams VALUES (4, 1, 4, 12, 11, 1, 2019);
INSERT INTO streams VALUES (5, 1, 5, 27, 11, 1, 2019);
INSERT INTO streams VALUES (6, 2, 2, 36, 11, 1, 2019);
INSERT INTO streams VALUES (7, 2, 6, 45, 11, 1, 2019);
INSERT INTO streams VALUES (8, 2, 7, 44, 11, 1, 2019);
INSERT INTO streams VALUES (9, 3, 4, 40, 10, 1, 2019);
INSERT INTO streams VALUES (10, 3, 5, 41, 11, 1, 2019);
INSERT INTO streams VALUES (11, 3, 1, 11, 11, 1, 2019);
INSERT INTO streams VALUES (12, 4, 3, 22, 10, 1, 2019);
INSERT INTO streams VALUES (13, 4, 3, 18, 11, 1, 2019);
INSERT INTO streams VALUES (14, 4, 6, 40, 11, 1, 2019);
INSERT INTO streams VALUES (15, 5, 2, 34, 11, 1, 2019);
INSERT INTO streams VALUES (16, 5, 4, 41, 11, 1, 2019);
INSERT INTO streams VALUES (17, 5, 5, 39, 12, 1, 2019);
INSERT INTO streams VALUES (18, 5, 6, 36, 12, 1, 2019);
INSERT INTO streams VALUES (19, 6, 1, 19, 11, 1, 2019);
INSERT INTO streams VALUES (20, 6, 3, 35, 11, 1, 2019);
INSERT INTO streams VALUES (21, 6, 7, 48, 11, 1, 2019);
INSERT INTO streams VALUES (22, 6, 1, 24, 12, 1, 2019);

This is what we have to play with:
# SELECT * FROM users;
 id |  name  | age | gender |  clv
----+--------+-----+--------+-------
  1 | John   |  23 | male   | 163.7
  2 | George |  42 | male   | 287.3
  3 | Mary   |  19 | female | 150.3
  4 | Kate   |  52 | female | 190.1
  5 | Bill   |  35 | male   | 350.8
  6 | Fiona  |  63 | female | 278.5
(6 rows)

# SELECT * FROM episodes;
 id | no  |        title        |  tv_series   | rating
----+-----+---------------------+--------------+--------
  1 | 101 | Pilot               | Breaking Bad |    8.9
  2 | 102 | Cat in the Bag...   | Breaking Bad |    8.7
  3 | 202 | Grilled             | Breaking Bad |    9.2
  4 | 101 | The National Anthem | Black Mirror |    7.9
  5 | 406 | Black Museum        | Black Mirror |    8.7
  6 | 104 | Old Cases           | The Wire     |    8.3
  7 | 306 | Homecoming          | The Wire     |    8.9
(7 rows)

# SELECT * FROM streams;
 id | user_id | episode_id | minutes | day | month | year
----+---------+------------+---------+-----+-------+------
  1 |       1 |          1 |      40 |  10 |     1 | 2019
  2 |       1 |          2 |      42 |  10 |     1 | 2019
  3 |       1 |          3 |      38 |  11 |     1 | 2019
  4 |       1 |          4 |      12 |  11 |     1 | 2019
  5 |       1 |          5 |      27 |  11 |     1 | 2019
  6 |       2 |          2 |      36 |  11 |     1 | 2019
  7 |       2 |          6 |      45 |  11 |     1 | 2019
  8 |       2 |          7 |      44 |  11 |     1 | 2019
  9 |       3 |          4 |      40 |  10 |     1 | 2019
 10 |       3 |          5 |      41 |  11 |     1 | 2019
 11 |       3 |          1 |      11 |  11 |     1 | 2019
 12 |       4 |          3 |      22 |  10 |     1 | 2019
 13 |       4 |          3 |      18 |  11 |     1 | 2019
 14 |       4 |          6 |      40 |  11 |     1 | 2019
 15 |       5 |          2 |      34 |  11 |     1 | 2019
 16 |       5 |          4 |      41 |  11 |     1 | 2019
 17 |       5 |          5 |      39 |  12 |     1 | 2019
 18 |       5 |          6 |      36 |  12 |     1 | 2019
 19 |       6 |          1 |      19 |  11 |     1 | 2019
 20 |       6 |          3 |      35 |  11 |     1 | 2019
 21 |       6 |          7 |      48 |  11 |     1 | 2019
 22 |       6 |          1 |      24 |  12 |     1 | 2019
(22 rows)

Example of poor results
Let's say that we want to focus only on users that watched 'Black Mirror' on 2019/01/11.
The base join is the same as the one we are going to use, so it will serve as a nice axample:
SELECT *
FROM users
  join streams on users.id = streams.user_id
  join episodes on episodes.id = streams.episode_id
WHERE day = 11 and episodes.tv_series = 'Black Mirror'
ORDER BY episodes.tv_series, users.id;

 id | name | age | gender |  clv  | id | user_id | episode_id | minutes | day | month | year | id | no  |        title        |  tv_series   | rating
----+------+-----+--------+-------+----+---------+------------+---------+-----+-------+------+----+-----+---------------------+--------------+--------
  1 | John |  23 | male   | 163.7 |  4 |       1 |          4 |      12 |  11 |     1 | 2019 |  4 | 101 | The National Anthem | Black Mirror |    7.9
  1 | John |  23 | male   | 163.7 |  5 |       1 |          5 |      27 |  11 |     1 | 2019 |  5 | 406 | Black Museum        | Black Mirror |    8.7
  3 | Mary |  19 | female | 150.3 | 10 |       3 |          5 |      41 |  11 |     1 | 2019 |  5 | 406 | Black Museum        | Black Mirror |    8.7
  5 | Bill |  35 | male   | 350.8 | 16 |       5 |          4 |      41 |  11 |     1 | 2019 |  4 | 101 | The National Anthem | Black Mirror |    7.9
(4 rows)

Let's directly calculate the user stats requested for this simple data set:
SELECT gender, COUNT(*) as total_users, AVG(age) as average_age, SUM(clv) as total_clv
FROM users
  join streams on users.id = streams.user_id
  join episodes on episodes.id = streams.episode_id
WHERE day = 11 and episodes.tv_series = 'Black Mirror'
GROUP BY gender
ORDER BY gender;

 gender | total_users |     average_age     | total_clv
--------+-------------+---------------------+-----------
 female |           1 | 19.0000000000000000 |     150.3
 male   |           3 | 27.0000000000000000 |     678.2

You can directly see the problem in the result above for male users (as they have duplicates):

total_users == 3 --> SHOULD BE 2 (counting John twice)
average_age == 27 --> SHOULD BE 29 (John brings the average down)
total_clv == 678.2 --> SHOULD BE 514.5 (John's clv was added twice)

So, that means that we are doing this the wrong way. We have to somehow remove the duplicates before calculating the stats specific to this relation.
Doing it right:
The base idea is that you can compute the full join with whatever restrictions you may have (where clauses) and then focus on each relation to compute the aggregates for that relation.
There are three steps required:

Compute the full join only for the date requested 2019/01/11 and "hold" the result to a temporary relation we'll call base_join (you don't exactly hold or store the result but this is not important)
For the relation of interest (users in our case) select the distinct rows based on the id of the relation and all the attributes that are going to be used for grouping later on. This is an important step for throwing away all the rows that mess with our aggregates, but not throw too much data away (hence the need to also use the other attributes that will be used for grouping). The reason is that you care for all those attributes like they are your base: in our case we count and sum and average based on the TV series name, the user gender and we also need the user ID in order to throw away all other duplicates. Let's call this intermediate result base_users
Compute the required aggregates over base_users by properly grouping by all the attributes.

End result (with additionally only selecting the records for 'Black Mirror' in order to compare with the problematic query of the previous section):
with base_join as (
	SELECT users.id as users_id, users.name as users_name, users.age as users_age,
	       users.gender as users_gender, users.clv as users_clv,
		   episodes.id as episodes_id, episodes.no as episodes_no, episodes.title as episodes_title,
	       episodes.tv_series as episodes_tv_series, episodes.rating as episodes_rating,
		   streams.id as streams_id, streams.minutes as streams_minutes
	FROM users
	  join streams on users.id = streams.user_id
	  join episodes on episodes.id = streams.episode_id
	WHERE day = 11
),
base_users as (
	SELECT DISTINCT ON (users_id, episodes_tv_series, users_gender) users_id, users_name, users_age, users_gender, users_clv,
		   episodes_tv_series, episodes_rating
	FROM base_join
)
SELECT episodes_tv_series, users_gender, COUNT(*) as total_users, AVG(users_age) as average_age, SUM(users_clv) as total_clv
FROM base_users
WHERE episodes_tv_series = 'Black Mirror'
GROUP BY episodes_tv_series, users_gender
ORDER BY episodes_tv_series, users_gender;

episodes_tv_series | users_gender | total_users |     average_age     | total_clv
--------------------+--------------+-------------+---------------------+-----------
 Black Mirror       | female       |           1 | 19.0000000000000000 |     150.3
 Black Mirror       | male         |           2 | 29.0000000000000000 |     514.5

The results for male users are now what they should be :-)
Generalizing to a FULL Query
Following the same approach, we add 2 more clauses for each table with an aggregate (all 3 tables in our case) and in the end we join the results "hold" in the three XXX_stats in order to get our proper full blown report:
with base_join as (
	SELECT users.id as users_id, users.name as users_name, users.age as users_age,
	       users.gender as users_gender, users.clv as users_clv,
		   episodes.id as episodes_id, episodes.no as episodes_no, episodes.title as episodes_title,
	       episodes.tv_series as episodes_tv_series, episodes.rating as episodes_rating,
		   streams.id as streams_id, streams.minutes as streams_minutes
	FROM users
	  join streams on users.id = streams.user_id
	  join episodes on episodes.id = streams.episode_id
	WHERE day = 11
),
base_users as (
	SELECT DISTINCT ON (users_id, episodes_tv_series, users_gender) users_id, users_name, users_age, users_gender, users_clv,
		   episodes_tv_series, episodes_rating
	FROM base_join
),
users_stats as (
	SELECT episodes_tv_series, users_gender, COUNT(*) as total_users, AVG(users_age) as average_age, SUM(users_clv) as total_clv
	FROM base_users
	GROUP BY episodes_tv_series, users_gender
	ORDER BY episodes_tv_series, users_gender
),
base_streams as (
	SELECT DISTINCT ON (streams_id, episodes_tv_series, users_gender) streams_id, streams_minutes,
		   episodes_tv_series, users_gender
	FROM base_join
),
streams_stats as (
	SELECT episodes_tv_series, users_gender, COUNT(*) as total_streams, SUM(streams_minutes) as total_streams_minutes
	FROM base_streams
	GROUP BY episodes_tv_series, users_gender
	ORDER BY episodes_tv_series, users_gender
),
base_episodes as (
	SELECT DISTINCT ON (episodes_id, episodes_tv_series, users_gender) episodes_id, episodes_rating,
		   episodes_tv_series, users_gender
	FROM base_join
),
episodes_stats as (
	SELECT episodes_tv_series, users_gender, COUNT(*) as total_episodes, AVG(episodes_rating) as avg_rating
	FROM base_episodes
	GROUP BY episodes_tv_series, users_gender
	ORDER BY episodes_tv_series, users_gender
),
result as (
	SELECT episodes_stats.episodes_tv_series, users_stats.users_gender,
	       users_stats.total_users, users_stats.average_age, users_stats.total_clv,
		   streams_stats.total_streams, streams_stats.total_streams_minutes,
		   episodes_stats.total_episodes, episodes_stats.avg_rating
	FROM users_stats
	  join streams_stats on
	    (users_stats.episodes_tv_series = streams_stats.episodes_tv_series) and
	    (users_stats.users_gender = streams_stats.users_gender)
	  join episodes_stats on
	    (users_stats.episodes_tv_series = episodes_stats.episodes_tv_series) and
	    (users_stats.users_gender = episodes_stats.users_gender)
)
SELECT * FROM result;

 episodes_tv_series | users_gender | total_users |     average_age     | total_clv | total_streams | total_streams_minutes | total_episodes | avg_rating
--------------------+--------------+-------------+---------------------+-----------+---------------+-----------------------+----------------+------------
 Black Mirror       | female       |           1 | 19.0000000000000000 |     150.3 |             1 |                    41 |              1 |        8.7
 Black Mirror       | male         |           2 | 29.0000000000000000 |     514.5 |             3 |                    80 |              2 |        8.3
 Breaking Bad       | female       |           3 | 44.6666666666666667 |     618.9 |             4 |                    83 |              2 |       9.05
 Breaking Bad       | male         |           3 | 33.3333333333333333 |     801.8 |             3 |                   108 |              2 |       8.95
 The Wire           | female       |           2 | 57.5000000000000000 |     468.6 |             2 |                    88 |              2 |        8.6
 The Wire           | male         |           1 | 42.0000000000000000 |     287.3 |             2 |                    89 |              2 |        8.6

Tada! We have everything we asked for :-) (Everything was double checked for correctness)
Sketch of an algorithm
This should be already evident, but in order to wrap up:

Do the full join requested with all the filters added in the WHERE clause, so that the base result set is as small as possible. Call it base_join

In order to be on the safe side, just select all attributes from all tables and alias each one from table_name.attribute_name to table_name__attribute_name.
Bonus: In reality, we only need the primary keys, and the attributes used in the followup steps for calculating aggregates, grouping and filtering in the end with a HAVING clause.

For each table MyTable with an aggregate requested:

Generate a base_mytable selecting all rows from base_join with:

a DISTINCT ON() clause over the primary key(s) of table MyTable and all the attributes that will be used for GROUP BYs. (properly reference the attributes using the table_name__attribute_name rule of (2))

select all the attributes (from all tables) that are used for GROUP BYs

add all attributes from MyTable that are required for aggregates

As an example, if have been asked to group by attributes product__name and store__region and we want to compute the total tax in table sales:
with base_join (
  ...
),
base_sales as (
  SELECT DISTINCT ON (sales__id, product__name, store__region)
         product__name, store__region, sales__tax
  FROM base_join
)

Generate a mytable_stats that will group by over the base_mytable and compute the proper aggregates:

Select clause: add all the the attributes (from all tables) that are used for GROUP BYs and the required Aggregates

I propose a naming convention of adding the aggregate name in front of the attribute name, e.g. sum__table_name__attribute_name or avg__table_name__attribute_name

Add all the the attributes that will be used for GROUP BYs in GROUP BY and ORDER BY clauses in the end.

As an example, if have been asked to group by attributes product__name and store__region and we want to compute the total tax in table sales:
with base_join (
  ...
),
base_sales as (
  SELECT DISTINCT ON (sales__id, product__name, store__region)
         product__name, store__region, sales__tax
  FROM base_join
),
sales_stats as (
  SELECT product__name, store__region, SUM(sales__tax) as sum__sales__tax
  FROM base_sales
  GROUP BY product__name, store__region
  ORDER BY product__name, store__region
)

After all tables are covered, add the final query that will:

join all the mytable_stats tables on all the attributes that are used for GROUP BYs (they serve as the keys binding the results together)
select all the attributes that are used for GROUP BYs
select all the aggregate attributes generated in the previous steps
profit!

bonus: also add any HAVING clause relevant to each mytable_stats or add them altogether in the final query (less optimal, but everything else is taking so much memory that this is the cherry on top of the cake). HAVING clauses that involve aggregates from multiple tables at the same expression must be added in the final query, after joining all the mytable_stats tables.
/cc @jschatz1 @mbergeron @bencodezen @derek-knox @cpetren
Please check and tell me if something feels not right or whether I missed something.

View 12 previous comments from the original issue on GitLab


WebsiteExistsException

WebsiteExistsException is thrown after deleting a website and recreating it again.
Steps to reproduce:

create a new website with identifier e.g. "testsite"
delete the site in the config menu
create the same website again will resul in a short message "Website exists." with following log entry:

OCA\CMSPico\Exceptions\WebsiteExistsException:
/var/www/html/custom_apps/cms_pico/lib/Controller/WebsitesController.php - line 121:

OCA\CMSPico\Service\WebsitesService->createWebsite(OCA\CMSPico\Model\Website {}, "sample_pico")
....

Server configuration detail
Operating system: Linux 5.4.0-100-generic 113-Ubuntu SMP Thu Feb 3 18:43:29 UTC 2022 x86_64
Webserver: nginx/1.21.3 (fpm-fcgi)
Database: mysql 10.2.14
PHP version: 8.0.11
Modules loaded: Core, date, libxml, openssl, pcre, sqlite3, zlib, ctype, curl, dom, fileinfo, filter, ftp, hash, iconv, json, mbstring, SPL, PDO, pdo_sqlite, session, posix, readline, Reflection, standard, SimpleXML, Phar, tokenizer, xml, xmlreader, xmlwriter, mysqlnd, cgi-fcgi, apcu, bcmath, exif, gd, gmp, imagick, intl, ldap, memcached, pcntl, pdo_mysql, pdo_pgsql, redis, sodium, zip, Zend OPcache
Nextcloud version: 22.2.0 - <IP_ADDRESS>

Works for me. Looks like something went wrong when deleting the website before, so please use phpMyAdmin (or similar) to manually delete the website from the oc_cms_pico_websites table. Use the site column to properly identify the website you want to delete. Please keep in mind that website identifiers are global, i.e. another user using the same identifier will prevent you from using the same identifier.

@PhrozenByte many thanks for your fast reply!
For Nextcloud install its the database nextcloud with table cms_pico_websites. After deleting the entries manually it works now. However, in the table I also found old sites from already deleted users. Maybe an autodelete can be triggered after deleting the user to prevent  existing site entries that are only accessible via cmdline or phpmyadmin.
Thanks all and a good start into the week

Can't reproduce this either: When deleting an user all user websites are deleted, too.

Thanks all and a good start into the week

Dito :v:


Improve type inference precision for Enumerable.Concat.

fixes #44

Coverage remained the same at 99.599% when pulling 05b4ca661e96a141395f8437e08d28cf71b29d39 on fsoikin:issue/44 into dfd6bd300bf79f8601d59659d9cdc4fe9e1d80b4 on ENikS:master.

Coverage remained the same at 99.599% when pulling 671fd50c91d95cdf065ee0b335ef7c2b2cc443db on fsoikin:issue/44 into 17361e95e18acf4f08198784174ed90dc1261059 on ENikS:master.

Yes, you're right about the typo.
But implementation doesn't really need to be changed, because you're losing type safety there anyway by taking an untyped factory and passing untyped arg to it. Types don't matter there. (I changed it anyway though).
This is something I was going to note as well, by the way. You don't have to go untyped, you could pass in a typed closure instead of both arg and factory:
class EnumerableImpl<T> implements Enumerable<T>, Iterable<T>, IEnumerable<T> {
    protected _factory: () => Iterable<T>;

    constructor(factory: () => Iterable<T>) {
        this._factory = factory;
        ...
    }

    ...

    public SelectMany<S, V>(
       selector: (x: T, index: number) => Iterable<S> = Constant.selfFn,
       result: (x: T, s: S) => V = (x, s) => s as any as V):
       Enumerable<V>
    {
        // The compiler ensures that Generator.SelectMany has the right arguments and returns the right type.
        // A whole class of bugs gone. The whole point of static typing.
        return new EnumerableImpl<V>(() => Generator.SelectMany( this, selector, result ) );
    }

Thank you for contributing


[SPARK-23908][SQL] Add transform function.

What changes were proposed in this pull request?
This pr adds transform function which transforms elements in an array using the function.
Optionally we can take the index of each element as the second argument.
> SELECT transform(array(1, 2, 3), x -> x + 1);
 array(2, 3, 4)
> SELECT transform(array(1, 2, 3), (x, i) -> x + i);
 array(1, 3, 5)

How was this patch tested?
Added tests.

@ueshin, thanks! I am a bot who has found some folks who might be able to help with the review:@rxin, @cloud-fan and @hvanhovell

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1603/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1615/
Test PASSed.

Test build #93950 has started for PR 21954 at commit c3bf6a0.

Test build #93934 has finished for PR 21954 at commit ee450c5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93934/
Test FAILed.

Test build #93950 has finished for PR 21954 at commit c3bf6a0.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class ResolveHigherOrderFunctions(catalog: SessionCatalog) extends Rule[LogicalPlan]
                  s\"its class is $
case class ResolveLambdaVariables(conf: SQLConf) extends Rule[LogicalPlan]
case class NamedLambdaVariable(
case class LambdaFunction(
trait HigherOrderFunction extends Expression
trait ArrayBasedHigherOrderFunction extends HigherOrderFunction with ExpectsInputTypes
case class ArrayTransform(

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93950/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1633/
Test PASSed.

Test build #93973 has started for PR 21954 at commit c3bf6a0.

Test build #93973 has finished for PR 21954 at commit c3bf6a0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class ResolveHigherOrderFunctions(catalog: SessionCatalog) extends Rule[LogicalPlan]
                  s\"its class is $
case class ResolveLambdaVariables(conf: SQLConf) extends Rule[LogicalPlan]
case class NamedLambdaVariable(
case class LambdaFunction(
trait HigherOrderFunction extends Expression
trait ArrayBasedHigherOrderFunction extends HigherOrderFunction with ExpectsInputTypes
case class ArrayTransform(

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93973/
Test FAILed.

Jenkins, retest this please.

Test build #94002 has started for PR 21954 at commit c3bf6a0.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1659/
Test PASSed.

Test build #94002 has finished for PR 21954 at commit c3bf6a0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class ResolveHigherOrderFunctions(catalog: SessionCatalog) extends Rule[LogicalPlan]
                  s\"its class is $
case class ResolveLambdaVariables(conf: SQLConf) extends Rule[LogicalPlan]
case class NamedLambdaVariable(
case class LambdaFunction(
trait HigherOrderFunction extends Expression
trait ArrayBasedHigherOrderFunction extends HigherOrderFunction with ExpectsInputTypes
case class ArrayTransform(

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94002/
Test FAILed.

retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1671/
Test PASSed.

Test build #94019 has started for PR 21954 at commit c3bf6a0.

Test build #94019 has finished for PR 21954 at commit c3bf6a0.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class ResolveHigherOrderFunctions(catalog: SessionCatalog) extends Rule[LogicalPlan]
                  s\"its class is $
case class ResolveLambdaVariables(conf: SQLConf) extends Rule[LogicalPlan]
case class NamedLambdaVariable(
case class LambdaFunction(
trait HigherOrderFunction extends Expression
trait ArrayBasedHigherOrderFunction extends HigherOrderFunction with ExpectsInputTypes
case class ArrayTransform(

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94019/
Test PASSed.


Add scalafmt config (based on cats)

@tpolecat do you have any suggestions for this?

@tpolecat do you have a config I could use instead of this? I was basing on cats, but if I remember correctly you didn't like what Cats does ;)


[Z0][VNM][E]: No VN_MAD in template for Virtual Network.

My setup is as follows:

Frontend node (control1), holds sunstone, mysql, etc
Compute node (compute1), host for VM's

When creating a new user (testuser) I see the following errors in /var/log/one/oned.log:
Sat Jan 28 22:54:09 2017 [Z0][ReM][D]: Req:6384 UID:0 VirtualNetworkAllocate invoked , "NAME="12-Private-Net...", -1 Sat Jan 28 22:54:09 2017 [Z0][VNM][E]: No VN_MAD in template for Virtual Network. Sat Jan 28 22:54:09 2017 [Z0][ReM][E]: Req:6384 UID:0 VirtualNetworkAllocate result FAILURE [VirtualNetworkAllocate] No VN_MAD in template for Virtual Network.
This causes the rest of the process to fail. I have looked into similar issues with OpenNebula like this, but it appears as though issues on the platform were already fixed in the latest version. Is there a way for me to fix this issue? It seems rather minor in the grand scheme of things.

solved by adding VN_MAD=ovswitch to the templates located in /var/lib/one/remotes/hooks/eywa


[FEATURE]

Description
dependant on  #268
add an embeded video (use any video for now)
this can help
https://www.youtube.com/watch?v=LH7LPMXv8Lg
Screenshots
No response

PR #274 implemented this feature with video section page


Allow Link components without pathname

Hi,
I have a feature request concerning <Link> API. My typical use case is that I want to have buttons to pass query parameters, but stay on the same page. Then, I was thinking of doing something like
<Link to={{ query: { page: 1 } }}

But it does not work because Link component expects to have a pathname value.
My problem is then that I have to do something like
<Link to={{ pathname: currentPath, query: { page: 1 } }}

but that I have to fetch the currentPath from somewhere. If I was in a Route component, I could just fetch this.props.location.pathname, but this is not my case, and I am in an included components, some levels deep.
I thought about using the context injection, but the injected object does not offer any possibility to retrieve current pathname apparently.
So, my feature request is to either allow a Link without pathname, and then defaults the pathname to the current pathname, or provide a way to access current pathname outside of props. Indeed, as of now, the only solution I could find is to pass the location props down to every component, which is quite verbose, especially since I only need it in a single component downwards.
Thanks!

You spread in the old location – { ...location, query: newQuery }. If we go ahead with https://github.com/reactjs/react-router/issues/3657 it will be even easier.
We don't really want to do this at the level of <Link> like this directly because the object spread syntax is just as terse but gives you more direct control over how you build the new location.

Ok for spreading the old location, but this means I must pass location as a prop down the component tree, no?
Sorry if I missed something :/

See #3657 for a proposal on sidestepping that requirement.


Number Extension Suggestion

I am planning to add a extension which has following features :

BigInt (you can do +-*/% and everything that BigInt supports.)
Decimal.js (custom sig fig, numbers > 1.79e308, maybe can replace BigInt too)
other JS Math built-in func and const (pi, e, phi, omega, sqrt2, sqrt 0.5, max safe int etc)

@Skyhigh173 Most of this is exist in Math.js, and I'm already doing new extension for Bigint https://turbowarp.org/editor?extension=https://true-fantom.github.io/tests/bigint.js

@True-Fantom I will do my own one with more features.

@Skyhigh173 What I showed you is a prototype, I'm actively working on it. And if you are talking about Mathematical functions that are not in Math, they will also be added soon. So it's not necessary.

@Skyhigh173 Can you please list all the functions that you lack so that I can add everything for sure.

@Skyhigh173 One more question. Are there any other useful constants besides these?

@True-Fantom why not make it a menu? (You can check JS Math built-in const)

@Skyhigh173

@True-Fantom You can check JS Math built-in const

I've already checked. Of the useful ones there are only pi and e, which is why I decided to ask you if you know any other constants (for example, I didn't know about the existence of omega at all)

@True-Fantom why not make it a menu?

To do this, you would have to remove existing blocks from the palette, and it would also take up more space (And also the menu would not be very useful if I left special characters, and them greatly shorten the length of blocks)

Could you add support for complex numbers?
or the factorial operation x!

@penta-quark-neutro Yes, I'm going to add a factorial in the next Math update, but this is the first time I've heard about complex numbers.. What do they do and what blocks should be added for them?

The complex numbers are the superior set to the real numbers that we use all the time (basically R∈C).
Currently I do not have my computer at hand, and there is a way to calculate them.
But basically a whole new interface is going to be created for these numbers since they are a composition (a+bi). Adding, subtracting, and multiplying them is easy, but it can be a bit more difficult with things like their logarithms and roots.
Basically the imaginary number i=root of -1, its simplest definition.

The complex numbers are the superior set to the real numbers that we use all the time (basically R∈C).
Currently I do not have my computer at hand, and there is a way to calculate them.
But basically a whole new interface is going to be created for these numbers since they are a composition (a+bi). Adding, subtracting, and multiplying them is easy, but it can be a bit more difficult with things like their logarithms and roots.
Basically the imaginary number i=root of -1, its simplest definition.

I have already done a complex number extension
But it needs testing

Since I don't know how to create extensions, I did this to calculate complex numbers.
it is not practical since round reporters are better and those blocks are not practical.
I also could not define the complex factorial.


FormBuilderDateTimePicker format input AM/PM

Hi, FormBuilderDateTimePicker set format "h:mm a",
Formatted date works fine.
Graphic control does not display the correct format.
return FormBuilderDateTimePicker(
      alwaysUse24HourFormat: false,
      attribute: 'date',
      onChanged: (value) {
        print(value);
      },
      format: DateFormat("h:mm a"),
      inputType: InputType.time,
      initialTime: TimeOfDay(hour: 8, minute: 0),
    );

To switch between 12 and 24-hour modes in TimePicker, all you need to do is toggle alwaysUse24HourFormat attribute. The format attribute is used only within the TextField.


Wait more for newlog and vlans

Tests for RPi (ROL) indicate that our timeouts for newlog and vlans do not cover real workloads (for slowest case with RPi running xen and zfs).

Previously, in the switch_net_vlans test on rpi4, an error was very likely to occur, now 10 out of 10 switch_net_vlans tests pass without problems.


first draft of pheno/qc matching

Would be great to get your feedback on this @htwangtw! I also have a notebook to look at the qc results with a few figures, should I push it or will that be more project specific?

Let's put it here, since we will be combining the QC data eventually :)

I need to make some changes to this so reverted to draft PR


Continue sync where last stopped

Use case
Would it be possible to implement a way for the application to restart sync where it last stopped?
I have 60 000+ files and one complete sync pass takes days (detecting and skipping one file consumes about 2 seconds)
I suppose it needs to be solution based on looping around a sequence of filters to chop up the complete sync task.
Maybe then restart sync using the last filter.

Interaction with iCloud is polling based. The app needs to traverse the directory tree from its root to leaves to detect changes. Resuming the sync from where it stopped (due to error, timeout etc.) will only help with that iteration of tree traversal. It will not speed up next iteration as it must traverse the whole tree to detect changes.
However, current approach is serial - process one path and then move on to the next path. I am working on next-gen of this app which will be utilizing concurrency to significantly improve the performance.

Hi, that would be awesome and worth waiting for!
Would it be possible to then also implement a "one time" sync option suitable for cron-jobs?

One time sync is currently available. Just set sync_interval to -1 to exit after sync.

Cool, thanks!

Thank you

@mandarons is there a branch available for the work going into concurrency?

Concurrency work hasn't started yet for this project. I am currently working on making the library that is used by this app, concurrent aioicloud.


⚠ Thumbnails API Endpoint has degraded performance

In df50525, Thumbnails API Endpoint (https://thumbnails.roblox.com/v1/assets?assetIds=82738847&format=Png&isCircular=false&size=30x30) experienced degraded performance:

HTTP code: 200
Response time: 5124 ms

Resolved: Thumbnails API Endpoint performance has improved in cc35498 after 11 minutes.


Upgrade Tauri to latest version

I considered downgrading zstd version in a fork, but the simpler solution for now is to just disable compression.
Compression wasn't there before in Tauri (at least there was no such feature) and Debian package size got much smaller after upgrade anyway (30.2M -> 22.7M).

Please test if this actually works, I only checked that it compiles on Linux.

For quasar we might need Leo's help

We need to update a few settings and security checks on tauri.conf.json. I have found at least these two as critical.
TO FIX fs scope
Solution:

The scope error is related to this new configurations. Adding scope will be able to read and write files and dirs on it. On Linux at least I was getting similar errors and were solved by adding this  to tauri > allowlist > fs > scope config.

Two problems:

The user will be limited to this scope, this will be a problem with selecting non-HOME plot directory, like another partition, external drives, etc.
Also, this configuration will be applied on the build and looks we can not update the scope after, still checking if there is another way to set this or maybe update the scope on user selection. I still can't find a solution to this.

@ozgunozerk  can you try adding this on tauri.conf.json
          "fs": {         "all": true,         "scope": ["$HOME/*"]       }
TO FIX Shell scope:
Looks we need to apply some confs also, but I can not test in here. @ozgunozerk this will probably solve the applescript execution problems. Just need to find the correct configuration.
https://tauri.studio/docs/api/js/modules/shell and https://tauri.studio/docs/api/js/modules/shell#security

Also check #90, will be fixed automatically if we get this upgrade running.

tauri action v0.3 creates problem. I believe it is not yet stable (also it is not latest in the release section of their repo). Switching back to v0.2 for now.

with march 03-23 update on rust nightly, ring create is creating problems on the building phase.

error: no rules expected the token `aarch64_apple`
   --> /Users/ozgunozerk/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.20/src/cpu.rs:257:13
    |
165 |     macro_rules! features {
    |     --------------------- when calling this macro
...
257 |             aarch64_apple: true,
    |             ^^^^^^^^^^^^^ no rules expected this token in macro call
error[E0425]: cannot find value AES in module cpu::arm
--> /Users/ozgunozerk/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.20/src/aead/aes.rs:381:65
|
381 |         if cpu::intel::AES.available(cpu_features) || cpu::arm::AES.available(cpu_features) {
|                                                                 ^^^ not found in cpu::arm
|
help: consider importing this constant
|
15  | use crate::cpu::intel::AES;
|
error[E0425]: cannot find value PMULL in module cpu::arm
--> /Users/ozgunozerk/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.20/src/aead/gcm.rs:315:26
|
315 |             || cpu::arm::PMULL.available(cpu_features)
|                          ^^^^^ not found in cpu::arm
error[E0425]: cannot find value ARMCAP_STATIC in this scope
--> /Users/ozgunozerk/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.20/src/cpu.rs:235:41
|
235 |             if self.mask == self.mask & ARMCAP_STATIC {
|                                         ^^^^^^^^^^^^^ not found in this scope
error[E0425]: cannot find value ARMCAP_STATIC in this scope
--> /Users/ozgunozerk/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.20/src/cpu.rs:288:36
|
288 |     static mut GFp_armcap_P: u32 = ARMCAP_STATIC;
|                                    ^^^^^^^^^^^^^ not found in this scope

Here is a report in:

ring: https://github.com/briansmith/ring/issues/1469
rust: https://github.com/rust-lang/rust/issues/95267

Thanks Nazar, I've opened a issue about ring, I will keep an eye on these threads you provided and update to nightly when this issue is resolved. For now, I believe it is ok to continue with 2022-02-15 version of rust.

Tested on macos and windows, it is working


first decompression with tree worker

work in progress!

Codecov Report

Merging #39 (718d0e1) into master (151167f) will increase coverage by 0.11%.
The diff coverage is 94.59%.

@@            Coverage Diff             @@
##           master      #39      +/-   ##
==========================================
+ Coverage   92.39%   92.50%   +0.11%
==========================================
  Files          25       26       +1
  Lines        1301     1375      +74
==========================================
+ Hits         1202     1272      +70
- Misses         99      103       +4

Impacted Files
Coverage Δ

stixcore/tmtc/tm/tm_21.py
98.80% <80.00%> (-1.20%)
:arrow_down:

stixcore/tmtc/packets.py
87.27% <85.71%> (-0.11%)
:arrow_down:

stixcore/processing/decompression.py
90.90% <90.90%> (ø)

stixcore/tmtc/parser.py
95.53% <98.03%> (+2.09%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 151167f...718d0e1. Read the comment docs.

Codecov Report

Merging #39 (718d0e1) into master (151167f) will increase coverage by 0.11%.
The diff coverage is 94.59%.

@@            Coverage Diff             @@
##           master      #39      +/-   ##
==========================================
+ Coverage   92.39%   92.50%   +0.11%
==========================================
  Files          25       26       +1
  Lines        1301     1375      +74
==========================================
+ Hits         1202     1272      +70
- Misses         99      103       +4

Impacted Files
Coverage Δ

stixcore/tmtc/tm/tm_21.py
98.80% <80.00%> (-1.20%)
:arrow_down:

stixcore/tmtc/packets.py
87.27% <85.71%> (-0.11%)
:arrow_down:

stixcore/processing/decompression.py
90.90% <90.90%> (ø)

stixcore/tmtc/parser.py
95.53% <98.03%> (+2.09%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 151167f...718d0e1. Read the comment docs.

I wonder if we shouldn't flatten the packets as an intermediate step, it would avoid all that set_path, get_path, get_all etc what do you think?

I wonder if we shouldn't flatten the packets as an intermediate step, it would avoid all that set_path, get_path, get_all etc what do you think?


Implement Lindenmayer-Systems for generating vegetation, texture overlays and world geometry

Using L-Systems we can:

generate trees and other vegetation
generate 2d patterns for use as texture overlays
generate world geometry like labyrinths or even whole cities

Resources:

http://en.wikipedia.org/wiki/L-system (general introduction)
https://www.youtube.com/watch?v=dJk_2tOcqgQ (minimal showcase)
https://www.youtube.com/watch?v=pZ8NLfUJ7p0 (not so minimal showcase)
http://www.kevs3d.co.uk/dev/lsystems/ (try it out by changing the generation string)
https://10klsystems.wordpress.com/examples/ (2d patterns)
http://algorithmicbotany.org/papers/abop/abop-ch1.pdf
https://www.cg.tuwien.ac.at/courses/Fraktale/PDF/fractals8.pdf
https://www.youtube.com/watch?v=-d2-PtK4F6Y (Pixel City)

++1 very neat


No error pop-up notification when throwing an error from extension activate() function

VSCode Version: 1.47.0-insider
OS Version: Linux x64 5.4.0-39-generic

Steps to Reproduce:
Create a vscode extension with the following code:
export async function activate() {
    throw new Error("Failed to bootstrap");
}
export async function deactivate() {}

There is no error notification about the failure to activate the extension.
In fact, this is very weird, this bug is kinda floating?
I clearly remember vscode to show a popup error notification with the error message some time ago (when creating this PR: https://github.com/rust-analyzer/rust-analyzer/pull/4993). But now, things stopped working I guess (https://github.com/rust-analyzer/rust-analyzer/issues/5091#issuecomment-650800525)? Now when you throw an error in activate() function nothing is displayed in the UI, but if you open the Developer Tools console, you will see the message there.
This is very strange, I don't know why it worked before and why it doesn't work now. I even remember it not working before I was writing the PR (and at the point of time when writing the PR it was suddenly working?!...).
Anyway, could you guys confirm this is a bug or not, and/or document the behaviour of throwing an error from activate() function?
I haven't seen any API to fail or cancel the extension activation (i.e. the user has misconfigured something and our extension cannot simply function for this reason so we would like to cancel the extension activation) and even after throwing an error in activate() the user can still use the commands registered in our extension package.json which doesn't seem right...

Does this issue occur when all extensions are disabled?: yup

Relates to #100856

This is expected. It all depends if you are debugging the extension or not.
When throwing from activate() and doing extension development (via F5), we show a notification. This is because this might be very relevant to the developer building the extension.
When throwing from activate() from a user installed extension (and running vscode normally), we do not show a notification. This is intentional. The decision has been to "eat up" this error and only log it to the log and to the developer tools in order to not appear that vscode is broken due to a misbehaving extension.


Remove log.debug in cache and writer

The cache and writer don't support log.debug, use log.msg instead. Fixes #401.

@jssjr I think it would make sense long-term to have a working log.debug() in master, but I don't think any of these changes mandate its use. Merging this in.


PD client should handle errors properly

Bug Report

What version of TiKV are you using?

v4.0.15
What operating system and CPU are you using?

doesn't matter
Steps to reproduce

Once PD client meets an error, it will reconnect and retry the request even if the request is invalid. The disconnection may result in other requests failure, e.g., region heartbeat. I saw lots of logs below. I suspect the disconnection makes region heartbeat fail and PD contains stale region information for a long time.
[2022/04/06 19:03:05.883 +08:00] [ERROR] [util.rs:348] ["request failed, retry"] [err_code=KV:Unknown] [err="Grpc(RpcFailure(RpcStatus { status: 2-UNKNOWN, details: Some(\"region not found, request region: id:56125222 start_key:\\\"7480000000014B7AFF3500000000000000F8\\\" region_epoch:<conf_ver:288104 version:4789963 > peers:<id:175253496 store_id:161941895 > peers:<id:175254079 store_id:161941897 > peers:<id:175443528 store_id:155883000 >\") }))"]
[2022/04/06 19:03:05.883 +08:00] [INFO] [util.rs:470] ["connecting to PD endpoint"] [endpoints=http://<IP_ADDRESS>:2379]
[2022/04/06 19:03:05.884 +08:00] [INFO] [<unknown>] ["New connected subchannel at 0x7fa29ebb7e90 for subchannel 0x7f99d359f2c0"]
[2022/04/06 19:03:05.885 +08:00] [INFO] [util.rs:470] ["connecting to PD endpoint"] [endpoints=http://<IP_ADDRESS>:2379]
[2022/04/06 19:03:05.886 +08:00] [INFO] [util.rs:533] ["connected to PD leader"] [endpoints=http://<IP_ADDRESS>:2379]
[2022/04/06 19:03:05.886 +08:00] [INFO] [util.rs:231] ["heartbeat sender and receiver are stale, refreshing ..."]
[2022/04/06 19:03:05.997 +08:00] [WARN] [util.rs:249] ["updating PD client done"] [spend=114.438081ms]

What did you expect?
PD client doesn't reconnect and retry invalid requests.
What did happened?
PD client does reconnect and retry.

@nolouch PTAL

/severity Major

ptal @HunDunDM

This is partial fixed by https://github.com/tikv/tikv/pull/12629 which avoids reconnecting on some error.
Encounter the same issue, the RegionNotFound or EpochIsStale returned by PD trigger mistakenly reconnection of the PD client as well.


feat: Onboard San francisco 311 service requests

Checklist
Note: Delete items below that aren't applicable to your pull request.

[x] Please merge this PR for me once it is approved.
[x] If this PR adds or edits a dataset or pipeline, it was reviewed and approved by the Google Cloud Public Datasets team beforehand.
[x] If this PR adds or edits a dataset or pipeline, I put all my code inside  datasets/san_francisco_311_service_requests> and nothing outside of that directory.
[x] This PR is appropriately labeled.

@adlersantos I have completed changes as directed.  Please review.  Thanks!


Support synchronising global state

Support synchronising global state

locale

@dbaeumer How feasible it is to move locale configuration to settings?

The biggest problem moving it to settings is performance. It is its own file since we need to read that file early on in the startup process and we didn't want to read large files and parse them. So would we be able to sync it separately ?

You are right and it might impact the performance. If so, lets not move it to settings and I can sync locale separately.

Thanks!

There are two issues I think are relevant to this:
1. Sessions across machines: You might want to think about how you want to handle session restores across different machines that doesn't have access to the same files, perhaps by design (e.g. work machine, home machine). I encountered this issue a while back in the portable version of VSCode, which actually prompted me to stop using the portable version: https://github.com/microsoft/vscode/issues/81340
2. Settings across machines: This is also part of a bigger problem of User/settings.json not being able to import other settings files. Ideally you should be able to set up a hierarchy of settings:

The global settings that applies to all machines
Work/Home machine specific settings that only applies to work machines or personal machines.
Project specific settings (e.g. .vscode/settings.json in project folders).
This issue is being tracked here, and I can see you're already aware of it :) https://github.com/microsoft/vscode/issues/17634

Sessions across machines:

Can you please elaborate more about the issue?

I'm not 100% sure how this is relevant to this issue since not many details are provided. Perhap you could elaborate beyond the title "Support synchronising global state" what this means?
I just meant everything I said in https://github.com/microsoft/vscode/issues/81340 - but I can re-iterate it here if necessary. If you're looking for VSCode to restore editor session across different machines, as the portable version inherently does, you're going to have potentially not access to the same files. So you get errors that you cannot open file X, Y and Z when you open VSCode on another machine.
For example, I'm on machine A and have files X, Y and Z open.
Then I start machine B and only files X are available, but Y and Z are not (e.g. some Cloud app not installed). I will then get errors that these files are not available on machine B. If I then continue to work, and restart machine A, it might lose files Y and Z being open because they were not part of the latest session on machine A.

I see your use case. We have plans to sync open editors state but definitely not in scope of this issue.

Fair enough 🙂So though the title of this issues says "Support synchronising global state", your inital post seem to suggest this is just about the locale setting for now (e.g. country + language)?

Yeah currently we only support syncing display language as part of global state. Planning to extend it to more global state like activity bar icons, views visibilities etc.,

Makes sense to add the basic functionality first 👍
I hope that somewhere down the line that grouping of session will be considered at some point (Group A will share these states, Group B will share these states, all groups will share these states etc.).

Syncing global state (core) is supported now. Next step is to let state owners to opt in to sync the state - #93332

@sandy081 how will extensions opt-in or out of this?

Currently it is only supported for core. Planning to extend this to extensions in next milestone.


 This causes a lot of network lag on servers. These messages should only be sent to closeby players

On that note, why are you not using normal blockentity client sync for this?

Yeah this was brought to my attention a few days ago, I'm currently working on fixing it.

It should be good now in version 1.3.2


Meeting minutes

This is placeholder for all the meeting minutes with GLEIF-NordLEI

Meeting: NordLEI: QVI Credential Revocation/Issuance | Participants: Kevin Griffin (@m00sey); Kent Bull; Maria Karjalainen; Daniel Lenksjö (@lenkan); David de Meijere (@daviddm); Arti Kulkarni (@artikulkarni) | Date - 18.Dec.2024 15:30 - 16:30
1. Proposed Business Split - NordLEI plans to separate the LOU (LEI) and QVI (vLEI) business, with new entity Global vLEI AB, taking over as designated candidate QVI/vLEI Business
2. Key steps to get the vLEI QVI credentials to candidate QVI Global vLEI AB

Expected timeline from NordLEI to get the vLEI QVI credential issuance to 'Global vLEI AB' is 20th Jan 2025 considering upcoming EBA initiatives.

Technical Dependencies

Global vLEI AB must ensure that its software is ready for the technical dry run. This software will be based on KERIA version 0.2.0.

The Issues reported by NordLEI doesn't have dependency for the dry run software and once all the issues are resolved the KERIA can be marked as the next version (0.3.0)

Currently there is no stable KERIA version recommended by GLEIF, only a soft recommendation.

[ ] Kevin (@m00sey) to tag the current KERIA version (build: bump to 0.2.0 in GLEIF's repository by 20.Dec.2024)

Qualification

Is the resubmission of the documents required?
Tentative timeline from the Qualification team?
[ ] Arti to redirect these questions to Qualification team to share the feedback with NordLEI (Maria)

Steps during the technical dry run

Revocation While revocation is not mandatory (due to the grace period), it is recommended as a clean step to ensure the revocation of the chained LE/ECR credentials for banks
NordLEI's current infrastructure can be maintained for up to 90 days (accounting for the vLEI EGF grace period) before decommissioning.

Technical Dry Run Playbook Steps

[ ] Kevin (@m00sey) to share the playbook template with David and Daniel.


Minimized files ignored in JS projects using jsconfig.json "path"

Bug Report

🔎 Search Terms

safe list
safelist
minimized file
.min.js

🕗 Version & Regression Information
TS Server 5.0.4 used in VSCode 1.78.1

⏯ Playground Link

N/A -- only reproducible in projects
💻 Code

https://github.com/rwalle/ts_bug_example
import _ from 'lodash/lodash.min.js';

Our project does not use npm module system. jsconfig.json is correctly set up: https://github.com/rwalle/ts_bug_example/blob/main/dev/apps/app1/jsconfig.json. We import from file directly instead of using package name.
🙁 Actual behavior
lodash.min.js ignored in module import autocompletion suggestion

It seems that all these files, plus certain jQuery files etc:
https://github.com/microsoft/TypeScript/blob/f0ff97611f2e9c8aff208f4b6520489fe387e9ab/src/server/editorServices.ts#L387
https://github.com/microsoft/TypeScript/blob/f0ff97611f2e9c8aff208f4b6520489fe387e9ab/src/server/editorServices.ts#L4112
I can understand why this is done for "regular" JS projects using npm modules or TS projects with or without @types, but this should not be generalized when looking up a file/files in arbitrary JS projects, especially when various options in compilerOptions of jsconfig.json is meant to specifically address such use cases.
🙂 Expected behavior
lodash.min.js should show up just like anotherFile.js.

To be clear, the only bug here is that it doesn't show up in the import auto complete list?

To be clear, the only bug here is that it doesn't show up in the import auto complete list?

Yes, that is correct

To be clear, the only bug here is that it doesn't show up in the import auto complete list?

It has been a few days since the last update, I want to follow up and check if this should be tracked as a bug? Thank you.

I would call this the intended behavior. We intentionally filter out .min files from this list because usually that's not what you want to see, and there are often a bunch of .min files and only one correct non-.min import. This is the first I've seen where this behavior wasn't desired, so unless this becomes a more common scenario for some reason, I think we'll leave this as-is.


docs: add runv get started doc

Signed-off-by: Ace-Tang<EMAIL_ADDRESS>
1.Describe what this PR did
2.Does this pull request fix one issue?

fixes #202
3.Describe how you did it
4.Describe how to verify it
5.Special notes for reviews

Is issue https://github.com/alibaba/pouch/issues/219 fixed now?

fix #219  #202

When I use container 1.0, it fails and returns such info:
[root@xx /etc/containerd]
#pouch create xxxxxxx/busybox:latest
container ID: 9afae0ada7742099e283930b6352b40efe769af810ed7416f8db35b0e10aea8b, name: 9afae0

[root@xx /etc/containerd]
#pouch start -i 9afae0
Error: failed to start container 9afae0: {"message":"failed to create task, container id: 9afae0ada7742099e283930b6352b40efe769af810ed7416f8db35b0e10aea8b: OCI runtime create failed: unable to retrieve OCI runtime error (open /var/lib/pouch/containerd/state/io.containerd.runtime.v1.linux/default/9afae0ada7742099e283930b6352b40efe769af810ed7416f8db35b0e10aea8b/log.json: no such file or directory): runv did not terminate sucessfully: Unsupported driver \"\"\n\nNAME:\n   runv create - create a container\n\nUSAGE:\n   runv create [command options] <container-id>\n\nWhere \"<container-id>\" is your name for the instance of the container that you\nare creating. The name you provide for the container instance must be unique on\nyour host.\n\nDESCRIPTION:\n   The create command creates an instance of a container for a bundle. The bundle\nis a directory with a specification file named \"config.json\" and a root\nfilesystem.\n\nThe specification file includes an args parameter. The args parameter is used\nto specify command(s) that get run when the container is started. To change the\ncommand(s) that get executed on start, edit the args parameter of the spec. See\n\"runv spec --help\" for more explanation.\n\nOPTIONS:\n   --bundle value, -b value  path to the root of the bundle directory, defaults to the current directory (default: \"/var/lib/pouch/containerd/state/io.containerd.runtime.v1.linux/default/9afae0ada7742099e283930b6352b40efe769af810ed7416f8db35b0e10aea8b\")\n   --console value           specify the pty slave path for use with the container\n   --console-socket value    specify the unix socket for sending the pty master back\n   --pid-file value          specify the file to write the process id to\n   --no-pivot                [ignore on runv] do not use pivot root to jail process inside rootfs.  This should be used whenever the rootfs is on top of a ramdisk\n   \nERROR: logging before flag.Parse: E1211 14:01:16.486300    6850 main.go:149] app.Run(os.Args) failed with err: &errors.errorString{s:\"Unsupported driver \\\"\\\"\"}\nUnsupported driver \"\"\n: unknown"}

Can you help find something clue in the error message?  @Ace-Tang

have you install qemu？ @allencloud

I have opened another PR to do this. #285
And I included your commit. @Ace-Tang
So I think we could close this. Please feel free to tell me if I missed anything.


How to limit start month and end month?

How can I limit the months, not to became an infinite scroll?
For example, I want to allow the user to access just 12 months before and after a given date.
Thanks!

It would be great to have this option :)

I'm working to add this option, will make a PR after :)

Done :)

Worked! Thanks!
@Eimji, have you already experienced a problem when you swipe the calendar, and sometimes it jumps back to the month you were at? (it was happening before your fix, don't worry ;) )


Remove unnecessary setting of default template

What
Remove the setting of the slimmer template to gem_layout in the roadmap and help controller.
Why
The default template is set to gem_layout in the application controller, so it doesn't need to be set again.
Visual changes
None.
Diffs:

Page
Differences

/help

/help/ab-testing

/help/cookies

/roadmap

⚠ This repo is Continuously Deployed: make sure you follow the guidance ⚠

@hannalaakso, @patrickpatrickpatrick, @matthillco Given it's been open for 7 months, would any of you be able to review and move this forward or close the PR?
CC: @chao-xian

I'm unclear why the Heroku review app is 404ing atm... it otherwise looks fine :/

This seems fine to me, I guess it would be good if we could redeploy to Heroku though just to double check.


Update cloudwatch, dynamodb, kinesis, s3, sns, ... to 2.20.45

About this PR
📦 Updates

software.amazon.awssdk:cloudwatch
software.amazon.awssdk:dynamodb
software.amazon.awssdk:kinesis
software.amazon.awssdk:s3
software.amazon.awssdk:sns
software.amazon.awssdk:sqs

from 2.20.11 to 2.20.45
Usage
✅ Please merge!
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

⚙ Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "software.amazon.awssdk" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "30 days" },
  dependency = { groupId = "software.amazon.awssdk" }
}]

labels: library-update, early-semver-patch, semver-spec-patch, commit-count:1

Superseded by #1021.


Support executing shell commands

After :s probably my most used command is :! .

Also, :read !cmd, :write !cmd, and :%!cmd.

:+1:

:+1:  on this.  Even without a file open, I'd love to click on a location in the tree, type ':' to get an ex-mode line, then be able to run ! in my tree location.

+1

+1

:+1:


Improving internal error API

The main improvement is that log_issue and raise_error are now uniform, and more flexible. The first argument is always a "position" which is anything from which we can obtain a range (e.g. an actual range, a term, an environment, etc). The last arg is the message, which can be a string or a list of documents. The variants raise_error_doc, raise_error_text, etc, are gone (except for OCaml code, but they are not used internally).
val raise_error
  (#pos_t:Type) {| hasRange pos_t |} (pos : pos_t) // A "position", of any type with a range
  (code : error_code) // An error code
  (#msg_t:_) {| is_error_message msg_t |} (msg : msg_t) // A "message", currently can be a 'string' or 'list document'
  : 'a

val log_issue
  (#pos_t:Type) {| hasRange pos_t |} (pos : pos_t) // A "position", of any type with a range
  (code : error_code) // An error code
  (#msg_t:_) {| is_error_message msg_t |} (msg : msg_t) // A "message", currently can be a 'string' or 'list document'
  : unit

Also, the Err exception is now gone. We just use Error which has an optional range already.


Contoso Chat / fixes env, roles

@cassiebreviu tested this for azdup (with 1 manual step) and if merged can then test with MI-related PR added.


NIFI-7630 Upgraded H2 dependency.

Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
Enables X functionality; fixes bug NIFI-YYYY.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[ ] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[ ] Has your PR been rebased against the latest commit within the target branch (typically master)?

[ ] Is your initial contribution a single, squashed commit? Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not squash or use --force when pushing to allow for clean monitoring of changes.

For code changes:

[ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[ ] Have you written or updated unit tests to verify your changes?
[ ] Have you verified that the full build is successful on JDK 8?
[ ] Have you verified that the full build is successful on JDK 11?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for build issues and submit an update to your PR as soon as possible.

mvn dependency:tree did not reveal any dependencies that would have L&N implications that I could see.

Let’s make sure we take an existing nifi with the old H2 Db and test dropping in the new NAR to make sure reading the old DB with the new H2 works. I’ve seen an issue before going to .200

Do you remember what that issue was?

@bbende I was getting that weird error with .198-.200, so I switched to .197 as a compromise for now.

I would have expected 199 to work, but I also haven't tested any upgrade paths for NiFi.


Issue 286/new message styling

Refactoring the new message function into a modal and applying MUI styling.
Opening this as a draft for now due to being away for a while and wanting to iron out any kinks. I'll have more time to sort through the most recent changes later, but wanted to update everyone what I'm working on right now.

Okay, still needs work (such as how clicking inside the modal opens and closes the messages in the background) and there are a couple of console errors to work through.
For me it's a bit of a toss-up between disabled and read-only, though I probably lean towards the latter. I think it should be immediately distinguishable from input that the user needs to provide, but still legible. disabled makes it a bit tougher to read than necessary, in my opinion. I'm trying a different variant for TextField for this as well.
Here it is with read-only:

And here it is with disabled (with bonus errors for Grammarly that I've never seen before):

Okay, still needs work (such as how clicking inside the modal opens and closes the messages in the background) and there are a couple of console errors to work through.
For me it's a bit of a toss-up between disabled and read-only, though I probably lean towards the latter. I think it should be immediately distinguishable from input that the user needs to provide, but still legible. disabled makes it a bit tougher to read than necessary, in my opinion. I'm trying a different variant for TextField for this as well.
Here it is with read-only:
And here it is with disabled (with bonus errors for Grammarly that I've never seen before):

read-only looks good.

When using the reply version of the modal, clicking around also opens and closes the messages in the background.
The issue is with the handleClick function on line 38 in MessagePreview.jsx.
Since that file is due for restyling anyway, we could leave this to be addressed at that point or have it be part of this PR as well.


Room header: actives members label should open the People tab

To reproduce:

Open a room, collapse the header.
Hit the "1/XX active members" label on the action bar.

It should open the People tab instead of simply the room details.

Validated in 0.6.10-dev (04df455)


refactor(modals): change file structure. update docs

Updated the documentation to reflect recent refactorings. We have also modularized the modals folder
BREAKING CHANGE: new names and paths to mixins
closes #74

:tada: This PR is included in version 4.0.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Make the new trace view better

I have two suggestions to make the new trace view better.

Add a new filter, highlight the top N duration span. This duration should exclude all child spans's duration. Calculate by UI. And notice, sometime, this may trigger negative duration,just ignore it.
Add a new filter, highlight the top N parent spans which have the max children spans.

Ref #215

@TinyAllen i will keep this open to track another feature requirement. Support broken trace segments in new trace view. OK? But please send the feature sync separated in two pull requests for main repo.

This view height should be self-adaption. @TinyAllen Could you consider optimize this?


Any more demo on the usage of UpsertStreamTableSink?

Hello,
I am currently trying to enhance Flink by completing the support of UpsertStreamTableSink & UpsertStream, but I could not find any usable examples or detailed information except for some code here.
I know that this feature is largely incomplete, however, I would like to see if there are any more practical usages or info about UpsertStreamTableSink to make sure my understanding does not deviate from the original ideas of Flink developers.
I am not sure whether it is appropriate to submit an issue here, nevertheless I find this project useful for me : )
Thank you 😄

Hi @kylemeow,
This repo contains code for a simple demo. I'm not aware of more examples for UpsertStreamTableSinks, except those uses in the Flink tests. But those are very simple.
The community is working on adding new connectors. Contributions are welcome in case you'd like to share your code.
Thanks, Fabian

Thank you for your reply, now I have finished the first step : )


Some code and paper questions about EPA

in class EPA part, these is a line:
x_SA = (attn_SA @ v_SA_projected.transpose(-2, -1)).permute(0, 3, 1, 2).reshape(B, N, C)
after read your paper and code, I think it should be:
x_SA = (attn_SA @ v_SA_projected.transpose(-2, -1)).permute(0, 2, 1, 3).reshape(B, N, C)
Though these two outputs share same shape and size, they are still different.

still in EPA part:
the way to compute x_CA in the repository is not exactly the same as it in the paper.
in the repository:
x_CA = (attn_CA @ v_CA).permute(0, 3, 1, 2).reshape(B, N, C)
while it should be this way according to the paper:
x_CA = ( v_CA.transpose(-2, -1)@attn_CA).permute(0, 2, 1, 3).reshape(B, N, C)
However, the method in the repository looks more easy to understand and logical when compared to the paper.

If I make something wrong, please let me know, thanks!

Hi @YifengDeng1 ,
Thanks for your question and apologize for the late reply.
(1) Yes, you are right. Permuting the dimension should be (0, 3, 1, 2) to keep the consistency. (2) I think both are fine. However, I applied one experiment with the proposed two modifications on Synapse and the output is almost similar (87.1 vs 87.2). I will clarify that in the next version.
Best regards,
Abdelrahman.

Hi@Amshaker,
Thanks for your reply! Still a little confusion.
(1)the dimension should be (0, 2, 1, 3), not (0, 3, 1, 2),right? (2) I totally undestand it, thanks!
Sincerely,
Yifeng.


Add turbo migrate / turbomod package for automated migrations

Describe the feature you'd like to request
We need a way to make breaking changes less painful and provide a smooth upgrade path between turbo versions.
Describe the solution you'd like
Add a new CLI package called turbo-migrate or turbomod that can hold turbo migration scripts / codemods for when we need to make breaking changes to user's monorepos, code, or turbo configurations.
At the moment, I think the package can be written in TypeScript. If we one day do need it written in Go, we can deal with it then. For now, we only need to modify package.json.

We should modelled after https://github.com/vercel/next.js/tree/canary/packages/next-codemod. We aren't using jscodeshift so that part is irrelevant for now. The parts that are good are the CLI API, the ability to select codemod and run it, throwing if git status isn't clean.

npx @turbo/migrate <transform> <path>

transform - name of transform, see available transforms below.
path - files or directory to transform
--dry Do a dry-run, no code will be edited
--print Prints the changed output for comparison
--force Force execute a codemod

We should add a page to the documentation listing what each codemod does, just like https://nextjs.org/docs/advanced-features/codemods
2 codemods should be written / can be helpful for stubs

Add packageManager key to package.json in the root and in each workspace's package.json
Move turbo config out of package.json and into a new turbo.json that lives at the root of the monorepo

Describe alternatives you've considered
N/A

Add packageManager key to package.json in the root and in each workspace's package.json

I think it's only needed in the root package.json. It might be better backwards from wherever Turbo was run until we find the root package.json.


[CAS-719] Internationalization of UI pages and log messages

Create the appropriate message bundles for the different languages. Subtasks will be created when we know what languages.
Reported by: Scott Battaglia, id: battags
Created: Fri, 17 Oct 2008 11:34:36 -0700
Updated: Fri, 17 Oct 2008 11:34:36 -0700
JIRA: https://issues.jasig.org/browse/CAS-719

Closing this issue for now as the functionality exists.


Qa

Coverage remained the same at 85.54% when pulling bf57ad7a8c9b58b848603d0bf8724e1353735169 on qa into 0e1ce4d5e315cc567297576c8e4fc4388cc80069 on master.


Network bandwidth for NC-Series?

Hello. I can't seem to find any information on the maximum network bandwidth for any GPU VM except for the NDv2-series.
It would be nice to detail on the far side column for every N-Series just as for NDv2-series.
Thanks.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: c946c151-c1a6-59bc-2e44-6a820547f48e
Version Independent ID: 572ed0c0-8c4d-3629-044d-c7a34c8281a6
Content: Azure Linux VM sizes - GPU
Content Source: articles/virtual-machines/linux/sizes-gpu.md
Service: virtual-machines-linux
GitHub Login: @jonbeck7
Microsoft Alias: jonbeck

Thanks for the feedback! We are currently investigating and will update you shortly.

@marcelofelman good point!
@jonbeck7 could you take a look and see if we can get this info added?

@MicahMcKittrick-MSFT please assign to either Jack Kabat who owns N series or @steveesp who owns networking speeds in the documentation

We are currently closing out long standing backlog in order to better determine which issues are still valid and which have been addressed. If you are still seeing a specific problem with this document please open a new issue so we can properly assign the feedback to the correct author and get it addressed ASAP.
#please-close


[WIP] Optimize the protocol handler start

Motivation
In K8s environment, the start of a KoP node might depend on a successful start of another KoP node. It's caused by the lookup request in loadOffsetTopics, here're the logs:
13:33:11.584 [pulsar-2-1] INFO  org.eclipse.jetty.server.RequestLog - <IP_ADDRESS> - - [08/Jul/2021:13:33:11 +0000] "GET /lookup/v2/topic/persistent/public/__kafka/__consumer_offsets-partition-0 HTTP/1.1" 307 0 "-" "Pulsar-Java-v2.8.0-rc-202106071430" 88
13:33:11.593 [pulsar-2-1] INFO  org.eclipse.jetty.server.RequestLog - <IP_ADDRESS> - - [08/Jul/2021:13:33:11 +0000] "GET /lookup/v2/topic/persistent/public/__kafka/__consumer_offsets-partition-0 HTTP/1.1" 307 0 "-" "Pulsar-Java-v2.8.0-rc-202106071430" 8
13:33:11.673 [pulsar-2-1] INFO  org.eclipse.jetty.server.RequestLog - <IP_ADDRESS> - - [08/Jul/2021:13:33:11 +0000] "GET /lookup/v2/topic/persistent/public/__kafka/__consumer_offsets-partition-0 HTTP/1.1" 307 0 "-" "Pulsar-Java-v2.8.0-rc-202106071430" 78
13:33:11.685 [pulsar-2-1] INFO  org.eclipse.jetty.server.RequestLog - <IP_ADDRESS> - - [08/Jul/2021:13:33:11 +0000] "GET /lookup/v2/topic/persistent/public/__kafka/__consumer_offsets-partition-0 HTTP/1.1" 307 0 "-" "Pulsar-Java-v2.8.0-rc-202106071430" 9
13:33:11.767 [pulsar-2-1] INFO  org.eclipse.jetty.server.RequestLog - <IP_ADDRESS> - - [08/Jul/2021:13:33:11 +0000] "GET /lookup/v2/topic/persistent/public/__kafka/__consumer_offsets-partition-0 HTTP/1.1" 307 0 "-" "Pulsar-Java-v2.8.0-rc-202106071430" 80
13:33:11.771 [main] ERROR io.streamnative.pulsar.handlers.kop.KafkaProtocolHandler - initGroupCoordinator failed with
org.apache.pulsar.client.admin.PulsarAdminException: java.util.concurrent.CompletionException: org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException: Could not complete the operation. Number of retries has been exhausted. Failed reason: test-broker-1.test-broker-headless.kop-test.svc.cluster.local
reason: test-broker-1.test-broker-headless.kop-test.svc.cluster.local
	at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:247) ~[io.streamnative-pulsar-client-admin-original-2.8.0-rc-202106071430.jar:2.8.0-rc-202106071430]
	at org.apache.pulsar.client.admin.internal.LookupImpl$1.failed(LookupImpl.java:85) ~[io.streamnative-pulsar-client-admin-original-2.8.0-rc-202106071430.jar:2.8.0-rc-202106071430]

<IP_ADDRESS> is the IP of another KoP node that had not started completely.
Actually loadOffsetTopics is not necessary, when a bundle load event happened, the offsets will be loaded in the bundle ownership listener. There's no reason to load the offsets in start() method, also when Kafka starts, it doesn't load offsets.
Modifications
The most important change is to remove the loadOffsetTopics method.
In addition, this PR does some code factors to KafkaProtocolHandler:

Remove unused fields
Don't catch unchecked exceptions to make start fail fast
Adjust the order of fields and methods to meet Codacy's requirement

Need to confirm if the tests became flaky after my change.

The GroupCoordinatorTest and GroupMetadataTest are flaky because of TwoPhaseCompactor$MockitoMock issues and can be fixed by #628. I'd rebase to master once #628 was merged.

@jiazhai PTAL

@BewareMyPower We need this one in the following 2.8.x release, do we need to cherry-pick it to branch-2.8?


Inline code/fixed-width looks a little small

Hi, Cayman folks!
I'm using Cayman for some of my sites on GitHub Pages.
When I use backticks to format text as inline code, it looks like the resulting fixed-width text is a bit smaller than the surrounding proportional-width text.
Here's an example from my Dispstr site, viewed in Chrome 89.0.4389.82 on macOS 10.14.6:

See how the fixed-width bits are shorter vertically, especially when comparing the ascent of lower-case characters, like "e" and "s"?
It seems especially pronounced in section headings:

Minor aesthetic and readability problem, I think.
Can anything be done to fix this, either in the Cayman theme itself, or something I can do in a GitHub-Pages-compatible customization in my Jekyll site?

I'm having the same problem:

I posted a question in Stack Overflow seeking guidance to changing just a portion of the .main_content scss:
  code {
    padding: 2px 4px;
    font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace;
    // font-size: 0.9rem;     // h1 was getting shrunk to .9rem
    color: $code-text-color;
    background-color: $code-bg-color;
    border-radius: 0.3rem;
  }

I created a file _sass/code.scss in my website with the above changes but it has no effect. Note I would rather not copy all the code across and miss out on future upgrades to Cayman Theme.
Thanks,

@apjanke
As shown below I've fixed the problem:

I made the fix by first copying the entire file https://github.com/pages-themes/cayman/blob/master/_sass/jekyll-theme-cayman.scss to my repo's _sass/ subdirectory.
Then I commented out the font-size: 0.9rem line below by inserting // at the start:
  code {
    padding: 2px 4px;
    font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace;
    // font-size: 0.9rem;  // Go with default font size
    color: $code-text-color;
    background-color: $code-bg-color;
    border-radius: 0.3rem;
  }

The things I don't like about my solution:

I can see why a normal code block should be 90% of the surrounding regular sized font in usual situations -- It simply looks better next to regular sized font.
I would rather have 90% of surrounding font rather than 100% which I now get. I don't know how to specify 90% of surrounding font though.
I don't like the fact if the Cayman Theme repo makes changes to this file I will not gain them as 398 lines of this file are now permanently in my own repo.

That said, I can live with this solution for now. If I improve it I will come back and update this answer.


add webserver

ajout d'un webserver permettant d'envoyer des commande depuis le site internet:
eg : une news a ete publie par darwyn : https://www.froggedtv.com/news/8

@alex-min  : i'm now using environment var for webserver port. thanks for your review.
@snickers54 : i removed the global var and used your solution, works fine.
@Philaeux, @alex-min , @snickers54 : i don't know why but i can't "spam" the command (which is good) i can only use it sometime (i'd say every 60 sec) or if i run another command just after (like !who) i think it's related to the sanitize threading.

I would like to run the webserver from the main class instead of running it in the irc thread, but i don't know how to have the reference to the irc object after, if you guys know, would be nice.

A user can't send the same message to a twitch chat too often, this is a IRC server limitation. You can try it on any twitch chat. But moderators don't have this limitation (they can spam as often as they like). So if you are testing in your own private channel, you can set your bot moderator.
You are missing a secret between the caller and the endpoint to secure your web server. Only a few people should be able to do a post. The best way to do it is to create a GRENOUILLE_API_KEY secret, shared between the bot and the caller to authorize the calls. Pass it in header/parameters. You also have to setup HTTPS to avoid secret stealing.
The sanitizer is a dirty thing I did because I failed to listen to some Twitch event. I might change it in the future.

I will look into your code in details in the afternoon.

Sorry https is not necessary, i'll manage the secure endpoint in haproxy. But the api secret is.

oh ok, thanks @Philaeux, didn't think of twitch irc limitations .
I thought about this stuff, didn't know if it would be that much critical for now, no one is supposed to know the ip address of the bot anyway, and basic people won't know they have to use something like postman to simulate a post request.
but i'll think about it after work :+1:

hey @Philaeux

I added the secret key stuff
I refactored grenouille_http class to start it from the bot_application class and not after irc is connected
I secured the send_msg by adding a try catch, it's light for now, a better solution would be to push message in an object and send them once we are connected again, but that's not the purpose of this pull request.

tell what you think about this right now  ?

Seems to work quite fine. I will do some rework to turn it into a thread, comment, and I will merge it. After deployment, the grenouillebot webserver will be accessible at grenouillebot.the-cluster.org. I will send you a generated secret by Skype

@Philaeux a lot of github accounts have been hacked, I wouldn't consider Skype as a secure channel to give such a secret.

@snickers54  it's fine we have our channel for the froggedtv project, with trusted accounts.
@Philaeux it's already running in a thread, not sure I understood your comment. let me know when you deploy this so I can deploy the code in the froggedtv website too


Consumer not being notified of new topics - Java and Cli [Bug]

Search before asking

[X] I searched in the issues and found nothing similar.

Version
OS: Ventura 13.5.2 on M1 Chip
Pulsar: 3.1.0
Minimal reproduce step

Create a docker-compose.yml and copy in the following:

version: '3.8'

services:
  pulsar:
    container_name: pulsar
    image: apachepulsar/pulsar:3.1.0
    ports:
      - "6650:6650"
      - "8080:8080"
      - "4181:4181"
    environment:
      - clusterName=local
      - authenticationEnabled=false
      - superUserRoles=admin
      - AWS_ACCESS_KEY_ID=accesskey
      - AWS_SECRET_ACCESS_KEY=secretkey
      - allowAutoTopicCreationType=partitioned
      - defaultNumPartitions=2
      - PULSAR_STANDALONE_USE_ZOOKEEPER=1
      - PULSAR_PREFIX_subscriptionPatternMaxLength=75
    command: >
      /bin/bash -c
      "bin/apply-config-from-env.py conf/standalone.conf
      && exec bin/pulsar standalone"

Run docker compose up --build in the same folder
Run docker exec -it <container id> bash
Run cd bin
Run ./pulsar-client consume -s example-sub "public/default/.*" --regex
In a new terminal repeat the docker exec and cd bin steps
Run ./pulsar-admin topics create-partitioned-topic persistent://public/default/example-topic --partitions 2
Run ./pulsar-admin topics subscriptions persistent://public/default/example-topic

What did you expect to see?
I expect to see example-sub listed on the subscriptions for example-topic showing that the consumer has been notified of the new topic.
What did you see instead?
I see no subscriptions on the topic even after waiting a sufficient amount of time.
Anything else?
Restarting the consumer results in the topic being discovered.
Are you willing to submit a PR?

[X] I'm willing to submit a PR!

i tried on 3.0.1(latest) and it works fine.

@muni-chada 3.0.1 isn't the latest

It appears the topic pattern needs to be a full topic url - e.g. this works:
./pulsar-client consume -s example-sub --regex "persistent://public/default/.*"

Thanks, @marksilcox . Would be interested if someone can point out where that changed as we never used to have to provide that (same with the -partition-\d on the end of the pattern.


rename and bump version

Codecov Report

Merging #7 into master will not change coverage.
The diff coverage is 100%.

@@           Coverage Diff           @@
##           master       #7   +/-   ##
=======================================
  Coverage   38.23%   38.23%
=======================================
  Files           6        6
  Lines          68       68
=======================================
  Hits           26       26
  Misses         42       42

Impacted Files
Coverage Δ

simqle/__init__.py
100% <100%> (ø)
:arrow_up:

simqle/sql_tools.py
29.16% <100%> (ø)
:arrow_up:

tests/test_simqle.py
100% <100%> (ø)
:arrow_up:

simqle/functions.py
16.66% <100%> (ø)
:arrow_up:

simqle/connections.py
44.44% <100%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update da5407a...e945801. Read the comment docs.

closes #6


Issue: TokenTextSplitter processes passages less than chunk_size and creates duplicate overlaps

Issue you'd like to raise.
Observation: TokenTextSplitter processes passages smaller than specified chunk_size, and creates a lot more smaller chunks with duplicated overlaps.
Impact: The smaller chunks with duplicated overlaps may be harmful to storage size, computation time, and it may confuse the retrieval model.
For example: The second TokenTextSplitter creates 366 more small chunks, even if new_chunks1 doesn't have any passages longer than chunk_size=500.

Suggestion:
In source code for langchain.text_splitter, split_text_on_tokens function, can we add the check for
If len(input_ids) <= tokenizer.tokens_per_chunk:  splits.append(tokenizer.decode(input_ids))

TokenTextSplitter in nodejs also takes up too much CPU usage.

I am waiting for this too. This chunk sizes are not predictable

Yes , I faced the same issue @c3-YuelingWu  . The ways this token text splitter works , isn't how it is intended to work .
For Example :-
When specified chunk_size = 1024 and overlap = 256  with input text tokens = 991 , it made two chunks ;
first = token[0 : 991]
second = token[768 : 991]
but logically it should work this ways .

If input text token >1024 , there should be two chunks .
first = token[0 : 1024]
second = token[768 : ]
and so on ..............

If input text token <=1024 , there should be one chunk only .
first = token[ 0 : ]


Include optional route table at subnet level in core module

Route tables is needed when having a centralized NVA in example HUB/Spoke topology and it would be useful to have the option at the subnet array in vnet_config

Closed by #144


Net charmrun should retry failed ssh connections

Original issue: https://charm.cs.illinois.edu/redmine/issues/706

A failed ssh connection from charmrun to a compute node is not necessarily a permanent error. Within some reasonable limit, charmrun should retry failed connection attempts that cannot be definitively pinned to a permanent cause.

Original date: 2015-04-07 01:29:13

We may need to revisit this later to add some delay or back-off, but for now, this is done.


uninitialized constant ProMotion::UITabBarController (NameError)

Hi 👋
While  following the README example (using the ProMotion v2.8.2) I am not able to compile/run the application using rake. The process crashes ending up with
cocoatouch/tab_bar_controller.: uninitialized constant ProMotion::UITabBarController (NameError)

I am using Ruby v2.5.0 on macOS v10.13.6 and ProMotion v2.8.2. You can inspect my full example application here on GitHub.
I'd appreciate any hints or help 🌻
Here's the full error log:
==========================================================================================
WARNING: Using offline templates. Run `motion repo` to download the latest templates.
==========================================================================================
Ignoring bcrypt-3.1.11 because its extensions are not built.  Try: gem pristine bcrypt --version 3.1.11
Ignoring bindex-0.5.0 because its extensions are not built.  Try: gem pristine bindex --version 0.5.0
Ignoring bootsnap-1.3.0 because its extensions are not built.  Try: gem pristine bootsnap --version 1.3.0
Ignoring byebug-10.0.2 because its extensions are not built.  Try: gem pristine byebug --version 10.0.2
Ignoring byebug-10.0.1 because its extensions are not built.  Try: gem pristine byebug --version 10.0.1
Ignoring byebug-9.0.6 because its extensions are not built.  Try: gem pristine byebug --version 9.0.6
Ignoring debug_inspector-0.0.2 because its extensions are not built.  Try: gem pristine debug_inspector --version 0.0.2
==========================================================================================
WARNING: Using offline templates. Run `motion repo` to download the latest templates.
==========================================================================================
     Build ./build/MacOSX-10.13-Development
  WARNING! Possible cyclical dependency between /Users/robin/.rvm/gems/ruby-2.5.0/gems/ProMotion-2.8.2/lib/ProMotion/collection/data/collection_data.rb and /Users/robin/.rvm/gems/ruby-2.5.0/gems/ProMotion-2.8.2/lib/ProMotion/collection/collection.rb's class hierarchy. Consider revision if runtime exceptions occur around undefined symbols.
  WARNING! Possible cyclical dependency between /Users/robin/.rvm/gems/ruby-2.5.0/gems/ProMotion-2.8.2/lib/ProMotion/table/data/table_data.rb and /Users/robin/.rvm/gems/ruby-2.5.0/gems/ProMotion-2.8.2/lib/ProMotion/table/table.rb's class hierarchy. Consider revision if runtime exceptions occur around undefined symbols.

'
*** First throw call stack:
(
	0   CoreFoundation                      0x00007fff37ef423b __exceptionPreprocess + 171
	1   libobjc.A.dylib                     0x00007fff5f177c76 objc_exception_throw + 48
	2   monkeys_and_pirates                 0x00000001002944f0 _ZL10__vm_raisev + 336
	3   monkeys_and_pirates                 0x0000000100294580 rb_vm_raise + 80
	4   monkeys_and_pirates                 0x000000010019fd69 rb_exc_raise + 9
	5   monkeys_and_pirates                 0x000000010019a218 rb_name_error + 216
	6   monkeys_and_pirates                 0x0000000100253020 rb_mod_const_missing + 80
	7   monkeys_and_pirates                 0x000000010027a5cc rb_vm_dispatch + 5564
	8   monkeys_and_pirates                 0x0000000100253ebc rb_const_get_0 + 1084
	9   monkeys_and_pirates                 0x000000010001ac3d vm_get_const + 269
	10  monkeys_and_pirates                 0x000000010001e3e8 rb_scope2 + 56
	11  monkeys_and_pirates                 0x000000010001e35b rb_scope1 + 123
	12  monkeys_and_pirates                 0x00000001000220eb MREP_57E22BD4157B407DB7A8B37C37FF2B7C + 2779
	13  monkeys_and_pirates                 0x0000000100001bbc RubyMotionInit + 204
	14  monkeys_and_pirates                 0x0000000100001f0b main + 107
	15  libdyld.dylib                       0x00007fff5fd91015 start + 1
	16  ???                                 0x0000000000000001 0x0 + 1
)
libc++abi.dylib: terminating with uncaught exception of type NameError

I've also tested this against Ruby 2.3.4, 2.4.0 and with ProMotion v2.7.1, which all results in the same problem. And I forgot to mentiomn using RubyMotion v5.16. Could be also the case that this issue is in fact a problem of the underlying RubyMotion (?) 🤔

@neumanrq Can you try this branch and let me know if it works? https://github.com/infinitered/ProMotion/tree/bugfix/issue-810
Use this in your gemfile:
gem 'ProMotion', github: 'infinitered/ProMotion', branch: 'bugfix/issue-810'

Doh! #faceplam. I didn't even catch that.


blocksprogress can exceed blockstotal

As per https://github.com/joehand/dat-next/blob/master/lib/download.js#L114-L115 - chances is this might be fixed in SLEEP b/c we get a better grip on bytes progress and no longer need to rely on hyperdrive-stats, but still worth opening an issue for I reckon (:

Closing now that this repo is deprecated - moved to https://github.com/datproject/dat/issues/638


Statically render all pages other than inventory sheets

All pages except for [sheetId].tsx should be statically rendered at build time.

NextJS is already doing this as per its default configuration.


Consumer Group reconnect loses Topic

Issue: After kafka and zookeeper are restarted my consumer fails to connect to the topic and hence doesn't consume messages.
During testing of reconnect, I find that when my consumer group emit's "connect" the first time it has topicPayloads set as expected and everything works. However after a reconnect (bringing docker down and back up, topicPayloads becomes an empty array (when I console log my consumer in .on("connect"))
Update:
I found that calling refreshMetadata inside the hook for "rebalancing" fixes that issue and my consumer has topicPayloads set and consumes messages. However after the reconnect it consume the same message twice everytime.

How many consumers do you have running and how many partitions? This would happen if you were to have more consumers than partitions